Job library#
The built-in jobs, by area. Each job’s page lists its parameters and has an example.
Files#
|
Copy from src to destination. |
|
Move src to dst. |
|
Delete the file passed in delete params. |
|
Create the file as requested and write contents if provided. |
|
Create the directory, and any missing parents (it may already exist). |
|
Use python shutil to copy directories and files. |
|
Move from a source to a destination. |
|
Use rsync to copy from source to destination. |
Check if a file exists. |
|
Check if a file contains a substring. |
|
|
Edit a .json file in place. |
|
Edit a .yaml file in place. |
Edit a file using user-provided functionality. |
|
Use polars to load a csv and dump it into a parquet. |
|
Use polars to load an excel and dump it into a csv. |
Analysis#
Perform a simple operation between two columns. |
|
Join two dataframes read from a parquet. |
|
|
Call a user defined function and save the results off. |
Call the external function. |
|
Call some external function with the data loaded into polars. |
Subprocesses#
Dispatch a command in a subprocess. |
Branching#
Submit a job based on a conditional value. |
|
Execute a job iff a condition is true. |
|
|
Create a case matrix as a cartesian product of the variables. |
Submit a set of pipelines based on the case matrix. |
|
|
Add variables output_dir and pipeline_id for each child pipeline. |
Wrappers#
Run the pipeline of jobs sequentially. |
|
Run a series of jobs locally for a set of iterations. |
|
Wrap a set of jobs in a new submitter. |
|
Run a set of jobs on a new host. |
|
Set the variables for subsequent defined jobs. |
|
Submit the set of jobs with the shared variables. |
|
Run jobs with a temporary directory. |
Virtual jobs#
Parameters to add a variable. |
Testing#
|
Do nothing and return success. |
Touch a file (create it with no values). |