pyantz.infrastructure.config.runners package

pyantz.infrastructure.config.runners package#

Submodules#

pyantz.infrastructure.config.runners.local_runner module#

Configuration for the local multiprocessing runner.

LocalRunnerConfig describes how many worker processes to spawn, where to store the SQLite queue, how often to poll for new jobs, and an optional overall timeout. It is the default runner for local development and testing.

Example of LocalRunnerConfig:

from pyantz.infrastructure.config import LocalRunnerConfig

config = LocalRunnerConfig(working_directory=".", number_processes=4)
print(config.type_, config.number_processes, config.use_same_proc)

local_proc 4 False
class LocalRunnerConfig(*, type_: Literal['local_proc'] = 'local_proc', working_directory: Path, poll_timeout: float = 1.0, number_processes: int = 2, timeout: float | None = None, progress_interval: float | None = 30.0, use_same_proc: bool = False)[source]#

Bases: BaseModel

Configuration for the local runner.

Parameters:

  • type_ (Literal['local_proc'])

  • working_directory (Path)

  • poll_timeout (float)

  • number_processes (int)

  • timeout (float | None)

  • progress_interval (float | None)

  • use_same_proc (bool)

_abc_impl = <_abc._abc_data object>#
number_processes: int#

How many worker processes to start.

poll_timeout: float#

Seconds to wait between each check of the queue while waiting for jobs.

progress_interval: float | None#

With several worker processes, log a progress report this often (in seconds); None turns the reports off.

timeout: float | None#

If set, the runner stops when it hasn’t received a job for this many seconds.

type_: Literal['local_proc']#

Identifies this submitter configuration.

use_same_proc: bool#

If true, run the jobs in this process rather than a pool (easier debugging and faster for small jobs).

working_directory: Path#

Location to save temporary files and the queue (queue.db).

pyantz.infrastructure.config.runners.slurm_runner module#

Slurm runner runs on a slurm HPC grid.

Note, setting inherit_dependencies=False creates a slurm submitter which doesn’t inherit dependencies.

Critically, this fully disallows inheriting dependencies. For example, consider the below pipeline

A -> B -> C

If A submits job A1 and A2, we would want B to wait until A1 and A2 have completed. Local and Slurm runner “inherit” this dependency by modifying B to be dependent on A1 and A2. This is no longer possible in this situation. Therefore, the run order may be

A -> B -> A1 -> C -> A2

In fact, any ordering of A1 and A2 is possible as they will be unlinked from the dependency tree and run as children.

Why use this? Some security policies disallow modifying jobs after submitting. There are a few ways to handle this, but in many cases, a user may not care.

For example, consider the same pipeline but imagine that C was submitting jobs. It doesn’t matter than C will spawn “dependency-less” children, it is the last job. So any execution order is find for the spawned chains to execute.

Something like the below is possible, which fits many use cases:

              C1
            /
A -> B -> C
            \ C2

Example of SlurmRunnerConfig:

from pyantz.infrastructure.config.runners import SlurmRunnerConfig

config = SlurmRunnerConfig(
    working_dir="/shared/pipeline", slurm_args=["--partition=short"]
)
print(config.type_, config.max_job_time_seconds, config.slurm_args)

slurm_sbatch 10000 ['--partition=short']
DEFAULT_JOB_TIMEOUT: int = 10000#

Default of SlurmRunnerConfig.max_job_time_seconds.

class SlurmRunnerConfig(*, type_: Literal['slurm_sbatch'] = 'slurm_sbatch', max_job_time_seconds: int = 10000, slurm_args: list[str] = <factory>, working_dir: Path, inherit_dependencies: bool = True)[source]#

Bases: BaseModel

Configuration used by the slurm submitters.

Parameters:

  • type_ (Literal['slurm_sbatch'])

  • max_job_time_seconds (int)

  • slurm_args (list[str])

  • working_dir (Path)

  • inherit_dependencies (bool)

_abc_impl = <_abc._abc_data object>#
inherit_dependencies: bool#

if true, A depends on P1 and P2 too (which needs scontrol to modify the pending jobs); if false, A depends only on P and may run before P1 and P2.

Type:

If some job A depends on job P and job P submits jobs P1 and P2

max_job_time_seconds: int#

Max time a job can be running before slurm stops it. Passed to sbatch as --time (in whole minutes, rounded up) unless slurm_args sets a time limit.

slurm_args: list[str]#

Extra arguments for every sbatch call. Unless they include --kill-on-invalid-dep, jobs are submitted with --kill-on-invalid-dep=yes so that when a job fails, the jobs waiting on it are cancelled instead of staying pending forever.

type_: Literal['slurm_sbatch']#

Identifies this submitter configuration.

working_dir: Path#

Directory shared between the submitting node and the compute nodes. Not checked when the config is parsed: with a remote host it only exists there (it is checked when the slurm runner starts).

Module contents#

Configurations for the various runners to be used by pyantz.