JobHandler

JobHandler(name=None, backend=None, work_dir='', local_max=0, local_dynamic=False, verbosity=1, success_func=None, finished_func=None, max_retries=0, theme=<Theme.Lovecraft: 1>, run_max=None, do_snapshot=True, use_snapshot=False, description=None, wrapper=None, profiler=None, listens=True, output_max_attempts=5, printer_bar_mode=True)

Main handle to setup and submit jobs. Internally stores most information in the JobHandlerConfig class, which is stored on disk as a snapshot of the JobHandler session.

  • name Name of the JobHandler. Defines the base directory name of the session.
  • backend Default backend instance used for the job setup.
  • work_dir Path where the base directory is created.
  • local_max Maximum number of local jobs that will be submitted at a time.
  • local_dynamic Switch to dynamically allocate jobs to run locally, up to the local_max maximum.
  • verbosity Verbosity of the shell output.
  • success_func Default success function used for the job setup.
  • finished_func Default finished function used for the job setup.
  • max_retries Maximum number of retries that are attempted for failing jobs.
  • theme Naming theme used to name the jobhandler and jobs.
  • run_max Maximum number of jobs that are submitted at a time.
  • do_snapshot Turn on/off snapshot creation. This is needed to load jobhandler instances in interactive slurmy.
  • use_snapshot Load snapshot from disk instead of creating new jobhandler.
  • description Description of jobhandler that is stored in the bookkeeping.
  • wrapper Default run script wrapper used for the job setup.
  • profiler Profiler to be used for profiling.
  • printer_bar_mode Turn bar mode of the printer on/off.

Member functions

add_job

JobHandler.add_job(self, backend=None, run_script=None, run_args=None, success_func=None, finished_func=None, post_func=None, max_retries=None, output=None, tags=None, parent_tags=None, name=None, job_type=<Type.BATCH: 0>, starttime=None)

Add a job to the list of jobs to be processed by the JobHandler.

  • backend Backend instance to be used by the job.
  • run_script The run script processed by the job. This can be a string specifying the content of the script or a the absolute path to an already existing script file.
  • run_args The arguments passed to the run script.
  • success_func Success function used for the job setup.
  • finished_func Finished function used for the job setup.
  • post_func Post execution function used for the job setup.
  • max_retries Maximum number of retries that are attempted when job is failing.
  • output Output file of the job. This must be an absolute path.
  • tags List of tags attached to the job.
  • parent_tags List of parent tags attached to the job.
  • name Name of the job. This must be a string that conforms with the restrictions on class property names. Slurmy will make sure that job names stay unique, even if the same job name is set multiple times.
  • job_type Type of the job. Can be set to Type.LOCAL to make it a local processing job.
  • starttime Timestamp at which job is started by the JobHandler.

Returns the job (Job).

cancel_jobs

JobHandler.cancel_jobs(self, tags=None, only_local=False, only_batch=False, make_snapshot=True)

Cancel running jobs.

  • tags Tags of jobs that will be cancelled.
  • only_local Cancel only local jobs.
  • only_batch Cancel only batch jobs.
  • make_snapshot Make a snapshot after cancelling jobs.

check

JobHandler.check(self, force_success_check=False, skip_eval=False, print_summary=True)

Check the status and tags of the jobs.

  • force_success_check Force the success routine to be run, even if the job is already in a post-finished state.
  • print_summary Print the job processing summary.

reset

JobHandler.reset(self, skip_jobs=False)

Reset the JobHandler session.

  • skip_jobs Skip job reset.

run_jobs

JobHandler.run_jobs(self, interval=1, retry=False)

Run the job submission routine. Jobs will be submitted continuously until all of them have been processed.

  • interval The interval at which the job submission will be done (in seconds). Can also be set to -1 to start every submission cycle manually (will not work if Listeners are used).
  • retry Retry jobs in status FAILED or CANCELLED. This will attempt one cycle of job retrying.

set_jobs_config_attr

JobHandler.set_jobs_config_attr(self, attr, val, tags=None, states=None)

Set the job config attribute of jobs associated to the JobHandler.

  • attr Name of the attribute which is set.
  • val Value that the attribute is set to.
  • tags Set of tags which the jobs have to match to.
  • states Set of job states which the jobs have to match to.

submit_jobs

JobHandler.submit_jobs(self, tags=None, make_snapshot=True, wait=True, retry=False, skip_eval=False)

Submit jobs according to the JobHandler configuration.

  • tags Tags of jobs that will be submitted.
  • make_snapshot Make a snapshot of the jobs and the JobHandler after the submission cycle.
  • wait Wait for locally submitted job.
  • retry Retry jobs in status FAILED or CANCELLED. This circumvents the automatic retry routine.
  • skip_eval Skip job status evaluation everywhere.

update_snapshot

JobHandler.update_snapshot(self, skip_jobs=False)

Update snapshots of the JobHandler and the associated Jobs on disk. Snaphots are only updated if something changed in the respective JobHandlerConfig or JobConfig.

  • skip_jobs Skip the job snapshot update.