execute

Runtime execution of modules

This module provides the functionality to check that Pimlico modules are ready to execute and execute them. It is used by the run command.

check_and_execute_modules(pipeline, module_names, force_rerun=False, debug=False, log=None, all_deps=False, check_only=False, exit_on_error=False, preliminary=False, email=None)

Main method called by the run command that first checks a pipeline, checks all pre-execution requirements of the modules to be executed and then executes each of them. The most common case is to execute just one module, but a sequence may be given.

Parameters:

• exit_on_error – drop out if a ModuleExecutionError occurs in any individual module, instead of continuing to the next module that can be run
• pipeline – loaded PipelineConfig
• module_names – list of names of modules to execute in the order they should be run
• force_rerun – execute modules, even if they’re already marked as complete
• debug – output debugging info
• log – logger, if you have one you want to reuse
• all_deps – also include unexecuted dependencies of the given modules
• check_only – run all checks, but stop before executing. Used for check command

Returns:

check_modules_ready(pipeline, modules, log, preliminary=False)

Check that a module is ready to be executed. Always called before execution begins.

Parameters:

• pipeline – loaded PipelineConfig
• modules – loaded ModuleInfo instances, given in the order they’re going to be executed. For each module, it’s assumed that those before it in the list have already been run when it is run.
• log – logger to output to

Returns:

If preliminary=True, list of problems that were ignored by allowing preliminary run. Otherwise, None – we raise an exception when we first encounter a problem

execute_modules(pipeline, modules, log, force_rerun=False, debug=False, exit_on_error=False, preliminary=False, email=None)

format_execution_dependency_tree(tree)

Takes a tree structure of modules and their inputs, tracing where inputs to a module come from, and formats it recursively for output to the logs.

Parameters:tree – pair (module name, inputs list), where each input is a tuple (input name, previous module output name, previous module subtree) Returns:list of lines of formatted string

send_final_report_email(pipeline, error_modules, success_modules, skipped_modules, all_modules)

send_module_report_email(pipeline, module, short_error, long_error)

exception ModuleExecutionError(*args, **kwargs)

Bases: Exception

Base for any errors encountered during execution of a module.

Note that the cause attribute is used to trace the cause of an exception, so a chain can be built.

This is now provided as standard using the raise ... from ... syntax in Python 3, which can be accessed in Python 2 using future’s raise_from(). The cause attribute should gradually be replaced by this, which works better.

exception ModuleNotReadyError(*args, **kwargs)

Bases: pimlico.core.modules.execute.ModuleExecutionError

exception ModuleAlreadyCompletedError(*args, **kwargs)

Bases: pimlico.core.modules.execute.ModuleExecutionError

exception StopProcessing

Bases: Exception