loaddump

class DumpCmd[source]

Bases: pimlico.cli.subcommands.PimlicoCLISubcommand

Dump the entire available output data from a given pipeline module to a tarball, so that it can easily be loaded into the same pipeline on another system. This is primarily to support spreading the execution of a pipeline between multiple machines, so that the output from a module can easily be transferred and loaded into a pipeline.

Dump to a tarball using this command, transfer the file between machines and then run the load command to import it there.

See also

Running on multiple computers: for a more detailed guide to transferring data across servers

command_name = 'dump'
command_help = 'Dump the entire available output data from a given pipeline module to a tarball, so that it can easily be loaded into the same pipeline on another system. This is primarily to support spreading the execution of a pipeline between multiple machines, so that the output from a module can easily be transferred and loaded into a pipeline'
command_desc = 'Dump the entire available output data from a given pipeline module to a tarball'
add_arguments(parser)[source]
run_command(pipeline, opts)[source]
class LoadCmd[source]

Bases: pimlico.cli.subcommands.PimlicoCLISubcommand

Load the output data for a given pipeline module from a tarball previously created by the dump command (typically on another machine). This is primarily to support spreading the execution of a pipeline between multiple machines, so that the output from a module can easily be transferred and loaded into a pipeline.

Dump to a tarball using the dump command, transfer the file between machines and then run this command to import it there.

See also

Running on multiple computers: for a more detailed guide to transferring data across servers

command_name = 'load'
command_help = "Load a module's output data from a tarball previously created by the dump command, usually on a different system. This will overwrite any output data the module already has completely, including metadata, run history, etc. You may load multiple modules' data at once"
command_desc = "Load a module's output data from a tarball previously created by the dump command"
add_arguments(parser)[source]
run_command(pipeline, opts)[source]