pimlico.datatypes.base module

Datatypes provide interfaces for reading (and in some cases writing) datasets. At their most basic, they define a way to iterate over a dataset linearly. Some datatypes may also provide other functionality, such as random access or compression.

As much as possible, Pimlico pipelines should use standard datatypes to connect up the output of modules with the input of others. Most datatypes have a lot in common, which should be reflected in their sharing common base classes. Custom datatypes will be needed for most datasets when they’re used as inputs, but as far as possible, these should be converted into standard datatypes, or at least a form where they can use standard idioms for iterating, etc, early in the pipeline.

exception pimlico.datatypes.base.DatatypeLoadError

Bases: exceptions.Exception

exception pimlico.datatypes.base.DatatypeWriteError

Bases: exceptions.Exception

class pimlico.datatypes.base.PimlicoDatatype(base_dir, pipeline, **kwargs)

Bases: object

The abstract superclass of all datatypes. Provides basic functionality for identifying where data should be stored and such.

Datatypes are used to specify the routines for reading the output from modules. They’re also used to specify how to read pipeline inputs. Most datatypes that have data simply read it in when required. Some (in particular those used as inputs) need a preparation phase to be run, where the raw data itself isn’t sufficient to implement the reading interfaces required. In this case, they should override prepare_data().

Datatypes may require/allow options to be set when they’re used to read pipeline inputs. These are specified, in the same way as module options, by input_module_options on the datatype class.

check_runtime_dependencies()

Like the similarly named method on executors, this check dependencies for using the datatype. It’s not called when checking basic config, but only when the datatype is needed.

Returns a list of pairs: (dependency short name, description/error message)

Deprecated since version 0.2: You should provide dependency information via get_software_dependencies() instead. This method will be called as well for backward compatibility until v1.

classmethod create_from_options(base_dir, pipeline, options={})

data_ready()

Check whether the data corresponding to this datatype instance exists and is ready to be read.

Default implementation just checks whether the data dir exists. Subclasses might want to add their own checks, or even override this, if the data dir isn’t needed.

classmethod datatype_full_class_name()

The fully qualified name of the class for this datatype, by which it is reference in config files. Generally, datatypes don’t need to override this, but type requirements that take the place of datatypes for type checking need to provide it.

get_detailed_status()

Returns a list of strings, containing detailed information about the data. Only called if data_ready() == True.

Subclasses may override this to supply useful (human-readable) information specific to the datatype. They should called the super method.

get_software_dependencies()

Check that all software required to read this datatype is installed and locatable. This is separate to metadata config checks, so that you don’t need to satisfy the dependencies for all modules in order to be able to run one of them. You might, for example, want to run different modules on different machines. This is called when a module is about to be executed and each of the dependencies is checked.

Returns a list of instances of subclasses of :class:~pimlico.core.dependencies.base.SoftwareDependency, representing the libraries that this module depends on.

Take care when providing dependency classes that you don’t put any import statements at the top of the Python module that will make loading the dependency type itself dependent on runtime dependencies. You’ll want to run import checks by putting import statements within this method.

You should call the super method for checking superclass dependencies.

prepare_data(output_dir, log)

datatype_name = 'base_datatype'

input_module_options = {}

Override to provide shell commands specific to this datatype. Should include the superclass’ list.

metadata

requires_data_preparation = False

shell_commands = []

class pimlico.datatypes.base.PimlicoDatatypeWriter(base_dir)

Bases: object

Abstract base class fo data writer associated with Pimlico datatypes.

write_metadata()

class pimlico.datatypes.base.IterableCorpus(*args, **kwargs)

Bases: pimlico.datatypes.base.PimlicoDatatype

Superclass of all datatypes which represent a dataset that can be iterated over document by document (or datapoint by datapoint - what exactly we’re iterating over may vary, though documents are most common). The actual type of the data depends on the subclass: it could be, e.g. coref output, etc.

At creation time, length should be provided in the metadata, denoting how many documents are in the dataset.

get_detailed_status()

datatype_name = 'iterable_corpus'

class pimlico.datatypes.base.IterableCorpusWriter(base_dir)

Bases: pimlico.datatypes.base.PimlicoDatatypeWriter

class pimlico.datatypes.base.DynamicOutputDatatype

Bases: object

Types of module outputs may be specified as a subclass of PimlicoDatatype, or alternatively as an instance of DynamicOutputType. In this case, get_datatype() is called when the output datatype is needed, passing in the module info instance for the module, so that a specialized datatype can be produced on the basis of options, input types, etc.

The dynamic type must provide certain pieces of information needed for typechecking.

get_base_datatype_class()

If it’s possible to say before the instance of a ModuleInfo is available what base datatype will be produced, implement this to return the class. By default, it returns None.

If this information is available, it will be used in documentation.

get_datatype(module_info)

datatype_name = None

class pimlico.datatypes.base.DynamicInputDatatypeRequirement

Bases: object

Types of module inputs may be given as a subclass of PimlicoDatatype, a tuple of datatypes, or an instance a DynamicInputDatatypeRequirement subclass. In this case, check_type(supplied_type) is called during typechecking to check whether the type that we’ve got conforms to the input type requirements.

Additionally, if datatype_doc_info is provided, it is used to represent the input type constraints in documentation.

check_type(supplied_type)

datatype_doc_info = None

class pimlico.datatypes.base.InvalidDocument(module_name, error_info=None)

Bases: object

Widely used in Pimlico to represent an empty document that is empty not because the original input document was empty, but because a module along the way had an error processing it. Document readers/writers should generally be robust to this and simply pass through the whole thing where possible, so that it’s always possible to work out, where one of these pops up, where the error occurred.

static invalid_document_or_text(text)

If the text represents an invalid document, parse it and return an InvalidDocument object. Otherwise, return the text as is.

static load(text)

pimlico.datatypes.base.load_datatype(path)

Try loading a datatype class for a given path. Raises a DatatypeLoadError if it’s not a valid datatype path.