core

Some basic core datatypes that are commonly used for passing simple data, like strings and dicts, through pipelines.

class Dict(*args, **kwargs)

Bases: pimlico.datatypes.base.PimlicoDatatype

Simply stores a Python dict, pickled to disk. All content in the dict should be pickleable.

datatype_name = 'dict'

datatype_supports_python2 = True

class Reader(datatype, setup, pipeline, module=None)

Bases: pimlico.datatypes.base.Reader

Reader class for Dict

class Setup(datatype, data_paths)

Bases: pimlico.datatypes.base.Setup

Setup class for Dict.Reader

get_required_paths()

May be overridden by subclasses to provide a list of paths (absolute, or relative to the data dir) that must exist for the data to be considered ready.

reader_type

alias of Dict.Reader

get_dict()

class Writer(datatype, base_dir, pipeline, module=None, **kwargs)

Bases: pimlico.datatypes.base.Writer

Writer class for Dict

required_tasks = ['dict']

write_dict(d)

metadata_defaults = {}

writer_param_defaults = {}

class StringList(*args, **kwargs)

Bases: pimlico.datatypes.base.PimlicoDatatype

Simply stores a Python list of strings, written out to disk in a readable form. Not the most efficient format, but if the list isn’t humungous it’s OK (e.g. storing vocabularies).

datatype_name = 'string_list'

datatype_supports_python2 = True

class Reader(datatype, setup, pipeline, module=None)

Bases: pimlico.datatypes.base.Reader

Reader class for StringList

class Setup(datatype, data_paths)

Bases: pimlico.datatypes.base.Setup

Setup class for StringList.Reader

get_required_paths()

May be overridden by subclasses to provide a list of paths (absolute, or relative to the data dir) that must exist for the data to be considered ready.

reader_type

alias of StringList.Reader

get_list()

class Writer(datatype, base_dir, pipeline, module=None, **kwargs)

Bases: pimlico.datatypes.base.Writer

Writer class for StringList

required_tasks = ['list']

write_list(l)

metadata_defaults = {}

writer_param_defaults = {}