pimlico.datatypes.keras module

Datatypes for storing and loading Keras models.

class KerasModelWriter(base_dir, **kwargs)

Bases: pimlico.datatypes.base.PimlicoDatatypeWriter

Writer for storing both types of Keras model (since they provide the same storage interface).

write_model(model)

write_architecture(model)

write_weights(model)

class KerasModel(base_dir, pipeline, module=None, additional_name=None, use_main_metadata=False, **kwargs)

Bases: pimlico.datatypes.base.PimlicoDatatype

Datatype for both types of Keras models, stored using Keras’ own storage mechanisms. This uses Keras’ method of storing the model architecture as JSON and stores the weights using hdf5.

datatype_name = 'keras_model'

custom_objects = {}

get_software_dependencies()

Check that all software required to read this datatype is installed and locatable. This is separate to metadata config checks, so that you don’t need to satisfy the dependencies for all modules in order to be able to run one of them. You might, for example, want to run different modules on different machines. This is called when a module is about to be executed and each of the dependencies is checked.

Returns a list of instances of subclasses of :class:~pimlico.core.dependencies.base.SoftwareDependency, representing the libraries that this module depends on.

Take care when providing dependency classes that you don’t put any import statements at the top of the Python module that will make loading the dependency type itself dependent on runtime dependencies. You’ll want to run import checks by putting import statements within this method.

You should call the super method for checking superclass dependencies.

get_custom_objects()

load_model()

class KerasModelBuilderClassWriter(base_dir, build_params, builder_class_path, **kwargs)

Bases: pimlico.datatypes.base.PimlicoDatatypeWriter

Writer for storing Keras models in the manner described in :cls:KerasModelBuilderClass.

write_weights(model)

class KerasModelBuilderClass(base_dir, pipeline, **kwargs)

Bases: pimlico.datatypes.base.PimlicoDatatype

An alternative way to store Keras models.

Create a class whose init method build the model architecture. It should take a kwarg called build_params, which is a JSON-encodable dictionary of parameters that determine how the model gets build (hyperparameters). When you initialize your model for training, create this hyperparameter dictionary and use it to instantiate the model class.

Use the KerasModelBuilderClassWriter to store the model during training. Create a writer, then start model training, storing the weights to the filename given by the weights_filename attribute of the writer. The hyperparameter dictionary will also be stored.

The writer also stores the fully-qualified path of the model-builder class. When we read the datatype and want to rebuild the model, we import the class, instantiate it and then set its weights to those we’ve stored.

The model builder class must have the model stored in an attribute model.

weights_filename

load_build_params()

create_builder_class(override_params=None)

load_model(override_params=None)

Instantiate the model builder class with the stored parameters and set the weights on the model to those stored.

Returns:model builder instance (keras model in attribute model