Core Pimlico modules

Pimlico comes with a substantial collection of module types that provide wrappers around existing NLP and machine learning tools, as well as a number of general tools for processing datasets that are useful for many applications.

Some modules that used to be among the core modules have not yet been updated since a big change in the datatypes system. They can be found in pimlico.old_datatypes.modules, but are not currently functional, until I get round to updating them.

• Corpus manipulation
  • Corpus concatenation
  • Corpus statistics
  • Human-readable formatting
  • Archive grouper (filter)
  • Interleaved corpora
  • Corpus document list filter
  • Random shuffle
  • Random shuffle (linear)
  • Corpus split
  • Store a corpus
  • Random subsample
  • Corpus subset
  • Corpus vocab builder
  • Token frequency counter
  • Tokenized corpus to ID mapper
  • ID to tokenized corpus mapper
• Embeddings
  • fastText embedding trainer
  • GloVe embedding trainer
  • Doc embedding mappers
  • Normalize embeddings
  • Store embeddings (internal)
  • Store in TSV format
  • Store in word2vec format
  • Word2vec embedding trainer
• Gensim topic modelling
  • Topic model topic coherence
  • LDA trainer
  • LDA document topic analysis
  • LDA top words
  • LDA-seq (DTM) trainer
  • LDA-seq (DTM) document topic analysis
• Input readers
  • Embeddings
  • Text corpora
  • XML files
• Malt dependency parser
• NLTK
  • NIST tokenizer
• OpenNLP tools
  • Constituency parser
  • POS-tagger
  • Tokenizer
• Output modules
  • Text corpus directory
• Scikit-learn tools
  • Sklearn logistic regression
• spaCy
  • NP chunk extractor
  • Text parser
  • Tokenizer
• Document-level text filters
  • Text to character level
  • Normalize tokenized text
  • Simple tokenization
  • Normalize raw text
• General utilities
  • Module output alias
  • Collect files
• Visualization tools
  • Bar chart plotter
  • Embedding space plotter