pysummarization.tokenizabledoc package

Submodules

pysummarization.tokenizabledoc.mecab_tokenizer module

class pysummarization.tokenizabledoc.mecab_tokenizer.MeCabTokenizer[source]

Bases: pysummarization.tokenizable_doc.TokenizableDoc

Tokenize string.

Japanese morphological analysis with MeCab.

tokenize(sentence_str)[source]

Tokenize str.

Parameters:sentence_str – tokenized string.
Returns:[token, token, token, …]

pysummarization.tokenizabledoc.simple_tokenizer module

class pysummarization.tokenizabledoc.simple_tokenizer.SimpleTokenizer[source]

Bases: pysummarization.tokenizable_doc.TokenizableDoc

Tokenize delimited sentence with a blank.

tokenize(sentence_str)[source]

Tokenize str.

Parameters:sentence_str – tokenized string.
Returns:[token, token, token, …]

Module contents