pysummarization.tokenizabledoc package

Submodules

pysummarization.tokenizabledoc.mecab_tokenizer module

class pysummarization.tokenizabledoc.mecab_tokenizer.MeCabTokenizer[source]

Bases: pysummarization.tokenizable_doc.TokenizableDoc

Tokenize string.

Japanese morphological analysis with MeCab.

get_part_of_speech()[source]

getter

part_of_speech

getter

set_part_of_speech(value)[source]

setter

tokenize(sentence_str)[source]

Tokenize str.

Parameters:sentence_str – tokenized string.
Returns:[token, token, token, …]

pysummarization.tokenizabledoc.simple_tokenizer module

class pysummarization.tokenizabledoc.simple_tokenizer.SimpleTokenizer[source]

Bases: pysummarization.tokenizable_doc.TokenizableDoc

Tokenize delimited sentence with a blank.

tokenize(sentence_str)[source]

Tokenize str.

Parameters:sentence_str – tokenized string.
Returns:[token, token, token, …]

Module contents