1.1 KiB
1.1 KiB
RLTK
An attempt to manually port some of nltk to rust.
from https://www.nltk.org/api/nltk.lm.html:
So as to avoid re-creating the text in memory, both train and vocab are lazy iterators. They are evaluated on demand at training time.
rltk has the same philosophy: everything is done using iterators (on iterators) on string slices.
Currently in it's infancy (but growing):
- rltk::lm::preprocessing::pad_both_ends(["a","b","c"], 2) -> "<s>", "a", "b", "c", "</s>"]
- rltk::util::pad_sequence == same as above with customisation
- rltk::util::pad_sequence_left == same
- rltk::util::pad_sequence_right == same
- rltk::util::ngrams(["a","b","c"],2) -> "a", "b"], ["b", "c"
- rltk::util::bigrams(["a","b","c"]) == ngrams(..., 2)
- rltk::util::trigrams(["a","b","c"]) == ngrams(..., 3)
- rltk::util::everygrams(["a","b","c"],2) == "a"], ["a", "b"], ["b"], ["b", "c"
- rltk::util::flatten("a"], ["a", "b"], ["b"], ["b", "c") == ["a", "a", "b", "b", "b", "c"]
- rltk::metrics::distance::edit_distance(): calculate the levenshtein distance between two words (see doc)