RubyGems Navigation menu

tokenizer 0.3.0

A simple multilingual tokenizer for NLP tasks. This tool provides a CLI and a library for linguistic tokenization which is an anavoidable step for many HLT (human language technology) tasks in the preprocessing phase for further syntactic, semantic and other higher level processing goals. Use it for tokenization of German, English and French texts.

Versions:

  1. 0.3.0 - January 20, 2016 (10 KB)
  2. 0.2.0 - January 11, 2016 (10 KB)
  3. 0.1.2 - September 3, 2015 (9.5 KB)
  4. 0.1.1 - August 24, 2011 (8 KB)
  5. 0.1.0 - May 18, 2011 (5.5 KB)
Show all versions (6 total)

Authors:

  • Andrei Beliankou

Owners:

9183f2f97a2f44594196cab39cbe5928

SHA 256 checksum:

f3fea578e7a2f9a9ae22de432ea1254e22ed1fc2e58e0ee670a0273582f51520

Total downloads 85,660

For this version 1,369

Show all versions (6 total)

Gemfile:
= Copy to clipboard Copied!

install:
=

License:

MIT

Required Ruby Version: >= 1.9.3

Links: