RubyGems Navigation menu

tokenizer 0.1.1

A simple multilingual tokenizer for NLP tasks. This tool provides a CLI and a library for linguistic tokenization which is an anavoidable step for many HLT (human language technology) tasks in the preprocessing phase for further syntactic, semantic and other higher level processing goals. Use it for tokenization of German, English and French texts.

Versions:

  1. 0.1.1 - August 24, 2011 (8 KB)
  2. 0.1.0 - May 18, 2011 (5.5 KB)
  3. 0.0.1.prealpha - May 5, 2011 (4 KB)

Authors:

  • Andrei Beliankou

Owners:

9183f2f97a2f44594196cab39cbe5928

Sha 256 checksum:

82819cb92efebc52d2c455c928f7c637c9041c9d3fd5bb3527016e18625c3112

Total downloads 6,407

For this version 4,870

Required Ruby Version: None

Licenses:

N/A

Gemfile:
= Copy to clipboard Copied!

install:
= Copy to clipboard Copied!

Links: