RubyGems Navigation menu

tokenkit 0.1.0.pre.1

TokenKit provides lightweight, Unicode-aware word-level tokenization with pattern preservation, backed by Rust for performance.

Gemfile:
=

install:
=

Versions:

  1. 0.1.0.pre.1 September 29, 2025 (38.5 KB)

Runtime Dependencies (1):

rb_sys ~> 0.9

Development Dependencies (5):

rake ~> 13.0
rake-compiler ~> 1.2
rspec ~> 3.0
simplecov ~> 0.22
standard ~> 1.3

Owners:

Pushed by:

Authors:

  • Chris Petersen

SHA 256 checksum:

=

Total downloads 325

For this version 325

Version Released:

License:

MIT

Required Ruby Version: >= 3.1.0

Links: