tokenkit 0.1.0.pre.1
TokenKit provides lightweight, Unicode-aware word-level tokenization with pattern preservation, backed by Rust for performance.
Gemfile:
=
install:
=
Versions:
- 0.1.0.pre.1 September 29, 2025 (38.5 KB)
Runtime Dependencies (1):
rb_sys
~> 0.9