tokenkit 0.1.0.pre.1
TokenKit provides lightweight, Unicode-aware word-level tokenization with pattern preservation, backed by Rust for performance.
          Gemfile:
          =
        
        
          install:
          =
        
      Versions:
- 0.1.0.pre.1 September 29, 2025 (38.5 KB)
Runtime Dependencies (1):
            rb_sys
            ~> 0.9
          
    