TokenKit provides lightweight, Unicode-aware word-level tokenization with pattern preservation, backed by Rust for performance.

Required Ruby Version

>= 3.1, < 3.5.dev

Authors

Chris Petersen

Versions

  1. 0.1.0.pre.2

    1. source

      1. 0.1.0.pre.2 source June 20, 2026 (885 KB)

    2. aarch64-linux

      1. 0.1.0.pre.2 multi-abi June 20, 2026 (3.88 MB)

    3. arm64-darwin

      1. 0.1.0.pre.2 multi-abi June 20, 2026 (3.62 MB)

    4. x86_64-linux

      1. 0.1.0.pre.2 multi-abi June 20, 2026 (4.05 MB)
  2. 0.1.0.pre.1

    1. source

      1. 0.1.0.pre.1 source September 29, 2025 (38.5 KB)

Requirements

Rust >= 1.85

Pushed by

cpetersen

SHA 256 checksum

← Previous version Next version →