Fast tokenization for Ruby using HuggingFace's Rust-powered tokenizers library. Supports GPT, BERT, LLaMA, Claude, and any HuggingFace tokenizer.

Required Ruby Version

>= 3.1

Authors

Johannes Dwi Cahyo

Versions

  1. 0.2.0

    1. source

      1. 0.2.0 source March 11, 2026 (10.5 KB)
  2. 0.1.1

    1. source

      1. 0.1.1 source March 10, 2026 (10.5 KB)
  3. 0.1.0

    1. source

      1. 0.1.0 source March 07, 2026 (9.5 KB)

SHA 256 checksum

← Previous version