Pure-Ruby facade over Hugging Face `tokenizers`, OpenAI `tiktoken_ruby`, and SentencePiece bindings that maps ruby_llm model identifiers (gpt-4o, llama-3, mistral, ...) to the correct tokenizer and exposes a small API for counting, analyzing, and truncating text against a model's context window. Includes an opt-in approximation backend for models with no published tokenizer (Claude).

Required Ruby Version

>= 3.3.0

Authors

Sal Scotto

Versions

  1. 0.1.2

    1. source

      1. 0.1.2 source June 13, 2026 (16.5 KB)
  2. 0.1.1

    1. source

      1. 0.1.1 source June 11, 2026 (16 KB)
  3. 0.1.0

    1. source

      1. 0.1.0 source June 05, 2026 (14.5 KB)

Pushed by

washu214

SHA 256 checksum

← Previous version