A Ruby gem providing a consistent interface for various AI/ML tokenizers including OpenAI GPT, Anthropic Claude, Google Gemini, Meta Llama, Mistral, Qwen, and embedding models like BERT, BGE, and multilingual-E5. Features caching, truncation, token counting, and error handling across different tokenization libraries.

Required Ruby Version

>= 3.2.0

Authors

Rafael Silva

Versions

  1. 0.4.2

    1. source

      1. 0.4.2 source February 27, 2026 (20.6 MB)
  2. 0.4.1

    1. source

      1. 0.4.1 source February 26, 2026 (20.6 MB)
  3. 0.4

    1. source

      1. 0.4 source January 06, 2026 (20.6 MB)
  4. 0.3.2

    1. source

      1. 0.3.2 source December 10, 2025 (20.6 MB)
  5. 0.3.1

    1. source

      1. 0.3.1 source July 07, 2025 (20.6 MB)
Show all versions (10 total)

SHA 256 checksum

← Previous version