discourse_ai-tokenizers 0.3.0

A Ruby gem providing a consistent interface for various AI/ML tokenizers including OpenAI GPT, Anthropic Claude, Google Gemini, Meta Llama, Mistral, Qwen, and embedding models like BERT, BGE, and multilingual-E5. Features caching, truncation, token counting, and error handling across different tokenization libraries.

Gemfile:
=

install:
=

Versions:

0.3.0 July 04, 2025 (20.6 MB)
0.2.0 July 02, 2025 (20.6 MB)
0.1.2 July 02, 2025 (20.6 MB)
0.1.1 July 01, 2025 (20.6 MB)
0.1.0 July 01, 2025 (20.6 MB)

Runtime Dependencies (3):

activesupport >= 6.0

tiktoken_ruby ~> 0.0.11.1

tokenizers ~> 0.5.4

Development Dependencies (2):

rubocop-discourse = 3.8.1

syntax_tree ~> 6.2.0

Show all transitive dependencies

Owners:

Pushed by:

Authors:

Rafael Silva

discourse_ai-tokenizers 0.3.0

Gemfile:
=

install:
=

Versions:

Runtime Dependencies (3):

Development Dependencies (2):

Owners:

Pushed by:

Authors:

SHA 256 checksum:

Total downloads 1,212

For this version 107

Version Released:
July 4, 2025 8:41pm

License:
MIT

Required Ruby Version: >= 3.2.0

Links:

discourse_ai-tokenizers 0.3.0

Gemfile: =

install: =

Versions:

Runtime Dependencies (3):

Development Dependencies (2):

Owners:

Pushed by:

Authors:

SHA 256 checksum:

Total downloads 1,212

For this version 107

Version Released: July 4, 2025 8:41pm

License: MIT

Required Ruby Version: >= 3.2.0

Links:

Gemfile:
=

install:
=

Version Released:
July 4, 2025 8:41pm

License:
MIT