discourse_ai-tokenizers 0.2.0

A Ruby gem providing a consistent interface for various AI/ML tokenizers including OpenAI GPT, Anthropic Claude, Google Gemini, Meta Llama, Mistral, Qwen, and embedding models like BERT, BGE, and multilingual-E5. Features caching, truncation, token counting, and error handling across different tokenization libraries.

Gemfile:
=

install:
=

Versions:

0.2.0 July 02, 2025 (20.6 MB)
0.1.2 July 02, 2025 (20.6 MB)
0.1.1 July 01, 2025 (20.6 MB)
0.1.0 July 01, 2025 (20.6 MB)

Runtime Dependencies (3):

activesupport >= 6.0

tiktoken_ruby ~> 0.0.11.1

tokenizers ~> 0.5.4

Development Dependencies (2):

rubocop-discourse = 3.8.1

syntax_tree ~> 6.2.0

Show all transitive dependencies

Owners:

Pushed by:

Authors:

Rafael Silva

discourse_ai-tokenizers 0.2.0

Gemfile:
=

install:
=

Versions:

Runtime Dependencies (3):

Development Dependencies (2):

Owners:

Pushed by:

Authors:

SHA 256 checksum:

Total downloads 899

For this version 581

Version Released:
July 2, 2025 5:02pm

License:
MIT

Required Ruby Version: >= 3.2.0

Links:

discourse_ai-tokenizers 0.2.0

Gemfile: =

install: =

Versions:

Runtime Dependencies (3):

Development Dependencies (2):

Owners:

Pushed by:

Authors:

SHA 256 checksum:

Total downloads 899

For this version 581

Version Released: July 2, 2025 5:02pm

License: MIT

Required Ruby Version: >= 3.2.0

Links:

Gemfile:
=

install:
=

Version Released:
July 2, 2025 5:02pm

License:
MIT