tokenizer-ruby 0.2.0

Fast tokenization for Ruby using HuggingFace's Rust-powered tokenizers library. Supports GPT, BERT, LLaMA, Claude, and any HuggingFace tokenizer.

Gemfile:
=

installation:
=

Versions:

0.2.0 March 11, 2026 (10,5 ko)
0.1.1 March 10, 2026 (10,5 ko)
0.1.0 March 07, 2026 (9,5 ko)

Dépendances de Runtime (1):

rb_sys ~> 0.9

Show all transitive dependencies

Propriétaires:

Pushed by:

Auteurs:

Johannes Dwi Cahyo

Total de contrôle SHA 256:

=