tokenizer-ruby 0.2.0
Fast tokenization for Ruby using HuggingFace's Rust-powered tokenizers library. Supports GPT, BERT, LLaMA, Claude, and any HuggingFace tokenizer.
Gemfile:
=
install:
=
Runtime Dependencies (1):
rb_sys
~> 0.9