[DEPRECATED] switch to 'j_r_r_token'. RuToken is a Ruby gem that wraps the tiktoken Rust library, enabling fast and efficient tokenization for OpenAI models. It supports multiple models including o200k_base, cl100k_base, p50k_base, and r50k_base.
Required Ruby Version
>= 2.7.0
Required Rubygems Version
>= 3.0.0
Authors
Logan Bresnahan
Versions
- 0.1.23 August 03, 2025 (13.5 KB)
- 0.1.22 August 03, 2025 (13.5 KB)
- 0.1.21 August 01, 2025 (13.5 KB)
- 0.1.20 August 01, 2025 x86_64-linux (24.6 MB)
- 0.1.20 August 01, 2025 arm64-darwin (23.9 MB)
