[DEPRECATED] switch to 'j_r_r_token'. RuToken is a Ruby gem that wraps the tiktoken Rust library, enabling fast and efficient tokenization for OpenAI models. It supports multiple models including o200k_base, cl100k_base, p50k_base, and r50k_base.

Required Ruby Version

>= 2.7.0

Required Rubygems Version

>= 3.0.0

Authors

Logan Bresnahan

Versions

  1. 0.1.23

    1. source

      1. 0.1.23 source August 03, 2025 (13.5 KB)
  2. 0.1.22

    1. source

      1. 0.1.22 source August 03, 2025 (13.5 KB)
  3. 0.1.21

    1. source

      1. 0.1.21 source August 01, 2025 (13.5 KB)
  4. 0.1.20

    1. source

      1. 0.1.20 source August 01, 2025 (13.5 KB)

    2. aarch64-linux

      1. 0.1.20 multi-abi August 01, 2025 (24.3 MB)

    3. aarch64-linux-musl

      1. 0.1.20 multi-abi August 01, 2025 (24.3 MB)

    4. arm64-darwin

      1. 0.1.20 multi-abi August 01, 2025 (23.9 MB)

    5. arm-linux

      1. 0.1.20 multi-abi August 01, 2025 (24.2 MB)

    6. x86_64-darwin

      1. 0.1.20 multi-abi August 01, 2025 (23.9 MB)

    7. x86_64-linux

      1. 0.1.20 multi-abi August 01, 2025 (24.6 MB)

    8. x86_64-linux-musl

      1. 0.1.20 multi-abi August 01, 2025 (24.6 MB)
  5. 0.1.19

    1. source

      1. 0.1.19 source July 31, 2025 (13.5 KB)

    2. arm-linux

      1. 0.1.19 multi-abi July 31, 2025 (24.2 MB)

    3. x86_64-linux

      1. 0.1.19 multi-abi July 31, 2025 (24.6 MB)

    4. x86_64-linux-musl

      1. 0.1.19 multi-abi July 31, 2025 (24.6 MB)
Show all versions (100 total)

SHA 256 checksum

← Previous version