An unofficial Ruby wrapper for Tiktoken, a BPE tokenizer written by and used by OpenAI. It can be used to count the number of tokens in text before sending it to OpenAI APIs.

Required Ruby Version

>= 2.7.0

Required Rubygems Version

>= 3.1.0

Authors

IAPark

Versions

  1. 0.0.17

    1. source

      1. 0.0.17 source July 19, 2026 (16 KB)

    2. aarch64-linux

      1. 0.0.17 multi-abi July 19, 2026 (16.3 MB)

    3. aarch64-linux-musl

      1. 0.0.17 multi-abi July 19, 2026 (16.3 MB)

    4. arm64-darwin

      1. 0.0.17 multi-abi July 19, 2026 (16 MB)

    5. arm-linux

      1. 0.0.17 multi-abi July 19, 2026 (16 MB)

    6. x64-mingw-ucrt

      1. 0.0.17 multi-abi July 19, 2026 (18.5 MB)

    7. x86_64-darwin

      1. 0.0.17 multi-abi July 19, 2026 (15.9 MB)

    8. x86_64-linux

      1. 0.0.17 multi-abi July 19, 2026 (16.4 MB)

    9. x86_64-linux-musl

      1. 0.0.17 multi-abi July 19, 2026 (16.4 MB)
  2. 0.0.16

    1. source

      1. 0.0.16 source May 02, 2026 (15 KB)

    2. aarch64-linux

      1. 0.0.16 multi-abi May 02, 2026 (16.2 MB)

    3. aarch64-linux-musl

      1. 0.0.16 multi-abi May 02, 2026 (16.2 MB)

    4. arm64-darwin

      1. 0.0.16 multi-abi May 02, 2026 (16 MB)

    5. arm-linux

      1. 0.0.16 multi-abi May 02, 2026 (16 MB)

    6. x64-mingw-ucrt

      1. 0.0.16 multi-abi May 02, 2026 (18.6 MB)

    7. x86_64-darwin

      1. 0.0.16 multi-abi May 02, 2026 (16 MB)

    8. x86_64-linux

      1. 0.0.16 multi-abi May 02, 2026 (16.4 MB)

    9. x86_64-linux-musl

      1. 0.0.16 multi-abi May 02, 2026 (16.4 MB)
  3. 0.0.15.1

    1. source

      1. 0.0.15.1 source January 07, 2026 (15 KB)

    2. aarch64-linux

      1. 0.0.15.1 multi-abi January 07, 2026 (16.2 MB)

    3. aarch64-linux-musl

      1. 0.0.15.1 multi-abi January 07, 2026 (16.2 MB)

    4. arm64-darwin

      1. 0.0.15.1 multi-abi January 07, 2026 (15.9 MB)

    5. arm-linux

      1. 0.0.15.1 multi-abi January 07, 2026 (16.1 MB)

    6. x64-mingw-ucrt

      1. 0.0.15.1 multi-abi January 07, 2026 (18.6 MB)

    7. x86_64-darwin

      1. 0.0.15.1 multi-abi January 07, 2026 (15.9 MB)

    8. x86_64-linux

      1. 0.0.15.1 multi-abi January 07, 2026 (16.4 MB)

    9. x86_64-linux-musl

      1. 0.0.15.1 multi-abi January 07, 2026 (16.4 MB)
  4. 0.0.15

    1. source

      1. 0.0.15 source January 06, 2026 (15 KB)

    2. aarch64-linux

      1. 0.0.15 multi-abi January 06, 2026 (16.2 MB)

    3. aarch64-linux-musl

      1. 0.0.15 multi-abi January 06, 2026 (16.2 MB)

    4. arm64-darwin

      1. 0.0.15 multi-abi January 06, 2026 (15.9 MB)

    5. x64-mingw-ucrt

      1. 0.0.15 multi-abi January 06, 2026 (18.6 MB)

    6. x86_64-darwin

      1. 0.0.15 multi-abi January 06, 2026 (15.9 MB)

    7. x86_64-linux-musl

      1. 0.0.15 multi-abi January 06, 2026 (16.4 MB)
  5. 0.0.14.1

    1. source

      1. 0.0.14.1 source December 20, 2025 (14.5 KB)

    2. aarch64-linux

      1. 0.0.14.1 multi-abi December 20, 2025 (12.2 MB)

    3. arm64-darwin

      1. 0.0.14.1 multi-abi December 20, 2025 (11.9 MB)

    4. x64-mingw-ucrt

      1. 0.0.14.1 multi-abi December 20, 2025 (13.9 MB)

    5. x86_64-darwin

      1. 0.0.14.1 multi-abi December 20, 2025 (11.9 MB)

    6. x86_64-linux

      1. 0.0.14.1 multi-abi December 20, 2025 (12.3 MB)
  6. 0.0.5

    1. source

      1. 0.0.5 source May 11, 2023 (14.5 KB)

    2. aarch64-linux

      1. 0.0.5 multi-abi May 11, 2023 (8.75 MB)

    3. arm64-darwin

      1. 0.0.5 multi-abi May 11, 2023 (6.27 MB)

    4. arm-linux

      1. 0.0.5 multi-abi May 11, 2023 (8.66 MB)

    5. x64-mingw32

      1. 0.0.5 multi-abi May 11, 2023 (3.73 MB)

    6. x64-mingw-ucrt

      1. 0.0.5 multi-abi May 11, 2023 (7.47 MB)

    7. x86_64-darwin

      1. 0.0.5 multi-abi May 11, 2023 (6.29 MB)

    8. x86_64-linux

      1. 0.0.5 multi-abi May 11, 2023 (8.87 MB)

    9. x86_64-linux-musl

      1. 0.0.5 multi-abi May 11, 2023 (8.86 MB)
Show all versions (157 total)

Pushed by

iapark

SHA 256 checksum

← Previous version Next version →