An unofficial Ruby wrapper for Tiktoken, a BPE tokenizer written by and used by OpenAI. It can be used to count the number of tokens in text before sending it to OpenAI APIs. This is a fork of tiktoken_ruby by IAPark, which has been cross-compiled for multiple platforms. This way compilation with Rust extensions doesn't need to happen wherever you are deploying it.

Required Ruby Version

>= 2.7.0

Required Rubygems Version

>= 3.1.0

Authors

IAPark, judy

Versions

  1. 0.0.6

    1. source

      1. 0.0.6 source August 24, 2023 (13.5 KB)

    2. aarch64-linux

      1. 0.0.6 multi-abi August 24, 2023 (9.42 MB)

    3. arm64-darwin

      1. 0.0.6 multi-abi August 24, 2023 (6.77 MB)

    4. arm-linux

      1. 0.0.6 multi-abi August 24, 2023 (9.39 MB)

    5. x64-mingw-ucrt

      1. 0.0.6 multi-abi August 24, 2023 (7.95 MB)

    6. x86_64-darwin

      1. 0.0.6 multi-abi August 24, 2023 (6.77 MB)

    7. x86_64-linux

      1. 0.0.6 multi-abi August 24, 2023 (9.49 MB)

    8. x86_64-linux-musl

      1. 0.0.6 multi-abi August 24, 2023 (9.48 MB)
Show all versions (17 total)

Pushed by

judy

SHA 256 checksum

← Previous version Next version →