Nine JSONL stages from a raw web crawl to dispersion-corrected frequency and n-gram tables: source gating, Taiwan-standard punctuation normalization, sentence segmentation, character-inventory and lexical-origin filtering, block-level evidence judgment, type deduplication, Viterbi word segmentation and external-sort counting. Resource-bounded by declared cores and memory.

Required Ruby Version

>= 3.4

Authors

Den Patin

Versions

  1. 0.1.0 August 03, 2026 (27 KB)

Pushed by

SHA 256 checksum