huggingface / tokenizers
File Age & Freshness

File age measurements show the distribution of file ages (days since the first commit) and the file freshness (days since the latest commit).

Summary
File Change History Overall
File Age Distribution Overall
Days since first update
  • There are 146 files with 29,051 lines of code in files.
    • 142 files that are 366+ days old (28,258 lines of code)
    • 2 files that are 181-365 days old (618 lines of code)
    • 0 files that are 91-180 days old (0 lines of code)
    • 0 files that are 31-90 days old (0 lines of code)
    • 2 files that are 1-30 days old (175 lines of code)
97% | 2% | 0% | 0% | <1%
Legend:
366+
181-365
91-180
31-90
1-30

explore: grouped by folders | grouped by age
File Freshness Distribution Overall
Days since last update
  • There are 146 files with 29,051 lines of code in files.
    • 52 files have been last changed 366+ days ago (3,822 lines of code)
    • 19 files have been last changed 181-365 days ago (4,526 lines of code)
    • 17 files have been last changed 91-180 days ago (3,319 lines of code)
    • 12 files have been last changed 31-90 days ago (3,888 lines of code)
    • 46 files have been last changed 1-30 days ago (13,496 lines of code)
13% | 15% | 11% | 13% | 46%
Legend:
366+
181-365
91-180
31-90
1-30

explore: grouped by folders | grouped by freshness
File Change History per File Extension
rs, py, md, yaml, mdx, json, ts, toml, pyi, js, rst, inc, gitignore, txt, css, gitattributes, bat, sh, editorconfig, html, ipynb, in, cfg, cjs, tpl, prettierignore
File Age Distribution per Extension
Days since first update
366+
181-365
91-180
31-90
1-30
rs96% | 2% | 0% | 0% | <1%
py100% | 0% | 0% | 0% | 0%
pyi100% | 0% | 0% | 0% | 0%
js100% | 0% | 0% | 0% | 0%
toml100% | 0% | 0% | 0% | 0%
css100% | 0% | 0% | 0% | 0%
cfg100% | 0% | 0% | 0% | 0%
in100% | 0% | 0% | 0% | 0%
ts100% | 0% | 0% | 0% | 0%
File Freshness Distribution per Extension
Days since last update
366+
181-365
91-180
31-90
1-30
rs8% | 16% | 10% | 12% | 51%
py42% | 1% | 19% | 35% | <1%
js100% | 0% | 0% | 0% | 0%
pyi25% | 35% | 16% | 8% | 14%
css100% | 0% | 0% | 0% | 0%
cfg100% | 0% | 0% | 0% | 0%
in100% | 0% | 0% | 0% | 0%
ts100% | 0% | 0% | 0% | 0%
toml<1% | 0% | 32% | 0% | 67%
File Change History per Logical Decomposition
primary
primary (file age distribution)
Days since first update
366+
181-365
91-180
31-90
1-30
tokenizers97% | <1% | 0% | 0% | 1%
bindings96% | 3% | 0% | 0% | 0%
primary (file freshness distribution)
Days since last update
366+
181-365
91-180
31-90
1-30
bindings22% | 30% | 19% | 10% | 16%
tokenizers5% | 4% | 5% | 15% | 68%
Oldest Files (Top 50)
File# lines# unitscreatedlast modified# changes
(days)		# contributorsfirst
contributor		latest
contributor
Cargo.toml
in tokenizers
 82 - 2019-11-01 2025-06-24 109 21 m.anthony.moi@gmail.com arthur.zucker@gmail.com
lib.rs
in bindings/python/src
 39 - 2019-11-01 2025-01-09 62 8 m.anthony.moi@gmail.com 13696594+tinyboxvk@users.no...
Cargo.toml
in bindings/python
 26 - 2019-11-01 2025-06-24 109 17 m.anthony.moi@gmail.com arthur.zucker@gmail.com
lib.rs
in tokenizers/src
 20 - 2019-11-01 2025-06-17 43 11 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
model.rs
in tokenizers/src/models/bpe
 834 26 2019-11-17 2025-06-19 69 15 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
trainer.rs
in tokenizers/src/models/bpe
 598 15 2019-11-17 2025-06-19 54 12 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
byte_level.rs
in tokenizers/src/pre_tokenizers
 513 18 2019-11-17 2025-06-19 71 12 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
word.rs
in tokenizers/src/models/bpe
 277 6 2019-11-17 2025-06-19 18 6 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
mod.rs
in tokenizers/src/pre_tokenizers
 258 2 2019-11-17 2025-06-19 20 8 m.anthony.moi@gmail.com vetterle@posteo.de
mod.rs
in tokenizers/src/processors
 112 4 2019-11-17 2024-08-06 13 6 m.anthony.moi@gmail.com 48595927+arthurzucker@users...
whitespace.rs
in tokenizers/src/pre_tokenizers
 94 5 2019-11-17 2025-03-18 30 7 m.anthony.moi@gmail.com c@farsight.net
mod.rs
in tokenizers/src/models
 59 2 2019-11-17 2025-06-19 29 10 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
mod.rs
in tokenizers/src/models/bpe
 59 3 2019-11-17 2024-06-24 19 6 m.anthony.moi@gmail.com cognetta.marco@gmail.com
mod.rs
in tokenizers/src/normalizers
 40 - 2019-11-17 2025-01-28 19 7 m.anthony.moi@gmail.com 48595927+arthurzucker@users...
mod.rs
in tokenizers/src/decoders
 39 - 2019-11-21 2024-08-06 15 5 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
tokenizer.rs
in bindings/python/src
 1060 74 2019-11-22 2024-12-31 108 17 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
models.rs
in bindings/python/src
 785 58 2019-11-22 2025-06-19 72 15 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
token.rs
in bindings/python/src
 40 7 2019-11-22 2023-08-16 6 3 m.anthony.moi@gmail.com mikelui@users.noreply.githu...
pre_tokenizers.rs
in bindings/python/src
 791 62 2019-11-23 2025-06-19 76 23 m.anthony.moi@gmail.com vetterle@posteo.de
decoders.rs
in bindings/python/src
 577 53 2019-11-23 2025-05-27 51 15 m.anthony.moi@gmail.com co63oc@users.noreply.github...
pyproject.toml
in bindings/python
 61 - 2019-11-27 2025-02-11 14 7 m.anthony.moi@gmail.com git@qaq.io
trainers.rs
in bindings/python/src
 836 59 2019-12-03 2024-12-31 47 16 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
in
MANIFEST.in
in bindings/python
 7 - 2019-12-03 2020-07-11 3 2 m.anthony.moi@gmail.com setu4993@yahoo.co.in
mod.rs
in tokenizers/src/models/wordpiece
 250 11 2019-12-07 2025-06-19 40 10 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
wordpiece.rs
in tokenizers/src/decoders
 74 3 2019-12-07 2025-06-18 18 6 m.anthony.moi@gmail.com c@farsight.net
encoding.rs
in bindings/python/src
 244 27 2019-12-10 2024-12-31 53 13 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
mod.rs
in tokenizers/src/tokenizer
 1185 31 2019-12-12 2025-06-24 137 21 m.anthony.moi@gmail.com 48595927+arthurzucker@users...
encoding.rs
in tokenizers/src/tokenizer
 780 10 2019-12-12 2025-06-19 57 12 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
bert.rs
in tokenizers/src/processors
 268 5 2019-12-12 2025-06-19 31 6 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
error.rs
in bindings/python/src
 35 2 2019-12-13 2024-12-31 13 5 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
processors.rs
in bindings/python/src
 642 48 2019-12-16 2025-01-28 43 15 m.anthony.moi@gmail.com 48595927+arthurzucker@users...
bert.rs
in tokenizers/src/pre_tokenizers
 76 4 2019-12-17 2022-08-24 21 5 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
normalizer.rs
in tokenizers/src/tokenizer
 2002 33 2019-12-28 2025-05-27 52 15 m.anthony.moi@gmail.com co63oc@users.noreply.github...
normalizers.rs
in bindings/python/src
 760 58 2019-12-29 2025-01-28 52 17 m.anthony.moi@gmail.com 48595927+arthurzucker@users...
bert.rs
in tokenizers/src/normalizers
 102 9 2019-12-29 2023-08-28 18 5 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
bpe_benchmark.rs
in tokenizers/benches
 115 4 2020-01-01 2025-06-17 27 5 epwalsh10@gmail.com patry.nicolas@protonmail.com
trainer.rs
in tokenizers/src/models/wordpiece
 147 4 2020-01-03 2025-06-19 19 8 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
metaspace.rs
in tokenizers/src/pre_tokenizers
 331 10 2020-01-07 2025-01-28 31 7 m.anthony.moi@gmail.com 48595927+arthurzucker@users...
unicode.rs
in tokenizers/src/normalizers
 91 7 2020-01-07 2022-01-26 10 4 m.anthony.moi@gmail.com dmishig@gmail.com
utils.rs
in tokenizers/src/normalizers
 48 5 2020-01-07 2025-01-28 13 6 m.anthony.moi@gmail.com 48595927+arthurzucker@users...
sentencepiece_extractor.py
in bindings/python/scripts
 93 4 2020-01-08 2024-03-12 5 5 m.anthony.moi@gmail.com 48595927+arthurzucker@users...
bpe.rs
in tokenizers/src/decoders
 31 2 2020-01-08 2022-06-02 11 4 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
roberta.rs
in tokenizers/src/processors
 309 5 2020-02-03 2025-06-19 30 7 mfuntowicz@users.noreply.gi... patry.nicolas@protonmail.com
delimiter.rs
in tokenizers/src/pre_tokenizers
 21 1 2020-02-04 2022-08-24 15 6 mfuntowicz@users.noreply.gi... patry.nicolas@protonmail.com
mod.rs
in tokenizers/src/models/wordlevel
 204 12 2020-02-05 2025-06-19 29 9 mfuntowicz@users.noreply.gi... patry.nicolas@protonmail.com
strip.rs
in tokenizers/src/normalizers
 133 6 2020-02-17 2022-01-26 10 5 mfuntowicz@users.noreply.gi... dmishig@gmail.com
truncation.rs
in tokenizers/src/utils
 279 12 2020-03-15 2025-06-19 21 7 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
padding.rs
in tokenizers/src/utils
 123 3 2020-03-15 2025-06-19 12 2 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
mod.rs
in tokenizers/src/utils
 112 3 2020-03-15 2025-06-19 21 8 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
iter.rs
in tokenizers/src/utils
 71 6 2020-03-15 2021-12-17 4 2 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
Files Not Recently Changed (Top 50)
File# lines# unitscreatedlast modified# changes
(days)		# contributorsfirst
contributor		latest
contributor
in
MANIFEST.in
in bindings/python
 7 - 2019-12-03 2020-07-11 3 2 m.anthony.moi@gmail.com setu4993@yahoo.co.in
conftest.py
in bindings/python
 12 3 2020-09-02 2020-09-02 1 1 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
mod.rs
in tokenizers/src/models/unigram
 8 - 2020-06-06 2020-09-15 7 1 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
mod.rs
in tokenizers/src/pre_tokenizers/unicode_scripts
 3 - 2020-10-20 2020-10-20 1 1 m.anthony.moi@gmail.com m.anthony.moi@gmail.com
__init__.py
in bindings/python/py_src/tokenizers/trainers
 6 - 2020-07-25 2020-11-17 5 3 sebastian.puetz@uni-tuebing... patry.nicolas@protonmail.com
__init__.py
in bindings/python/py_src/tokenizers/models
 6 - 2020-07-25 2020-11-17 5 3 sebastian.puetz@uni-tuebing... patry.nicolas@protonmail.com
visualizer-styles.css
in bindings/python/py_src/tokenizers/tools
 138 - 2020-12-04 2020-12-04 1 1 tal@lighttag.io tal@lighttag.io
iter.rs
in tokenizers/src/utils
 71 6 2020-03-15 2021-12-17 4 2 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
unicode.rs
in tokenizers/src/normalizers
 91 7 2020-01-07 2022-01-26 10 4 m.anthony.moi@gmail.com dmishig@gmail.com
strip.rs
in tokenizers/src/normalizers
 133 6 2020-02-17 2022-01-26 10 5 mfuntowicz@users.noreply.gi... dmishig@gmail.com
bpe.rs
in tokenizers/src/decoders
 31 2 2020-01-08 2022-06-02 11 4 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
onig.rs
in tokenizers/src/utils
 15 - 2022-06-10 2022-06-10 1 1 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
delimiter.rs
in tokenizers/src/pre_tokenizers
 21 1 2020-02-04 2022-08-24 15 6 mfuntowicz@users.noreply.gi... patry.nicolas@protonmail.com
bert.rs
in tokenizers/src/pre_tokenizers
 76 4 2019-12-17 2022-08-24 21 5 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
digits.rs
in tokenizers/src/pre_tokenizers
 93 4 2020-09-03 2022-08-24 8 3 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
__init__.py
in bindings/python/py_src/tokenizers/processors
 7 - 2020-07-25 2022-08-25 5 3 sebastian.puetz@uni-tuebing... patry.nicolas@protonmail.com
__init__.py
in bindings/python/py_src/tokenizers/tools
 1 - 2020-12-04 2022-10-05 2 2 tal@lighttag.io patry.nicolas@protonmail.com
__init__.py
in bindings/python/py_src/tokenizers/implementations
 6 - 2020-07-25 2022-10-05 4 3 sebastian.puetz@uni-tuebing... patry.nicolas@protonmail.com
cfg
setup.cfg
in bindings/python
 51 - 2022-10-05 2022-10-05 1 1 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
bert_wordpiece.py
in bindings/python/py_src/tokenizers/implementations
 131 4 2020-07-25 2023-03-08 9 6 sebastian.puetz@uni-tuebing... kurttutan.mert@gmail.com
fuse.rs
in tokenizers/src/decoders
 34 2 2023-03-23 2023-03-23 1 1 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
prepend.rs
in tokenizers/src/normalizers
 56 2 2023-03-23 2023-03-23 1 1 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
pre_tokenizer.rs
in tokenizers/src/pre_tokenizers/unicode_scripts
 132 6 2020-09-15 2023-07-16 7 3 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
token.rs
in bindings/python/src
 40 7 2019-11-22 2023-08-16 6 3 m.anthony.moi@gmail.com mikelui@users.noreply.githu...
rustfmt.toml
in bindings/node
 1 - 2023-08-28 2023-08-28 1 1 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
mod.rs
in bindings/node/src/tasks
 2 - 2023-08-28 2023-08-28 1 1 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
build.rs
in bindings/node
 4 1 2023-08-28 2023-08-28 1 1 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
types.ts
in bindings/node
 6 - 2023-08-28 2023-08-28 1 1 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
lib.rs
in bindings/node/src
 13 - 2023-08-28 2023-08-28 1 1 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
arc_rwlock_serde.rs
in bindings/node/src
 15 - 2023-08-28 2023-08-28 1 1 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
utils.rs
in bindings/node/src
 42 - 2023-08-28 2023-08-28 1 1 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
trainers.rs
in bindings/node/src
 59 4 2023-08-28 2023-08-28 1 1 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
models.rs
in bindings/node/src/tasks
 67 6 2023-08-28 2023-08-28 1 1 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
bert.rs
in tokenizers/src/normalizers
 102 9 2019-12-29 2023-08-28 18 5 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
processors.rs
in bindings/node/src
 110 2 2023-08-28 2023-08-28 1 1 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
tokenizer.rs
in bindings/node/src/tasks
 111 8 2023-08-28 2023-08-28 1 1 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
normalizers.rs
in bindings/node/src
 159 1 2023-08-28 2023-08-28 1 1 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
encoding.rs
in bindings/node/src
 310 5 2023-08-28 2023-08-28 1 1 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
__init__.py
in bindings/python/py_src/tokenizers
 52 - 2020-07-25 2023-09-05 39 8 sebastian.puetz@uni-tuebing... arthur.zucker@gmail.com
base_tokenizer.py
in bindings/python/py_src/tokenizers/implementations
 131 35 2020-07-25 2023-09-07 10 6 sebastian.puetz@uni-tuebing... 48595927+arthurzucker@users...
progress.rs
in tokenizers/src/utils
 12 - 2020-11-11 2023-10-05 2 1 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
pyi
__init__.pyi
in bindings/python/py_src/tokenizers/trainers
 28 2 2020-07-25 2024-03-12 12 6 sebastian.puetz@uni-tuebing... 48595927+arthurzucker@users...
pyi
__init__.pyi
in bindings/python/py_src/tokenizers/processors
 40 17 2020-07-25 2024-03-12 12 4 sebastian.puetz@uni-tuebing... 48595927+arthurzucker@users...
sentencepiece_extractor.py
in bindings/python/scripts
 93 4 2020-01-08 2024-03-12 5 5 m.anthony.moi@gmail.com 48595927+arthurzucker@users...
spm_parity_check.py
in bindings/python/scripts
 208 5 2020-08-21 2024-03-12 9 3 patry.nicolas@protonmail.com 48595927+arthurzucker@users...
sequence.rs
in tokenizers/src/decoders
 48 2 2022-06-02 2024-03-29 2 2 patry.nicolas@protonmail.com e.a.platanios@gmail.com
decoders.rs
in bindings/node/src
 161 1 2023-08-28 2024-03-30 2 2 patry.nicolas@protonmail.com 48595927+arthurzucker@users...
pre_tokenizers.rs
in bindings/node/src
 231 3 2023-08-28 2024-03-30 2 2 patry.nicolas@protonmail.com 48595927+arthurzucker@users...
index.js
in bindings/node
 376 1 2023-08-28 2024-03-30 2 2 patry.nicolas@protonmail.com 48595927+arthurzucker@users...
sentencepiece_unigram.py
in bindings/python/py_src/tokenizers/implementations
 120 4 2020-08-21 2024-04-17 22 8 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
Most Recently Created Files (Top 50)
File# lines# unitscreatedlast modified# changes
(days)		# contributorsfirst
contributor		latest
contributor
fixed_length.rs
in tokenizers/src/pre_tokenizers
 111 5 2025-06-19 2025-06-19 1 1 vetterle@posteo.de vetterle@posteo.de
llama3_benchmark.rs
in tokenizers/benches
 64 - 2025-06-17 2025-06-17 1 1 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
serde_pyo3.rs
in bindings/python/src/utils
 454 44 2024-08-07 2024-12-31 2 1 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
byte_level.rs
in tokenizers/src/normalizers
 164 3 2024-07-15 2025-06-19 6 3 48595927+arthurzucker@users... patry.nicolas@protonmail.com
tokenizer.rs
in bindings/node/src
 463 6 2023-08-28 2024-08-07 2 2 patry.nicolas@protonmail.com 48595927+arthurzucker@users...
index.js
in bindings/node
 376 1 2023-08-28 2024-03-30 2 2 patry.nicolas@protonmail.com 48595927+arthurzucker@users...
encoding.rs
in bindings/node/src
 310 5 2023-08-28 2023-08-28 1 1 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
models.rs
in bindings/node/src
 303 11 2023-08-28 2025-06-19 2 1 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
pre_tokenizers.rs
in bindings/node/src
 231 3 2023-08-28 2024-03-30 2 2 patry.nicolas@protonmail.com 48595927+arthurzucker@users...
decoders.rs
in bindings/node/src
 161 1 2023-08-28 2024-03-30 2 2 patry.nicolas@protonmail.com 48595927+arthurzucker@users...
normalizers.rs
in bindings/node/src
 159 1 2023-08-28 2023-08-28 1 1 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
tokenizer.rs
in bindings/node/src/tasks
 111 8 2023-08-28 2023-08-28 1 1 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
processors.rs
in bindings/node/src
 110 2 2023-08-28 2023-08-28 1 1 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
models.rs
in bindings/node/src/tasks
 67 6 2023-08-28 2023-08-28 1 1 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
trainers.rs
in bindings/node/src
 59 4 2023-08-28 2023-08-28 1 1 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
utils.rs
in bindings/node/src
 42 - 2023-08-28 2023-08-28 1 1 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
Cargo.toml
in bindings/node
 18 - 2023-08-28 2025-06-24 16 3 patry.nicolas@protonmail.com arthur.zucker@gmail.com
arc_rwlock_serde.rs
in bindings/node/src
 15 - 2023-08-28 2023-08-28 1 1 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
lib.rs
in bindings/node/src
 13 - 2023-08-28 2023-08-28 1 1 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
types.ts
in bindings/node
 6 - 2023-08-28 2023-08-28 1 1 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
build.rs
in bindings/node
 4 1 2023-08-28 2023-08-28 1 1 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
mod.rs
in bindings/node/src/tasks
 2 - 2023-08-28 2023-08-28 1 1 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
rustfmt.toml
in bindings/node
 1 - 2023-08-28 2023-08-28 1 1 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
unigram_benchmark.rs
in tokenizers/benches
 37 1 2023-05-22 2025-06-17 3 3 dmishig@gmail.com patry.nicolas@protonmail.com
byte_fallback.rs
in tokenizers/src/decoders
 91 2 2023-03-23 2025-05-27 3 3 patry.nicolas@protonmail.com 48595927+arthurzucker@users...
strip.rs
in tokenizers/src/decoders
 67 2 2023-03-23 2025-01-09 3 2 patry.nicolas@protonmail.com 13696594+tinyboxvk@users.no...
prepend.rs
in tokenizers/src/normalizers
 56 2 2023-03-23 2023-03-23 1 1 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
fuse.rs
in tokenizers/src/decoders
 34 2 2023-03-23 2023-03-23 1 1 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
cfg
setup.cfg
in bindings/python
 51 - 2022-10-05 2022-10-05 1 1 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
sequence.rs
in tokenizers/src/processors
 156 6 2022-08-25 2025-06-19 4 2 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
layout_benchmark.rs
in tokenizers/benches
 64 1 2022-08-25 2025-06-17 2 1 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
fancy.rs
in tokenizers/src/utils
 55 2 2022-06-10 2024-12-31 3 1 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
onig.rs
in tokenizers/src/utils
 15 - 2022-06-10 2022-06-10 1 1 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
sequence.rs
in tokenizers/src/decoders
 48 2 2022-06-02 2024-03-29 2 2 patry.nicolas@protonmail.com e.a.platanios@gmail.com
from_pretrained.rs
in tokenizers/src/utils
 58 1 2021-08-19 2025-05-27 7 6 m.anthony.moi@gmail.com 48595927+arthurzucker@users...
iterators.rs
in bindings/python/src/utils
 90 3 2020-12-15 2024-12-31 6 4 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
visualizer.py
in bindings/python/py_src/tokenizers/tools
 209 13 2020-12-04 2025-01-09 4 4 tal@lighttag.io 13696594+tinyboxvk@users.no...
visualizer-styles.css
in bindings/python/py_src/tokenizers/tools
 138 - 2020-12-04 2020-12-04 1 1 tal@lighttag.io tal@lighttag.io
__init__.py
in bindings/python/py_src/tokenizers/tools
 1 - 2020-12-04 2022-10-05 2 2 tal@lighttag.io patry.nicolas@protonmail.com
split.rs
in tokenizers/src/pre_tokenizers
 33 2 2020-11-27 2025-01-28 8 5 patrick.v.platen@gmail.com 48595927+arthurzucker@users...
stub.py
in bindings/python
 141 9 2020-11-17 2025-01-09 5 4 patry.nicolas@protonmail.com 13696594+tinyboxvk@users.no...
progress.rs
in tokenizers/src/utils
 12 - 2020-11-11 2023-10-05 2 1 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
mod.rs
in tokenizers/src/pre_tokenizers/unicode_scripts
 3 - 2020-10-20 2020-10-20 1 1 m.anthony.moi@gmail.com m.anthony.moi@gmail.com
trainer.rs
in tokenizers/src/models/wordlevel
 151 6 2020-10-07 2025-06-19 12 5 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
normalization.rs
in bindings/python/src/utils
 477 54 2020-09-18 2025-01-28 19 10 m.anthony.moi@gmail.com 48595927+arthurzucker@users...
pretokenization.rs
in bindings/python/src/utils
 254 18 2020-09-18 2024-11-15 11 7 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
mod.rs
in bindings/python/src/utils
 62 3 2020-09-18 2024-10-25 8 4 m.anthony.moi@gmail.com 92270426+sftse@users.norepl...
regex.rs
in bindings/python/src/utils
 20 1 2020-09-18 2024-08-01 6 4 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
convert.py
in bindings/python/scripts
 342 32 2020-09-17 2025-05-27 11 6 patry.nicolas@protonmail.com co63oc@users.noreply.github...
replace.rs
in tokenizers/src/normalizers
 131 11 2020-09-16 2025-01-28 9 5 patry.nicolas@protonmail.com 48595927+arthurzucker@users...
Most Recently Changed Files (Top 50)
File# lines# unitscreatedlast modified# changes
(days)		# contributorsfirst
contributor		latest
contributor
mod.rs
in tokenizers/src/tokenizer
 1185 31 2019-12-12 2025-06-24 137 21 m.anthony.moi@gmail.com 48595927+arthurzucker@users...
Cargo.toml
in tokenizers
 82 - 2019-11-01 2025-06-24 109 21 m.anthony.moi@gmail.com arthur.zucker@gmail.com
Cargo.toml
in bindings/python
 26 - 2019-11-01 2025-06-24 109 17 m.anthony.moi@gmail.com arthur.zucker@gmail.com
Cargo.toml
in bindings/node
 18 - 2023-08-28 2025-06-24 16 3 patry.nicolas@protonmail.com arthur.zucker@gmail.com
template.rs
in tokenizers/src/processors
 854 35 2020-09-04 2025-06-19 22 7 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
model.rs
in tokenizers/src/models/bpe
 834 26 2019-11-17 2025-06-19 69 15 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
added_vocabulary.rs
in tokenizers/src/tokenizer
 799 32 2020-06-16 2025-06-19 40 10 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
pre_tokenizers.rs
in bindings/python/src
 791 62 2019-11-23 2025-06-19 76 23 m.anthony.moi@gmail.com vetterle@posteo.de
models.rs
in bindings/python/src
 785 58 2019-11-22 2025-06-19 72 15 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
encoding.rs
in tokenizers/src/tokenizer
 780 10 2019-12-12 2025-06-19 57 12 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
trainer.rs
in tokenizers/src/models/unigram
 685 21 2020-06-06 2025-06-19 39 8 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
trainer.rs
in tokenizers/src/models/bpe
 598 15 2019-11-17 2025-06-19 54 12 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
lattice.rs
in tokenizers/src/models/unigram
 588 13 2020-06-14 2025-06-19 20 4 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
byte_level.rs
in tokenizers/src/pre_tokenizers
 513 18 2019-11-17 2025-06-19 71 12 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
model.rs
in tokenizers/src/models/unigram
 509 18 2020-06-21 2025-06-19 35 8 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
roberta.rs
in tokenizers/src/processors
 309 5 2020-02-03 2025-06-19 30 7 mfuntowicz@users.noreply.gi... patry.nicolas@protonmail.com
models.rs
in bindings/node/src
 303 11 2023-08-28 2025-06-19 2 1 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
truncation.rs
in tokenizers/src/utils
 279 12 2020-03-15 2025-06-19 21 7 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
word.rs
in tokenizers/src/models/bpe
 277 6 2019-11-17 2025-06-19 18 6 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
bert.rs
in tokenizers/src/processors
 268 5 2019-12-12 2025-06-19 31 6 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
mod.rs
in tokenizers/src/pre_tokenizers
 258 2 2019-11-17 2025-06-19 20 8 m.anthony.moi@gmail.com vetterle@posteo.de
mod.rs
in tokenizers/src/models/wordpiece
 250 11 2019-12-07 2025-06-19 40 10 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
serialization.rs
in tokenizers/src/models/bpe
 217 6 2020-04-09 2025-06-19 12 3 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
mod.rs
in tokenizers/src/models/wordlevel
 204 12 2020-02-05 2025-06-19 29 9 mfuntowicz@users.noreply.gi... patry.nicolas@protonmail.com
byte_level.rs
in tokenizers/src/normalizers
 164 3 2024-07-15 2025-06-19 6 3 48595927+arthurzucker@users... patry.nicolas@protonmail.com
sequence.rs
in tokenizers/src/processors
 156 6 2022-08-25 2025-06-19 4 2 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
trainer.rs
in tokenizers/src/models/wordlevel
 151 6 2020-10-07 2025-06-19 12 5 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
trainer.rs
in tokenizers/src/models/wordpiece
 147 4 2020-01-03 2025-06-19 19 8 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
serialization.rs
in tokenizers/src/models/wordpiece
 133 6 2020-04-09 2025-06-19 6 2 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
padding.rs
in tokenizers/src/utils
 123 3 2020-03-15 2025-06-19 12 2 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
serialization.rs
in tokenizers/src/models/wordlevel
 114 7 2020-04-09 2025-06-19 8 2 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
mod.rs
in tokenizers/src/utils
 112 3 2020-03-15 2025-06-19 21 8 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
fixed_length.rs
in tokenizers/src/pre_tokenizers
 111 5 2025-06-19 2025-06-19 1 1 vetterle@posteo.de vetterle@posteo.de
pyi
__init__.pyi
in bindings/python/py_src/tokenizers/pre_tokenizers
 90 38 2020-07-25 2025-06-19 24 12 sebastian.puetz@uni-tuebing... vetterle@posteo.de
cache.rs
in tokenizers/src/utils
 85 2 2020-09-08 2025-06-19 4 1 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
mod.rs
in tokenizers/src/models
 59 2 2019-11-17 2025-06-19 29 10 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
trie.rs
in tokenizers/src/models/unigram
 43 - 2020-06-26 2025-06-19 7 2 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
__init__.py
in bindings/python/py_src/tokenizers/pre_tokenizers
 14 - 2020-07-25 2025-06-19 10 5 sebastian.puetz@uni-tuebing... vetterle@posteo.de
wordpiece.rs
in tokenizers/src/decoders
 74 3 2019-12-07 2025-06-18 18 6 m.anthony.moi@gmail.com c@farsight.net
bert_benchmark.rs
in tokenizers/benches
 132 3 2020-08-04 2025-06-17 6 4 sebastian.puetz@uni-tuebing... patry.nicolas@protonmail.com
bpe_benchmark.rs
in tokenizers/benches
 115 4 2020-01-01 2025-06-17 27 5 epwalsh10@gmail.com patry.nicolas@protonmail.com
mod.rs
in tokenizers/benches/common
 76 - 2020-08-04 2025-06-17 4 3 sebastian.puetz@uni-tuebing... patry.nicolas@protonmail.com
layout_benchmark.rs
in tokenizers/benches
 64 1 2022-08-25 2025-06-17 2 1 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
llama3_benchmark.rs
in tokenizers/benches
 64 - 2025-06-17 2025-06-17 1 1 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
unigram_benchmark.rs
in tokenizers/benches
 37 1 2023-05-22 2025-06-17 3 3 dmishig@gmail.com patry.nicolas@protonmail.com
lib.rs
in tokenizers/src
 20 - 2019-11-01 2025-06-17 43 11 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
normalizer.rs
in tokenizers/src/tokenizer
 2002 33 2019-12-28 2025-05-27 52 15 m.anthony.moi@gmail.com co63oc@users.noreply.github...
decoders.rs
in bindings/python/src
 577 53 2019-11-23 2025-05-27 51 15 m.anthony.moi@gmail.com co63oc@users.noreply.github...
convert.py
in bindings/python/scripts
 342 32 2020-09-17 2025-05-27 11 6 patry.nicolas@protonmail.com co63oc@users.noreply.github...
serialization.rs
in tokenizers/src/tokenizer
 215 6 2020-04-09 2025-05-27 16 6 m.anthony.moi@gmail.com 48595927+arthurzucker@users...