huggingface / tokenizers
File Change Frequency

File change frequency (churn) shows the distribution of file updates (days with at least one commit).

Overview
File Change Frequency Overall
  • There are 146 files with 29,051 lines of code.
    • 4 files changed more than 100 times (2,353 lines of code)
    • 11 files changed 51-100 times (7,923 lines of code)
    • 22 files changed 21-50 times (6,925 lines of code)
    • 61 files changed 6-20 times (7,241 lines of code)
    • 48 files changed 1-5 times (4,609 lines of code)
8% | 27% | 23% | 24% | 15%
Legend:
101+
51-100
21-50
6-20
1-5

explore: grouped by folders | grouped by update frequency | data
Contributors Count Frequency Overall
  • There are 146 files with 29,051 lines of code.
    • 0 files changed by more than 25 contributors (0 lines of code)
    • 18 files changed by 11-25 contributors (11,825 lines of code)
    • 39 files changed by 6-10 contributors (8,280 lines of code)
    • 60 files changed by 2-5 contributors (6,582 lines of code)
    • 29 files changed by 1 contributor (2,364 lines of code)
0% | 40% | 28% | 22% | 8%
Legend:
26+
11-25
6-10
2-5
1

explore: grouped by folders | grouped by contributors count | data
File Change Frequency per File Extension
rs, py, md, yaml, mdx, json, ts, toml, pyi, js, rst, inc, gitignore, txt, css, gitattributes, bat, sh, editorconfig, html, ipynb, in, cfg, cjs, tpl, prettierignore
File Change Frequency per Extension
The number of recorded file updates
101+
51-100
21-50
6-20
1-5
rs8% | 30% | 24% | 22% | 13%
toml57% | 0% | 0% | 42% | <1%
pyi0% | 0% | 49% | 50% | 0%
py0% | 0% | 9% | 63% | 26%
js0% | 0% | 0% | 0% | 100%
css0% | 0% | 0% | 0% | 100%
cfg0% | 0% | 0% | 0% | 100%
in0% | 0% | 0% | 0% | 100%
ts0% | 0% | 0% | 0% | 100%
File Change Frequency per Logical Decomposition
primary
primary (file change frequency)
The number of recorded file updates
101+
51-100
21-50
6-20
1-5
tokenizers7% | 28% | 29% | 28% | 6%
bindings8% | 25% | 15% | 20% | 28%
Most Frequently Changed Files (Top 50)

See data for all files...

File# lines# unitscreatedlast modified# changes
(days)		# contributorsfirst
contributor		latest
contributor
mod.rs
in tokenizers/src/tokenizer
 1185 31 2019-12-12 2025-06-24 137 21 m.anthony.moi@gmail.com 48595927+arthurzucker@users...
Cargo.toml
in bindings/python
 26 - 2019-11-01 2025-06-24 109 17 m.anthony.moi@gmail.com arthur.zucker@gmail.com
Cargo.toml
in tokenizers
 82 - 2019-11-01 2025-06-24 109 21 m.anthony.moi@gmail.com arthur.zucker@gmail.com
tokenizer.rs
in bindings/python/src
 1060 74 2019-11-22 2024-12-31 108 17 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
pre_tokenizers.rs
in bindings/python/src
 791 62 2019-11-23 2025-06-19 76 23 m.anthony.moi@gmail.com vetterle@posteo.de
models.rs
in bindings/python/src
 785 58 2019-11-22 2025-06-19 72 15 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
byte_level.rs
in tokenizers/src/pre_tokenizers
 513 18 2019-11-17 2025-06-19 71 12 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
model.rs
in tokenizers/src/models/bpe
 834 26 2019-11-17 2025-06-19 69 15 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
lib.rs
in bindings/python/src
 39 - 2019-11-01 2025-01-09 62 8 m.anthony.moi@gmail.com 13696594+tinyboxvk@users.no...
encoding.rs
in tokenizers/src/tokenizer
 780 10 2019-12-12 2025-06-19 57 12 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
trainer.rs
in tokenizers/src/models/bpe
 598 15 2019-11-17 2025-06-19 54 12 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
encoding.rs
in bindings/python/src
 244 27 2019-12-10 2024-12-31 53 13 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
normalizers.rs
in bindings/python/src
 760 58 2019-12-29 2025-01-28 52 17 m.anthony.moi@gmail.com 48595927+arthurzucker@users...
normalizer.rs
in tokenizers/src/tokenizer
 2002 33 2019-12-28 2025-05-27 52 15 m.anthony.moi@gmail.com co63oc@users.noreply.github...
decoders.rs
in bindings/python/src
 577 53 2019-11-23 2025-05-27 51 15 m.anthony.moi@gmail.com co63oc@users.noreply.github...
trainers.rs
in bindings/python/src
 836 59 2019-12-03 2024-12-31 47 16 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
lib.rs
in tokenizers/src
 20 - 2019-11-01 2025-06-17 43 11 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
processors.rs
in bindings/python/src
 642 48 2019-12-16 2025-01-28 43 15 m.anthony.moi@gmail.com 48595927+arthurzucker@users...
mod.rs
in tokenizers/src/models/wordpiece
 250 11 2019-12-07 2025-06-19 40 10 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
added_vocabulary.rs
in tokenizers/src/tokenizer
 799 32 2020-06-16 2025-06-19 40 10 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
__init__.py
in bindings/python/py_src/tokenizers
 52 - 2020-07-25 2023-09-05 39 8 sebastian.puetz@uni-tuebing... arthur.zucker@gmail.com
trainer.rs
in tokenizers/src/models/unigram
 685 21 2020-06-06 2025-06-19 39 8 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
model.rs
in tokenizers/src/models/unigram
 509 18 2020-06-21 2025-06-19 35 8 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
pyi
__init__.pyi
in bindings/python/py_src/tokenizers
 221 90 2020-07-25 2024-11-05 31 9 sebastian.puetz@uni-tuebing... 48595927+arthurzucker@users...
bert.rs
in tokenizers/src/processors
 268 5 2019-12-12 2025-06-19 31 6 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
metaspace.rs
in tokenizers/src/pre_tokenizers
 331 10 2020-01-07 2025-01-28 31 7 m.anthony.moi@gmail.com 48595927+arthurzucker@users...
whitespace.rs
in tokenizers/src/pre_tokenizers
 94 5 2019-11-17 2025-03-18 30 7 m.anthony.moi@gmail.com c@farsight.net
roberta.rs
in tokenizers/src/processors
 309 5 2020-02-03 2025-06-19 30 7 mfuntowicz@users.noreply.gi... patry.nicolas@protonmail.com
mod.rs
in tokenizers/src/models
 59 2 2019-11-17 2025-06-19 29 10 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
mod.rs
in tokenizers/src/models/wordlevel
 204 12 2020-02-05 2025-06-19 29 9 mfuntowicz@users.noreply.gi... patry.nicolas@protonmail.com
bpe_benchmark.rs
in tokenizers/benches
 115 4 2020-01-01 2025-06-17 27 5 epwalsh10@gmail.com patry.nicolas@protonmail.com
pyi
__init__.pyi
in bindings/python/py_src/tokenizers/pre_tokenizers
 90 38 2020-07-25 2025-06-19 24 12 sebastian.puetz@uni-tuebing... vetterle@posteo.de
sentencepiece_unigram.py
in bindings/python/py_src/tokenizers/implementations
 120 4 2020-08-21 2024-04-17 22 8 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
template.rs
in tokenizers/src/processors
 854 35 2020-09-04 2025-06-19 22 7 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
bert.rs
in tokenizers/src/pre_tokenizers
 76 4 2019-12-17 2022-08-24 21 5 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
mod.rs
in tokenizers/src/utils
 112 3 2020-03-15 2025-06-19 21 8 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
truncation.rs
in tokenizers/src/utils
 279 12 2020-03-15 2025-06-19 21 7 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
mod.rs
in tokenizers/src/pre_tokenizers
 258 2 2019-11-17 2025-06-19 20 8 m.anthony.moi@gmail.com vetterle@posteo.de
lattice.rs
in tokenizers/src/models/unigram
 588 13 2020-06-14 2025-06-19 20 4 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
mod.rs
in tokenizers/src/normalizers
 40 - 2019-11-17 2025-01-28 19 7 m.anthony.moi@gmail.com 48595927+arthurzucker@users...
mod.rs
in tokenizers/src/models/bpe
 59 3 2019-11-17 2024-06-24 19 6 m.anthony.moi@gmail.com cognetta.marco@gmail.com
pyi
__init__.pyi
in bindings/python/py_src/tokenizers/models
 93 35 2020-07-25 2024-06-11 19 7 sebastian.puetz@uni-tuebing... nprisbrey@users.noreply.git...
trainer.rs
in tokenizers/src/models/wordpiece
 147 4 2020-01-03 2025-06-19 19 8 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
normalization.rs
in bindings/python/src/utils
 477 54 2020-09-18 2025-01-28 19 10 m.anthony.moi@gmail.com 48595927+arthurzucker@users...
wordpiece.rs
in tokenizers/src/decoders
 74 3 2019-12-07 2025-06-18 18 6 m.anthony.moi@gmail.com c@farsight.net
bert.rs
in tokenizers/src/normalizers
 102 9 2019-12-29 2023-08-28 18 5 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
word.rs
in tokenizers/src/models/bpe
 277 6 2019-11-17 2025-06-19 18 6 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
Cargo.toml
in bindings/node
 18 - 2023-08-28 2025-06-24 16 3 patry.nicolas@protonmail.com arthur.zucker@gmail.com
serialization.rs
in tokenizers/src/tokenizer
 215 6 2020-04-09 2025-05-27 16 6 m.anthony.moi@gmail.com 48595927+arthurzucker@users...
delimiter.rs
in tokenizers/src/pre_tokenizers
 21 1 2020-02-04 2022-08-24 15 6 mfuntowicz@users.noreply.gi... patry.nicolas@protonmail.com
Files With Most Contributors (Top 50)
Based on the number of unique email addresses found in commits.

See data for all files...

File# lines# unitscreatedlast modified# changes
(days)		# contributorsfirst
contributor		latest
contributor
pre_tokenizers.rs
in bindings/python/src
 791 62 2019-11-23 2025-06-19 76 23 m.anthony.moi@gmail.com vetterle@posteo.de
mod.rs
in tokenizers/src/tokenizer
 1185 31 2019-12-12 2025-06-24 137 21 m.anthony.moi@gmail.com 48595927+arthurzucker@users...
Cargo.toml
in tokenizers
 82 - 2019-11-01 2025-06-24 109 21 m.anthony.moi@gmail.com arthur.zucker@gmail.com
Cargo.toml
in bindings/python
 26 - 2019-11-01 2025-06-24 109 17 m.anthony.moi@gmail.com arthur.zucker@gmail.com
tokenizer.rs
in bindings/python/src
 1060 74 2019-11-22 2024-12-31 108 17 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
normalizers.rs
in bindings/python/src
 760 58 2019-12-29 2025-01-28 52 17 m.anthony.moi@gmail.com 48595927+arthurzucker@users...
trainers.rs
in bindings/python/src
 836 59 2019-12-03 2024-12-31 47 16 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
models.rs
in bindings/python/src
 785 58 2019-11-22 2025-06-19 72 15 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
model.rs
in tokenizers/src/models/bpe
 834 26 2019-11-17 2025-06-19 69 15 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
normalizer.rs
in tokenizers/src/tokenizer
 2002 33 2019-12-28 2025-05-27 52 15 m.anthony.moi@gmail.com co63oc@users.noreply.github...
decoders.rs
in bindings/python/src
 577 53 2019-11-23 2025-05-27 51 15 m.anthony.moi@gmail.com co63oc@users.noreply.github...
processors.rs
in bindings/python/src
 642 48 2019-12-16 2025-01-28 43 15 m.anthony.moi@gmail.com 48595927+arthurzucker@users...
encoding.rs
in bindings/python/src
 244 27 2019-12-10 2024-12-31 53 13 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
byte_level.rs
in tokenizers/src/pre_tokenizers
 513 18 2019-11-17 2025-06-19 71 12 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
encoding.rs
in tokenizers/src/tokenizer
 780 10 2019-12-12 2025-06-19 57 12 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
trainer.rs
in tokenizers/src/models/bpe
 598 15 2019-11-17 2025-06-19 54 12 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
pyi
__init__.pyi
in bindings/python/py_src/tokenizers/pre_tokenizers
 90 38 2020-07-25 2025-06-19 24 12 sebastian.puetz@uni-tuebing... vetterle@posteo.de
lib.rs
in tokenizers/src
 20 - 2019-11-01 2025-06-17 43 11 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
mod.rs
in tokenizers/src/models/wordpiece
 250 11 2019-12-07 2025-06-19 40 10 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
added_vocabulary.rs
in tokenizers/src/tokenizer
 799 32 2020-06-16 2025-06-19 40 10 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
mod.rs
in tokenizers/src/models
 59 2 2019-11-17 2025-06-19 29 10 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
normalization.rs
in bindings/python/src/utils
 477 54 2020-09-18 2025-01-28 19 10 m.anthony.moi@gmail.com 48595927+arthurzucker@users...
pyi
__init__.pyi
in bindings/python/py_src/tokenizers
 221 90 2020-07-25 2024-11-05 31 9 sebastian.puetz@uni-tuebing... 48595927+arthurzucker@users...
mod.rs
in tokenizers/src/models/wordlevel
 204 12 2020-02-05 2025-06-19 29 9 mfuntowicz@users.noreply.gi... patry.nicolas@protonmail.com
sentencepiece_bpe.py
in bindings/python/py_src/tokenizers/implementations
 83 4 2020-07-25 2025-05-27 13 9 sebastian.puetz@uni-tuebing... 86721361+coqueue@users.nore...
lib.rs
in bindings/python/src
 39 - 2019-11-01 2025-01-09 62 8 m.anthony.moi@gmail.com 13696594+tinyboxvk@users.no...
trainer.rs
in tokenizers/src/models/unigram
 685 21 2020-06-06 2025-06-19 39 8 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
__init__.py
in bindings/python/py_src/tokenizers
 52 - 2020-07-25 2023-09-05 39 8 sebastian.puetz@uni-tuebing... arthur.zucker@gmail.com
model.rs
in tokenizers/src/models/unigram
 509 18 2020-06-21 2025-06-19 35 8 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
sentencepiece_unigram.py
in bindings/python/py_src/tokenizers/implementations
 120 4 2020-08-21 2024-04-17 22 8 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
mod.rs
in tokenizers/src/utils
 112 3 2020-03-15 2025-06-19 21 8 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
mod.rs
in tokenizers/src/pre_tokenizers
 258 2 2019-11-17 2025-06-19 20 8 m.anthony.moi@gmail.com vetterle@posteo.de
trainer.rs
in tokenizers/src/models/wordpiece
 147 4 2020-01-03 2025-06-19 19 8 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
metaspace.rs
in tokenizers/src/pre_tokenizers
 331 10 2020-01-07 2025-01-28 31 7 m.anthony.moi@gmail.com 48595927+arthurzucker@users...
whitespace.rs
in tokenizers/src/pre_tokenizers
 94 5 2019-11-17 2025-03-18 30 7 m.anthony.moi@gmail.com c@farsight.net
roberta.rs
in tokenizers/src/processors
 309 5 2020-02-03 2025-06-19 30 7 mfuntowicz@users.noreply.gi... patry.nicolas@protonmail.com
template.rs
in tokenizers/src/processors
 854 35 2020-09-04 2025-06-19 22 7 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
truncation.rs
in tokenizers/src/utils
 279 12 2020-03-15 2025-06-19 21 7 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
mod.rs
in tokenizers/src/normalizers
 40 - 2019-11-17 2025-01-28 19 7 m.anthony.moi@gmail.com 48595927+arthurzucker@users...
pyi
__init__.pyi
in bindings/python/py_src/tokenizers/models
 93 35 2020-07-25 2024-06-11 19 7 sebastian.puetz@uni-tuebing... nprisbrey@users.noreply.git...
pyproject.toml
in bindings/python
 61 - 2019-11-27 2025-02-11 14 7 m.anthony.moi@gmail.com git@qaq.io
pretokenization.rs
in bindings/python/src/utils
 254 18 2020-09-18 2024-11-15 11 7 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
bert.rs
in tokenizers/src/processors
 268 5 2019-12-12 2025-06-19 31 6 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
mod.rs
in tokenizers/src/models/bpe
 59 3 2019-11-17 2024-06-24 19 6 m.anthony.moi@gmail.com cognetta.marco@gmail.com
word.rs
in tokenizers/src/models/bpe
 277 6 2019-11-17 2025-06-19 18 6 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
wordpiece.rs
in tokenizers/src/decoders
 74 3 2019-12-07 2025-06-18 18 6 m.anthony.moi@gmail.com c@farsight.net
serialization.rs
in tokenizers/src/tokenizer
 215 6 2020-04-09 2025-05-27 16 6 m.anthony.moi@gmail.com 48595927+arthurzucker@users...
delimiter.rs
in tokenizers/src/pre_tokenizers
 21 1 2020-02-04 2022-08-24 15 6 mfuntowicz@users.noreply.gi... patry.nicolas@protonmail.com
utils.rs
in tokenizers/src/normalizers
 48 5 2020-01-07 2025-01-28 13 6 m.anthony.moi@gmail.com 48595927+arthurzucker@users...
mod.rs
in tokenizers/src/processors
 112 4 2019-11-17 2024-08-06 13 6 m.anthony.moi@gmail.com 48595927+arthurzucker@users...
Files With Least Contributors (Top 50)
Based on the number of unique email addresses found in commits.

See data for all files...

File# lines# unitscreatedlast modified# changes
(days)		# contributorsfirst
contributor		latest
contributor
serde_pyo3.rs
in bindings/python/src/utils
 454 44 2024-08-07 2024-12-31 2 1 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
encoding.rs
in bindings/node/src
 310 5 2023-08-28 2023-08-28 1 1 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
models.rs
in bindings/node/src
 303 11 2023-08-28 2025-06-19 2 1 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
normalizers.rs
in bindings/node/src
 159 1 2023-08-28 2023-08-28 1 1 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
visualizer-styles.css
in bindings/python/py_src/tokenizers/tools
 138 - 2020-12-04 2020-12-04 1 1 tal@lighttag.io tal@lighttag.io
fixed_length.rs
in tokenizers/src/pre_tokenizers
 111 5 2025-06-19 2025-06-19 1 1 vetterle@posteo.de vetterle@posteo.de
tokenizer.rs
in bindings/node/src/tasks
 111 8 2023-08-28 2023-08-28 1 1 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
processors.rs
in bindings/node/src
 110 2 2023-08-28 2023-08-28 1 1 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
cache.rs
in tokenizers/src/utils
 85 2 2020-09-08 2025-06-19 4 1 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
models.rs
in bindings/node/src/tasks
 67 6 2023-08-28 2023-08-28 1 1 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
layout_benchmark.rs
in tokenizers/benches
 64 1 2022-08-25 2025-06-17 2 1 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
llama3_benchmark.rs
in tokenizers/benches
 64 - 2025-06-17 2025-06-17 1 1 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
trainers.rs
in bindings/node/src
 59 4 2023-08-28 2023-08-28 1 1 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
prepend.rs
in tokenizers/src/normalizers
 56 2 2023-03-23 2023-03-23 1 1 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
fancy.rs
in tokenizers/src/utils
 55 2 2022-06-10 2024-12-31 3 1 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
cfg
setup.cfg
in bindings/python
 51 - 2022-10-05 2022-10-05 1 1 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
utils.rs
in bindings/node/src
 42 - 2023-08-28 2023-08-28 1 1 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
fuse.rs
in tokenizers/src/decoders
 34 2 2023-03-23 2023-03-23 1 1 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
onig.rs
in tokenizers/src/utils
 15 - 2022-06-10 2022-06-10 1 1 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
arc_rwlock_serde.rs
in bindings/node/src
 15 - 2023-08-28 2023-08-28 1 1 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
lib.rs
in bindings/node/src
 13 - 2023-08-28 2023-08-28 1 1 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
progress.rs
in tokenizers/src/utils
 12 - 2020-11-11 2023-10-05 2 1 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
conftest.py
in bindings/python
 12 3 2020-09-02 2020-09-02 1 1 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
mod.rs
in tokenizers/src/models/unigram
 8 - 2020-06-06 2020-09-15 7 1 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
types.ts
in bindings/node
 6 - 2023-08-28 2023-08-28 1 1 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
build.rs
in bindings/node
 4 1 2023-08-28 2023-08-28 1 1 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
mod.rs
in tokenizers/src/pre_tokenizers/unicode_scripts
 3 - 2020-10-20 2020-10-20 1 1 m.anthony.moi@gmail.com m.anthony.moi@gmail.com
mod.rs
in bindings/node/src/tasks
 2 - 2023-08-28 2023-08-28 1 1 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
rustfmt.toml
in bindings/node
 1 - 2023-08-28 2023-08-28 1 1 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
tokenizer.rs
in bindings/node/src
 463 6 2023-08-28 2024-08-07 2 2 patry.nicolas@protonmail.com 48595927+arthurzucker@users...
index.js
in bindings/node
 376 1 2023-08-28 2024-03-30 2 2 patry.nicolas@protonmail.com 48595927+arthurzucker@users...
pre_tokenizers.rs
in bindings/node/src
 231 3 2023-08-28 2024-03-30 2 2 patry.nicolas@protonmail.com 48595927+arthurzucker@users...
pre_tokenizer.rs
in tokenizers/src/tokenizer
 228 5 2020-07-24 2024-08-08 12 2 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
decoders.rs
in bindings/node/src
 161 1 2023-08-28 2024-03-30 2 2 patry.nicolas@protonmail.com 48595927+arthurzucker@users...
sequence.rs
in tokenizers/src/processors
 156 6 2022-08-25 2025-06-19 4 2 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
serialization.rs
in tokenizers/src/models/wordpiece
 133 6 2020-04-09 2025-06-19 6 2 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
padding.rs
in tokenizers/src/utils
 123 3 2020-03-15 2025-06-19 12 2 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
serialization.rs
in tokenizers/src/models/wordlevel
 114 7 2020-04-09 2025-06-19 8 2 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
iter.rs
in tokenizers/src/utils
 71 6 2020-03-15 2021-12-17 4 2 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
strip.rs
in tokenizers/src/decoders
 67 2 2023-03-23 2025-01-09 3 2 patry.nicolas@protonmail.com 13696594+tinyboxvk@users.no...
sequence.rs
in tokenizers/src/decoders
 48 2 2022-06-02 2024-03-29 2 2 patry.nicolas@protonmail.com e.a.platanios@gmail.com
trie.rs
in tokenizers/src/models/unigram
 43 - 2020-06-26 2025-06-19 7 2 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
in
MANIFEST.in
in bindings/python
 7 - 2019-12-03 2020-07-11 3 2 m.anthony.moi@gmail.com setu4993@yahoo.co.in
__init__.py
in bindings/python/py_src/tokenizers/tools
 1 - 2020-12-04 2022-10-05 2 2 tal@lighttag.io patry.nicolas@protonmail.com
serialization.rs
in tokenizers/src/models/bpe
 217 6 2020-04-09 2025-06-19 12 3 m.anthony.moi@gmail.com patry.nicolas@protonmail.com
spm_parity_check.py
in bindings/python/scripts
 208 5 2020-08-21 2024-03-12 9 3 patry.nicolas@protonmail.com 48595927+arthurzucker@users...
pattern.rs
in tokenizers/src/tokenizer
 187 13 2020-07-28 2025-05-27 6 3 m.anthony.moi@gmail.com co63oc@users.noreply.github...
byte_level.rs
in tokenizers/src/normalizers
 164 3 2024-07-15 2025-06-19 6 3 48595927+arthurzucker@users... patry.nicolas@protonmail.com
pre_tokenizer.rs
in tokenizers/src/pre_tokenizers/unicode_scripts
 132 6 2020-09-15 2023-07-16 7 3 patry.nicolas@protonmail.com patry.nicolas@protonmail.com
digits.rs
in tokenizers/src/pre_tokenizers
 93 4 2020-09-03 2022-08-24 8 3 patry.nicolas@protonmail.com patry.nicolas@protonmail.com