huggingface / tokenizers
Temporal Dependencies

A temporal dependency occurs when developers change two or more files at the same time (i.e. they are a part of the same commit).


Files Most Frequently Changed Together (Top 50)

data...

Pairs # same commits # commits 1 # commits 2 latest commit
tokenizers/Cargo.toml
bindings/python/Cargo.toml
3 137 (2%) 135 (2%) 2025-06-24
tokenizers/src/tokenizer/added_vocabulary.rs
tokenizers/src/models/wordlevel/trainer.rs
2 68 (2%) 12 (16%) 2025-06-19
tokenizers/src/models/unigram/model.rs
tokenizers/src/models/bpe/model.rs
2 47 (4%) 108 (1%) 2025-06-19
tokenizers/src/tokenizer/added_vocabulary.rs
tokenizers/src/models/unigram/trainer.rs
2 68 (2%) 54 (3%) 2025-06-19
tokenizers/Cargo.toml
bindings/node/Cargo.toml
2 137 (1%) 17 (11%) 2025-06-24
tokenizers/benches/llama3_benchmark.rs
tokenizers/benches/common/mod.rs
2 2 (100%) 5 (40%) 2025-06-17
tokenizers/src/pre_tokenizers/byte_level.rs
tokenizers/Cargo.toml
2 89 (2%) 137 (1%) 2025-06-19
tokenizers/src/models/unigram/lattice.rs
tokenizers/src/models/bpe/word.rs
2 27 (7%) 20 (10%) 2025-06-19
tokenizers/benches/llama3_benchmark.rs
tokenizers/Cargo.toml
2 2 (100%) 137 (1%) 2025-06-17
tokenizers/src/models/unigram/trainer.rs
tokenizers/src/models/bpe/model.rs
2 54 (3%) 108 (1%) 2025-06-19
tokenizers/src/models/unigram/trainer.rs
tokenizers/src/models/unigram/model.rs
2 54 (3%) 47 (4%) 2025-06-19
tokenizers/src/tokenizer/added_vocabulary.rs
tokenizers/src/models/bpe/model.rs
2 68 (2%) 108 (1%) 2025-06-19
tokenizers/src/models/wordlevel/trainer.rs
tokenizers/src/models/unigram/model.rs
2 12 (16%) 47 (4%) 2025-06-19
tokenizers/src/models/wordlevel/trainer.rs
tokenizers/src/models/bpe/trainer.rs
2 12 (16%) 79 (2%) 2025-06-19
tokenizers/src/models/bpe/word.rs
bindings/python/Cargo.toml
2 20 (10%) 135 (1%) 2025-06-19
bindings/python/src/pre_tokenizers.rs
bindings/python/py_src/tokenizers/pre_tokenizers/__init__.pyi
2 97 (2%) 27 (7%) 2025-06-19
tokenizers/benches/common/mod.rs
tokenizers/Cargo.toml
2 5 (40%) 137 (1%) 2025-06-17
tokenizers/src/models/bpe/trainer.rs
tokenizers/src/models/bpe/model.rs
2 79 (2%) 108 (1%) 2025-06-19
tokenizers/src/models/unigram/trainer.rs
tokenizers/src/models/bpe/trainer.rs
2 54 (3%) 79 (2%) 2025-06-19
tokenizers/src/pre_tokenizers/byte_level.rs
tokenizers/src/normalizers/byte_level.rs
2 89 (2%) 6 (33%) 2025-06-19
tokenizers/src/normalizers/byte_level.rs
tokenizers/Cargo.toml
2 6 (33%) 137 (1%) 2025-06-19
tokenizers/src/models/unigram/lattice.rs
tokenizers/Cargo.toml
2 27 (7%) 137 (1%) 2025-06-19
tokenizers/src/models/unigram/model.rs
tokenizers/src/models/bpe/trainer.rs
2 47 (4%) 79 (2%) 2025-06-19
tokenizers/src/models/wordlevel/trainer.rs
tokenizers/src/models/bpe/model.rs
2 12 (16%) 108 (1%) 2025-06-19
tokenizers/src/models/wordlevel/trainer.rs
tokenizers/src/models/unigram/trainer.rs
2 12 (16%) 54 (3%) 2025-06-19
tokenizers/src/tokenizer/added_vocabulary.rs
tokenizers/src/models/bpe/trainer.rs
2 68 (2%) 79 (2%) 2025-06-19
tokenizers/src/models/unigram/lattice.rs
bindings/python/Cargo.toml
2 27 (7%) 135 (1%) 2025-06-19
bindings/python/Cargo.toml
bindings/node/Cargo.toml
2 135 (1%) 17 (11%) 2025-06-24
tokenizers/src/models/bpe/word.rs
tokenizers/Cargo.toml
2 20 (10%) 137 (1%) 2025-06-19
tokenizers/src/tokenizer/added_vocabulary.rs
tokenizers/src/models/unigram/model.rs
2 68 (2%) 47 (4%) 2025-06-19
tokenizers/src/models/wordpiece/trainer.rs
tokenizers/src/models/unigram/trainer.rs
1 22 (4%) 54 (1%) 2025-06-19
tokenizers/src/processors/bert.rs
tokenizers/src/normalizers/byte_level.rs
1 35 (2%) 6 (16%) 2025-06-19
tokenizers/src/models/wordpiece/trainer.rs
tokenizers/Cargo.toml
1 22 (4%) 137 (<1%) 2025-06-19
tokenizers/src/models/wordpiece/trainer.rs
tokenizers/src/models/mod.rs
1 22 (4%) 38 (2%) 2025-06-19
tokenizers/src/models/wordlevel/serialization.rs
tokenizers/src/models/unigram/trie.rs
1 9 (11%) 7 (14%) 2025-06-19
tokenizers/src/models/unigram/lattice.rs
bindings/python/src/models.rs
1 27 (3%) 90 (1%) 2025-06-19
tokenizers/src/models/wordlevel/mod.rs
tokenizers/src/models/bpe/model.rs
1 36 (2%) 108 (<1%) 2025-06-19
tokenizers/src/models/unigram/model.rs
bindings/node/Cargo.toml
1 47 (2%) 17 (5%) 2025-06-19
tokenizers/src/utils/cache.rs
tokenizers/src/normalizers/byte_level.rs
1 4 (25%) 6 (16%) 2025-06-19
tokenizers/src/processors/template.rs
tokenizers/src/models/wordlevel/trainer.rs
1 25 (4%) 12 (8%) 2025-06-19
tokenizers/src/models/unigram/trainer.rs
bindings/python/src/models.rs
1 54 (1%) 90 (1%) 2025-06-19
tokenizers/src/models/wordpiece/serialization.rs
tokenizers/src/models/unigram/trainer.rs
1 8 (12%) 54 (1%) 2025-06-19
tokenizers/src/utils/padding.rs
tokenizers/src/models/wordlevel/trainer.rs
1 12 (8%) 12 (8%) 2025-06-19
tokenizers/src/models/mod.rs
tokenizers/src/models/bpe/word.rs
1 38 (2%) 20 (5%) 2025-06-19
tokenizers/src/models/wordlevel/serialization.rs
tokenizers/src/models/bpe/trainer.rs
1 9 (11%) 79 (1%) 2025-06-19
tokenizers/src/processors/roberta.rs
bindings/node/Cargo.toml
1 32 (3%) 17 (5%) 2025-06-19
tokenizers/src/utils/cache.rs
tokenizers/src/models/unigram/trie.rs
1 4 (25%) 7 (14%) 2025-06-19
tokenizers/src/utils/truncation.rs
tokenizers/src/models/mod.rs
1 21 (4%) 38 (2%) 2025-06-19
tokenizers/benches/unigram_benchmark.rs
tokenizers/Cargo.toml
1 3 (33%) 137 (<1%) 2025-06-17
tokenizers/src/processors/bert.rs
bindings/node/src/models.rs
1 35 (2%) 2 (50%) 2025-06-19
Dependencies between files in same commits
The number on the lines shows the number of shared commits.
G [tokenizers/Cargo.toml] [tokenizers/Cargo.toml] [bindings/python/Cargo.toml] [bindings/python/Cargo.toml] [tokenizers/Cargo.toml]--[bindings/python/Cargo.toml] 3 [bindings/node/Cargo.toml] [bindings/node/Cargo.toml] [tokenizers/Cargo.toml]--[bindings/node/Cargo.toml] 2 [bindings/python/Cargo.toml]--[bindings/node/Cargo.toml] 2 [tokenizers/src/tokenizer/added_vocabulary.rs] [tokenizers/src/tokenizer/added_vocabulary.rs] [tokenizers/src/models/wordlevel/trainer.rs] [tokenizers/src/models/wordlevel/trainer.rs] [tokenizers/src/tokenizer/added_vocabulary.rs]--[tokenizers/src/models/wordlevel/trainer.rs] 2 [tokenizers/src/models/unigram/model.rs] [tokenizers/src/models/unigram/model.rs] [tokenizers/src/tokenizer/added_vocabulary.rs]--[tokenizers/src/models/unigram/model.rs] 2 [tokenizers/src/models/bpe/model.rs] [tokenizers/src/models/bpe/model.rs] [tokenizers/src/tokenizer/added_vocabulary.rs]--[tokenizers/src/models/bpe/model.rs] 2 [tokenizers/src/models/unigram/trainer.rs] [tokenizers/src/models/unigram/trainer.rs] [tokenizers/src/tokenizer/added_vocabulary.rs]--[tokenizers/src/models/unigram/trainer.rs] 2 [tokenizers/src/models/bpe/trainer.rs] [tokenizers/src/models/bpe/trainer.rs] [tokenizers/src/tokenizer/added_vocabulary.rs]--[tokenizers/src/models/bpe/trainer.rs] 2 [tokenizers/src/models/wordlevel/trainer.rs]--[tokenizers/src/models/unigram/model.rs] 2 [tokenizers/src/models/wordlevel/trainer.rs]--[tokenizers/src/models/bpe/model.rs] 2 [tokenizers/src/models/wordlevel/trainer.rs]--[tokenizers/src/models/unigram/trainer.rs] 2 [tokenizers/src/models/wordlevel/trainer.rs]--[tokenizers/src/models/bpe/trainer.rs] 2 [tokenizers/src/models/unigram/model.rs]--[tokenizers/src/models/bpe/model.rs] 2 [tokenizers/src/models/unigram/model.rs]--[bindings/node/Cargo.toml] 1 [tokenizers/src/models/unigram/model.rs]--[tokenizers/src/models/bpe/trainer.rs] 2 [tokenizers/src/models/unigram/trainer.rs]--[tokenizers/src/models/unigram/model.rs] 2 [tokenizers/src/models/unigram/trainer.rs]--[tokenizers/src/models/bpe/model.rs] 2 [tokenizers/src/models/unigram/trainer.rs]--[tokenizers/src/models/bpe/trainer.rs] 2 [bindings/python/src/models.rs] [bindings/python/src/models.rs] [tokenizers/src/models/unigram/trainer.rs]--[bindings/python/src/models.rs] 1 [tokenizers/benches/llama3_benchmark.rs] [tokenizers/benches/llama3_benchmark.rs] [tokenizers/benches/llama3_benchmark.rs]--[tokenizers/Cargo.toml] 2 [tokenizers/benches/common/mod.rs] [tokenizers/benches/common/mod.rs] [tokenizers/benches/llama3_benchmark.rs]--[tokenizers/benches/common/mod.rs] 2 [tokenizers/benches/common/mod.rs]--[tokenizers/Cargo.toml] 2 [tokenizers/src/pre_tokenizers/byte_level.rs] [tokenizers/src/pre_tokenizers/byte_level.rs] [tokenizers/src/pre_tokenizers/byte_level.rs]--[tokenizers/Cargo.toml] 2 [tokenizers/src/normalizers/byte_level.rs] [tokenizers/src/normalizers/byte_level.rs] [tokenizers/src/pre_tokenizers/byte_level.rs]--[tokenizers/src/normalizers/byte_level.rs] 2 [tokenizers/src/models/unigram/lattice.rs] [tokenizers/src/models/unigram/lattice.rs] [tokenizers/src/models/unigram/lattice.rs]--[tokenizers/Cargo.toml] 2 [tokenizers/src/models/unigram/lattice.rs]--[bindings/python/Cargo.toml] 2 [tokenizers/src/models/bpe/word.rs] [tokenizers/src/models/bpe/word.rs] [tokenizers/src/models/unigram/lattice.rs]--[tokenizers/src/models/bpe/word.rs] 2 [tokenizers/src/models/unigram/lattice.rs]--[bindings/python/src/models.rs] 1 [tokenizers/src/models/bpe/word.rs]--[tokenizers/Cargo.toml] 2 [tokenizers/src/models/bpe/word.rs]--[bindings/python/Cargo.toml] 2 [tokenizers/src/models/bpe/trainer.rs]--[tokenizers/src/models/bpe/model.rs] 2 [bindings/python/src/pre_tokenizers.rs] [bindings/python/src/pre_tokenizers.rs] [bindings/python/py_src/tokenizers/pre_tokenizers/__init__.pyi] [bindings/python/py_src/tokenizers/pre_tokenizers/__init__.pyi] [bindings/python/src/pre_tokenizers.rs]--[bindings/python/py_src/tokenizers/pre_tokenizers/__init__.pyi] 2 [tokenizers/src/normalizers/byte_level.rs]--[tokenizers/Cargo.toml] 2 [tokenizers/src/models/wordpiece/trainer.rs] [tokenizers/src/models/wordpiece/trainer.rs] [tokenizers/src/models/wordpiece/trainer.rs]--[tokenizers/Cargo.toml] 1 [tokenizers/src/models/wordpiece/trainer.rs]--[tokenizers/src/models/unigram/trainer.rs] 1 [tokenizers/src/models/mod.rs] [tokenizers/src/models/mod.rs] [tokenizers/src/models/wordpiece/trainer.rs]--[tokenizers/src/models/mod.rs] 1 [tokenizers/src/processors/bert.rs] [tokenizers/src/processors/bert.rs] [tokenizers/src/processors/bert.rs]--[tokenizers/src/normalizers/byte_level.rs] 1 [bindings/node/src/models.rs] [bindings/node/src/models.rs] [tokenizers/src/processors/bert.rs]--[bindings/node/src/models.rs] 1 [tokenizers/src/models/mod.rs]--[tokenizers/src/models/bpe/word.rs] 1 [tokenizers/src/models/wordlevel/serialization.rs] [tokenizers/src/models/wordlevel/serialization.rs] [tokenizers/src/models/wordlevel/serialization.rs]--[tokenizers/src/models/bpe/trainer.rs] 1 [tokenizers/src/models/unigram/trie.rs] [tokenizers/src/models/unigram/trie.rs] [tokenizers/src/models/wordlevel/serialization.rs]--[tokenizers/src/models/unigram/trie.rs] 1 [tokenizers/src/models/wordlevel/mod.rs] [tokenizers/src/models/wordlevel/mod.rs] [tokenizers/src/models/wordlevel/mod.rs]--[tokenizers/src/models/bpe/model.rs] 1 [tokenizers/src/utils/cache.rs] [tokenizers/src/utils/cache.rs] [tokenizers/src/utils/cache.rs]--[tokenizers/src/normalizers/byte_level.rs] 1 [tokenizers/src/utils/cache.rs]--[tokenizers/src/models/unigram/trie.rs] 1 [tokenizers/src/processors/template.rs] [tokenizers/src/processors/template.rs] [tokenizers/src/processors/template.rs]--[tokenizers/src/models/wordlevel/trainer.rs] 1 [tokenizers/src/models/wordpiece/serialization.rs] [tokenizers/src/models/wordpiece/serialization.rs] [tokenizers/src/models/wordpiece/serialization.rs]--[tokenizers/src/models/unigram/trainer.rs] 1 [tokenizers/src/utils/padding.rs] [tokenizers/src/utils/padding.rs] [tokenizers/src/utils/padding.rs]--[tokenizers/src/models/wordlevel/trainer.rs] 1 [tokenizers/src/processors/roberta.rs] [tokenizers/src/processors/roberta.rs] [tokenizers/src/processors/roberta.rs]--[bindings/node/Cargo.toml] 1 [tokenizers/src/utils/truncation.rs] [tokenizers/src/utils/truncation.rs] [tokenizers/src/utils/truncation.rs]--[tokenizers/src/models/mod.rs] 1 [tokenizers/benches/unigram_benchmark.rs] [tokenizers/benches/unigram_benchmark.rs] [tokenizers/benches/unigram_benchmark.rs]--[tokenizers/Cargo.toml] 1
Download: SVG DOT (open online Graphviz editor)

Open 2D force graph (file dependencies)... Open 3D force graph (file dependencies)...
Open 2D force graph (file dependencies with commits)... Open 3D force graph (file dependencies with commits)...
Dependencies between components in same commits (primary)
The number on the lines shows the number of shared commits.
G tokenizers tokenizers tokenizers--tokenizers 6 bindings bindings tokenizers--bindings 4 bindings--bindings 4
Download: SVG DOT (open online Graphviz editor)

Open 2D force graph (file dependencies)... Open 3D force graph (file dependencies)...