huggingface / tokenizers Duplication

Intro

For duplication, we look at places in code where there are 6 or more lines of code that are exactly the same.
Before duplication is calculated, the code is cleaned to remove empty lines, comments, and frequently duplicated constructs such as imports.
You should aim at having as little as possible (<5%) of duplicated code as high-level of duplication can lead to maintenance difficulties, poor factoring, and logical contradictions.

Learn more...

Duplication Overall

17% duplication:

25,202 cleaned lines of cleaned code (without empty lines, comments, and frequently duplicated constructs such as imports)
4,475 duplicated lines

893 duplicates

dependency graphs: 2D graph | 3D graph | 3D graph (with duplicates)...

Duplication per Extension

Duplication per Component (primary)

Longest Duplicates

The list of 50 longest duplicates.

See data for all 893 duplicates...

Size	#	Folders	Files	Lines	Code
58	x 2	tokenizers/src/tokenizer tokenizers/src/tokenizer	normalizer.rs normalizer.rs	1989:2049 (3%) 2051:2111 (3%)	view
40	x 2	tokenizers/src/tokenizer tokenizers/src/tokenizer	normalizer.rs normalizer.rs	1632:1673 (2%) 1675:1716 (2%)	view
37	x 2	tokenizers/src/pre_tokenizers tokenizers/src/processors	byte_level.rs sequence.rs	510:547 (7%) 126:163 (26%)	view
35	x 2	tokenizers/src/processors tokenizers/src/processors	bert.rs roberta.rs	56:95 (14%) 86:125 (12%)	view
31	x 2	tokenizers/src/processors tokenizers/src/processors	bert.rs roberta.rs	97:130 (12%) 127:160 (10%)	view
28	x 2	bindings/python/src bindings/python/src	trainers.rs trainers.rs	152:179 (3%) 372:399 (3%)	view
26	x 2	bindings/python/src bindings/python/src	trainers.rs trainers.rs	320:348 (3%) 527:555 (3%)	view
23	x 2	tokenizers/src/processors tokenizers/src/processors	bert.rs roberta.rs	172:200 (9%) 213:241 (8%)	view
22	x 2	tokenizers/src/models/unigram tokenizers/src/models/wordlevel	trainer.rs trainer.rs	635:662 (3%) 104:131 (16%)	view
22	x 2	tokenizers/src/tokenizer tokenizers/src/tokenizer	normalizer.rs normalizer.rs	1996:2018 (1%) 2120:2142 (1%)	view
22	x 2	tokenizers/src/tokenizer tokenizers/src/tokenizer	normalizer.rs normalizer.rs	2058:2080 (1%) 2120:2142 (1%)	view
21	x 2	tokenizers/src/processors tokenizers/src/processors	bert.rs template.rs	254:274 (8%) 930:950 (2%)	view
20	x 2	bindings/python/src/utils bindings/python/src/utils	serde_pyo3.rs serde_pyo3.rs	427:453 (5%) 467:493 (5%)	view
20	x 2	tokenizers/src/pre_tokenizers tokenizers/src/pre_tokenizers	bert.rs sequence.rs	28:47 (28%) 61:80 (33%)	view
18	x 2	tokenizers/src/tokenizer tokenizers/src/tokenizer	normalizer.rs normalizer.rs	2062:2080 (<1%) 2192:2210 (<1%)	view
18	x 2	tokenizers/src/tokenizer tokenizers/src/tokenizer	normalizer.rs normalizer.rs	1639:1657 (<1%) 1725:1744 (<1%)	view
18	x 2	tokenizers/src/tokenizer tokenizers/src/tokenizer	normalizer.rs normalizer.rs	2000:2018 (<1%) 2192:2210 (<1%)	view
18	x 2	bindings/python/src bindings/python/src	trainers.rs trainers.rs	237:257 (2%) 789:809 (2%)	view
18	x 2	bindings/python/src bindings/python/src	trainers.rs trainers.rs	238:258 (2%) 452:472 (2%)	view
18	x 2	tokenizers/src/tokenizer tokenizers/src/tokenizer	normalizer.rs normalizer.rs	2124:2142 (<1%) 2192:2210 (<1%)	view
18	x 2	bindings/node/src bindings/node/src	decoders.rs pre_tokenizers.rs	96:115 (12%) 161:180 (9%)	view
18	x 2	tokenizers/src/tokenizer tokenizers/src/tokenizer	normalizer.rs normalizer.rs	1682:1700 (<1%) 1725:1744 (<1%)	view
18	x 2	tokenizers/src/processors tokenizers/src/processors	template.rs template.rs	1048:1065 (2%) 1099:1116 (2%)	view
18	x 2	bindings/python/scripts bindings/python/scripts	convert.py convert.py	119:139 (5%) 254:274 (5%)	view
17	x 2	bindings/python/src bindings/python/src	trainers.rs trainers.rs	643:660 (2%) 789:806 (2%)	view
17	x 2	bindings/python/src bindings/python/src	trainers.rs trainers.rs	452:471 (2%) 790:809 (2%)	view
17	x 2	bindings/python/src/utils bindings/python/src/utils	serde_pyo3.rs serde_pyo3.rs	399:421 (4%) 431:453 (4%)	view
17	x 2	bindings/python/src/utils bindings/python/src/utils	serde_pyo3.rs serde_pyo3.rs	399:421 (4%) 471:493 (4%)	view
17	x 2	bindings/python/src bindings/python/src	trainers.rs trainers.rs	237:254 (2%) 643:660 (2%)	view
17	x 2	tokenizers/src/pre_tokenizers tokenizers/src/processors	byte_level.rs sequence.rs	488:504 (3%) 101:117 (12%)	view
17	x 2	tokenizers/src/tokenizer tokenizers/src/tokenizer	mod.rs mod.rs	1390:1408 (1%) 1443:1461 (1%)	view
17	x 2	bindings/python/src bindings/python/src	tokenizer.rs tokenizer.rs	1038:1055 (1%) 1097:1114 (1%)	view
17	x 2	tokenizers/src/processors tokenizers/src/processors	bert.rs template.rs	230:246 (6%) 908:924 (2%)	view
16	x 2	bindings/python/src bindings/python/src	trainers.rs trainers.rs	329:346 (2%) 683:700 (2%)	view
16	x 2	tokenizers/src/processors tokenizers/src/processors	bert.rs roberta.rs	237:252 (6%) 280:295 (5%)	view
16	x 2	tokenizers/src/processors tokenizers/src/processors	template.rs template.rs	903:918 (2%) 981:996 (2%)	view
16	x 2	bindings/python/src bindings/python/src	trainers.rs trainers.rs	452:468 (2%) 644:660 (2%)	view
16	x 2	bindings/python/src bindings/python/src	trainers.rs trainers.rs	536:553 (2%) 683:700 (2%)	view
15	x 2	tokenizers/src/models/bpe tokenizers/src/models/wordlevel	trainer.rs trainer.rs	623:642 (2%) 111:130 (11%)	view
15	x 2	tokenizers/src/models/bpe tokenizers/src/models/unigram	trainer.rs trainer.rs	623:642 (2%) 642:661 (2%)	view
15	x 2	tokenizers/src/processors tokenizers/src/processors	bert.rs bert.rs	135:152 (6%) 162:179 (6%)	view
15	x 2	tokenizers/src/processors tokenizers/src/processors	template.rs template.rs	1069:1083 (1%) 1101:1115 (1%)	view
15	x 2	tokenizers/src/tokenizer tokenizers/src/tokenizer	normalizer.rs normalizer.rs	1523:1539 (<1%) 1642:1657 (<1%)	view
15	x 2	bindings/python/src bindings/python/src	trainers.rs trainers.rs	329:345 (2%) 856:872 (2%)	view
15	x 2	tokenizers/src/tokenizer tokenizers/src/tokenizer	normalizer.rs normalizer.rs	1523:1539 (<1%) 1685:1700 (<1%)	view
15	x 2	tokenizers/src/processors tokenizers/src/processors	bert.rs template.rs	213:228 (6%) 891:906 (1%)	view
15	x 2	tokenizers/src/tokenizer tokenizers/src/tokenizer	normalizer.rs normalizer.rs	2120:2134 (<1%) 2239:2253 (<1%)	view
15	x 2	tokenizers/src/tokenizer tokenizers/src/tokenizer	normalizer.rs normalizer.rs	1523:1539 (<1%) 1728:1744 (<1%)	view
15	x 2	tokenizers/src/tokenizer tokenizers/src/tokenizer	normalizer.rs normalizer.rs	2058:2072 (<1%) 2239:2253 (<1%)	view
15	x 2	bindings/python/src bindings/python/src	trainers.rs trainers.rs	683:699 (2%) 856:872 (2%)	view

Duplicated Units

The list of top 4 duplicated units.

See data for all 4 unit duplicates...

Size	#	Folders	Files	Lines	Code
27	x 2	tokenizers/src/models/unigram tokenizers/src/models/wordlevel	trainer.rs trainer.rs	628:657 97:126	view
11	x 2	tokenizers/src/models/unigram tokenizers/src/models/bpe	trainer.rs trainer.rs	86:98 223:235	view
6	x 3	bindings/python/src bindings/python/src bindings/python/src	decoders.rs pre_tokenizers.rs normalizers.rs	551:558 840:847 658:665	view
6	x 2	bindings/python/src bindings/python/src	pre_tokenizers.rs normalizers.rs	851:858 669:676	view