huggingface / nanotron Duplication

Intro

For duplication, we look at places in code where there are 6 or more lines of code that are exactly the same.
Before duplication is calculated, the code is cleaned to remove empty lines, comments, and frequently duplicated constructs such as imports.
You should aim at having as little as possible (<5%) of duplicated code as high-level of duplication can lead to maintenance difficulties, poor factoring, and logical contradictions.

Learn more...

Duplication Overall

18% duplication:

20,776 cleaned lines of cleaned code (without empty lines, comments, and frequently duplicated constructs such as imports)
3,946 duplicated lines

303 duplicates

dependency graphs: 2D graph | 3D graph | 3D graph (with duplicates)...

Duplication per Extension

Duplication per Component (primary)

Longest Duplicates

The list of 50 longest duplicates.

See data for all 303 duplicates...

Size	#	Folders	Files	Lines	Code
115	x 2	src/nanotron/nn src/nanotron/nn	llama3_ring_attention.py ring_attention.py	662:808 (17%) 446:592 (24%)	view
94	x 2	src/nanotron/nn src/nanotron/nn	llama3_ring_attention.py ring_attention.py	504:650 (14%) 288:432 (20%)	view
77	x 2	src/nanotron/models src/nanotron/models	llama.py qwen.py	1097:1224 (8%) 963:1091 (9%)	view
61	x 2	src/nanotron/generation src/nanotron/generation	decode.py decode.py	355:433 (9%) 673:751 (9%)	view
53	x 2	src/nanotron/generation src/nanotron/generation	decode.py decode.py	469:532 (8%) 751:814 (8%)	view
33	x 2	src/nanotron/models src/nanotron/models	starcoder2.py starcoder2.py	783:825 (2%) 1024:1066 (2%)	view
27	x 2	src/nanotron/parallel/tensor_parallel src/nanotron/parallel/tensor_parallel	functional.py functional.py	38:79 (5%) 118:159 (5%)	view
27	x 2	src/nanotron/generation src/nanotron/generation	decode.py decode.py	316:351 (4%) 624:661 (4%)	view
24	x 2	src/nanotron/nn src/nanotron/nn	ring_attention_lucidrain.py ring_attention_lucidrain.py	1584:1607 (1%) 1623:1646 (1%)	view
23	x 2	src/nanotron/nn src/nanotron/nn	ring_attention_lucidrain.py ring_attention_lucidrain.py	1190:1212 (1%) 1624:1646 (1%)	view
23	x 2	src/nanotron/nn src/nanotron/nn	ring_attention_lucidrain.py ring_attention_lucidrain.py	1190:1212 (1%) 1585:1607 (1%)	view
22	x 2	src/nanotron/models src/nanotron/models	llama.py qwen.py	1038:1062 (2%) 908:931 (2%)	view
21	x 2	src/nanotron/data src/nanotron/data	dataloader.py dataloader.py	165:185 (8%) 205:225 (8%)	view
21	x 2	src/nanotron/models src/nanotron/models	llama.py qwen.py	965:992 (2%) 842:869 (2%)	view
20	x 2	src/nanotron/models src/nanotron/models	starcoder2.py starcoder2.py	745:774 (1%) 987:1015 (1%)	view
19	x 2	src/nanotron/generation src/nanotron/generation	sampler.py sampler.py	38:67 (11%) 245:276 (11%)	view
19	x 2	src/nanotron/models src/nanotron/models	llama.py qwen.py	876:897 (2%) 736:756 (2%)	view
19	x 2	src/nanotron/models src/nanotron/models	llama.py qwen.py	1070:1093 (2%) 939:959 (2%)	view
18	x 2	src/nanotron/generation src/nanotron/generation	decode.py decode.py	252:275 (2%) 584:607 (2%)	view
17	x 2	src/nanotron/data/nemo_dataset src/nanotron/data/nemo_dataset	helpers.cpp helpers.cpp	248:264 (4%) 507:523 (4%)	view
17	x 2	src/nanotron/nn src/nanotron/nn	llama3_ring_attention.py llama3_ring_attention.py	431:447 (2%) 471:487 (2%)	view
16	x 2	src/nanotron/models src/nanotron/models	starcoder2.py starcoder2.py	668:687 (1%) 949:967 (1%)	view
16	x 2	src/nanotron/models src/nanotron/models	llama.py starcoder2.py	894:911 (1%) 1335:1352 (1%)	view
16	x 2	src/nanotron/parallel/pipeline_parallel src/nanotron/parallel/pipeline_parallel	engine.py engine.py	139:164 (7%) 189:214 (7%)	view
16	x 2	src/nanotron/models src/nanotron/models	llama.py qwen.py	821:839 (1%) 686:704 (2%)	view
16	x 2	src/nanotron/serialize src/nanotron/serialize	weights.py weights.py	226:245 (5%) 359:378 (5%)	view
16	x 2	src/nanotron/nn src/nanotron/nn	llama3_ring_attention.py llama3_ring_attention.py	392:407 (2%) 431:446 (2%)	view
16	x 2	src/nanotron/nn src/nanotron/nn	llama3_ring_attention.py llama3_ring_attention.py	392:407 (2%) 471:486 (2%)	view
15	x 2	src/nanotron/data src/nanotron/data	dataloader_builder.py tokenized_bytes.py	42:56 (24%) 693:707 (2%)	view
15	x 2	src/nanotron/nn src/nanotron/nn	llama3_ring_attention.py llama3_ring_attention.py	306:320 (2%) 487:501 (2%)	view
15	x 2	src/nanotron/nn src/nanotron/nn	ring_attention_lucidrain.py ring_attention_lucidrain.py	1251:1265 (1%) 1457:1471 (1%)	view
15	x 2	src/nanotron/eval src/nanotron/eval	evaluation_tasks.py evaluation_tasks.py	118:132 (4%) 347:361 (4%)	view
15	x 2	src/nanotron/nn src/nanotron/nn	llama3_ring_attention.py llama3_ring_attention.py	450:464 (2%) 490:504 (2%)	view
15	x 2	src/nanotron/nn src/nanotron/nn	llama3_ring_attention.py llama3_ring_attention.py	411:425 (2%) 450:464 (2%)	view
15	x 2	src/nanotron/nn src/nanotron/nn	llama3_ring_attention.py ring_attention.py	134:148 (2%) 48:63 (3%)	view
15	x 2	src/nanotron/nn src/nanotron/nn	llama3_ring_attention.py llama3_ring_attention.py	411:425 (2%) 490:504 (2%)	view
14	x 2	src/nanotron/config src/nanotron/config	config.py lighteval_config.py	423:437 (2%) 14:28 (13%)	view
14	x 2	src/nanotron/models src/nanotron/models	llama.py starcoder2.py	976:993 (1%) 1375:1394 (1%)	view
14	x 2	src/nanotron/data/nemo_dataset src/nanotron/data/nemo_dataset	helpers.cpp helpers.cpp	395:416 (3%) 643:663 (3%)	view
14	x 2	src/nanotron/models src/nanotron/models	llama.py starcoder2.py	266:285 (1%) 248:267 (1%)	view
14	x 2	src/nanotron/nn src/nanotron/nn	llama3_ring_attention.py ring_attention.py	315:329 (2%) 189:203 (2%)	view
14	x 2	src/nanotron/eval src/nanotron/eval	evaluation_tasks.py evaluation_tasks.py	152:165 (4%) 212:225 (4%)	view
14	x 2	src/nanotron/nn src/nanotron/nn	llama3_ring_attention.py ring_attention.py	269:282 (2%) 141:154 (2%)	view
13	x 2	src/nanotron/data src/nanotron/data	samplers.py samplers.py	243:256 (5%) 333:346 (5%)	view
13	x 2	src/nanotron/nn src/nanotron/nn	ring_attention_lucidrain.py ring_attention_lucidrain.py	760:772 (1%) 1520:1532 (1%)	view
13	x 2	src/nanotron/models src/nanotron/models	starcoder2.py starcoder2.py	832:845 (1%) 1077:1090 (1%)	view
13	x 2	src/nanotron/models src/nanotron/models	llama.py starcoder2.py	427:441 (1%) 949:962 (1%)	view
13	x 2	src/nanotron/models src/nanotron/nn	qwen.py moe.py	548:566 (1%) 194:212 (10%)	view
13	x 2	src/nanotron/data src/nanotron/data	samplers.py samplers.py	194:207 (5%) 243:256 (5%)	view
13	x 2	src/nanotron/models src/nanotron/models	qwen.py starcoder2.py	853:869 (1%) 1375:1391 (1%)	view

Duplicated Units

The list of top 1 duplicated units.

See data for all 1 unit duplicate

Size	#	Folders	Files	Lines	Code
32	x 2	src/nanotron/nn src/nanotron/nn	llama3_ring_attention.py ring_attention.py	0:0 0:0	view