huggingface / optimum-neuron Duplication

Intro

For duplication, we look at places in code where there are 6 or more lines of code that are exactly the same.
Before duplication is calculated, the code is cleaned to remove empty lines, comments, and frequently duplicated constructs such as imports.
You should aim at having as little as possible (<5%) of duplicated code as high-level of duplication can lead to maintenance difficulties, poor factoring, and logical contradictions.

Learn more...

Duplication Overall

18% duplication:

28,425 cleaned lines of cleaned code (without empty lines, comments, and frequently duplicated constructs such as imports)
5,131 duplicated lines

914 duplicates

dependency graphs: 2D graph | 3D graph | 3D graph (with duplicates)...

Duplication per Extension

Duplication per Component (primary)

Longest Duplicates

The list of 50 longest duplicates.

See data for all 914 duplicates...

Size	#	Folders	Files	Lines	Code
57	x 2	notebooks/sagemaker notebooks/sagemaker	deploy-llama-3-3-70b.ipynb deploy-mixtral-8x7b.ipynb	39:95 (15%) 41:97 (13%)	view
53	x 2	notebooks/sagemaker notebooks/sagemaker	deploy-llama-3-3-70b.ipynb deploy-mixtral-8x7b.ipynb	314:366 (14%) 373:425 (12%)	view
50	x 2	optimum/neuron/models/training/granite optimum/neuron/models/training/llama	modeling_granite.py modeling_llama.py	212:276 (22%) 658:722 (7%)	view
43	x 2	optimum/neuron/pipelines/diffusers optimum/neuron/pipelines/diffusers	pipeline_controlnet.py pipeline_controlnet_sd_xl.py	270:322 (16%) 338:390 (12%)	view
39	x 2	optimum/neuron/models/training/llama optimum/neuron/models/training/qwen3	modeling_llama.py modeling_qwen3.py	469:511 (5%) 98:140 (23%)	view
36	x 2	optimum/neuron/models/inference/granite optimum/neuron/models/inference/llama	modeling_granite.py modeling_llama.py	48:88 (33%) 367:407 (9%)	view
32	x 2	notebooks/sagemaker notebooks/sagemaker	deploy-llama-3-3-70b.ipynb deploy-mixtral-8x7b.ipynb	255:286 (8%) 314:345 (7%)	view
31	x 2	optimum/neuron optimum/neuron/models/bert	modeling.py model.py	105:143 (5%) 72:110 (13%)	view
30	x 2	optimum/neuron/peft/tuners/lora optimum/neuron/peft/tuners/lora	layer.py layer.py	209:239 (7%) 252:282 (7%)	view
29	x 2	optimum/neuron/models/training/llama optimum/neuron/models/training/qwen3	modeling_llama.py modeling_qwen3.py	437:467 (4%) 62:92 (17%)	view
27	x 2	optimum/neuron optimum/neuron/models/bert	modeling.py model.py	269:303 (5%) 163:197 (11%)	view
27	x 2	optimum/neuron optimum/neuron/models/bert	modeling.py model.py	222:255 (5%) 119:153 (11%)	view
27	x 2	optimum/neuron optimum/neuron/models/bert	modeling.py model.py	363:398 (5%) 249:284 (11%)	view
26	x 2	optimum/neuron/peft/tuners/lora optimum/neuron/peft/tuners/lora	layer.py layer.py	173:205 (6%) 319:351 (6%)	view
26	x 2	optimum/neuron optimum/neuron/models/bert	modeling.py model.py	317:350 (5%) 207:240 (10%)	view
24	x 2	optimum/commands/export optimum/commands/export	neuron.py neuronx.py	38:62 (19%) 46:70 (8%)	view
23	x 2	optimum/neuron/models/training/granite optimum/neuron/models/training/llama	modeling_granite.py modeling_llama.py	304:330 (10%) 850:876 (3%)	view
23	x 2	optimum/neuron/pipelines/diffusers optimum/neuron/pipelines/diffusers	pipeline_controlnet.py pipeline_controlnet_sd_xl.py	57:179 (8%) 72:240 (6%)	view
22	x 2	optimum/neuron optimum/neuron/models/bert	modeling.py model.py	412:437 (4%) 294:319 (9%)	view
21	x 2	optimum/neuron optimum/neuron/models/bert	modeling.py model.py	364:387 (4%) 120:142 (8%)	view
21	x 2	optimum/neuron/models/bert optimum/neuron/models/bert	model.py model.py	120:142 (8%) 250:273 (8%)	view
21	x 2	optimum/neuron optimum/neuron	modeling.py modeling.py	223:245 (4%) 364:387 (4%)	view
21	x 2	notebooks/sagemaker notebooks/sagemaker	deploy-llama-3-3-70b.ipynb deploy-mixtral-8x7b.ipynb	17:37 (5%) 19:39 (4%)	view
21	x 2	optimum/neuron/pipelines/diffusers optimum/neuron/pipelines/diffusers	pipeline_controlnet.py pipeline_controlnet_sd_xl.py	241:263 (8%) 314:336 (6%)	view
21	x 2	optimum/neuron optimum/neuron/models/bert	modeling.py model.py	223:245 (4%) 250:273 (8%)	view
21	x 2	notebooks/sagemaker notebooks/sagemaker	deploy-llama-3-3-70b.ipynb deploy-mixtral-8x7b.ipynb	154:174 (5%) 218:238 (4%)	view
20	x 2	optimum/neuron optimum/neuron/models/bert	modeling.py model.py	269:289 (3%) 120:140 (8%)	view
20	x 2	optimum/neuron optimum/neuron/models/bert	modeling.py model.py	269:289 (3%) 250:271 (8%)	view
20	x 2	optimum/neuron/models/bert optimum/neuron/models/bert	model.py model.py	120:140 (8%) 163:183 (8%)	view
20	x 2	optimum/neuron/models/bert optimum/neuron/models/bert	model.py model.py	163:183 (8%) 250:271 (8%)	view
20	x 2	optimum/neuron optimum/neuron	modeling.py modeling.py	269:289 (3%) 364:385 (3%)	view
20	x 2	optimum/neuron/peft/tuners/lora optimum/neuron/peft/tuners/lora	layer.py layer.py	123:145 (5%) 284:306 (5%)	view
20	x 2	optimum/neuron optimum/neuron/models/bert	modeling.py model.py	223:243 (3%) 163:183 (8%)	view
20	x 2	optimum/neuron optimum/neuron	modeling.py modeling.py	223:243 (3%) 269:289 (3%)	view
20	x 2	optimum/neuron optimum/neuron/models/bert	modeling.py model.py	364:385 (3%) 163:183 (8%)	view
19	x 2	optimum/neuron/models/inference/llama optimum/neuron/models/inference/qwen3	modeling_llama.py modeling_qwen3.py	442:463 (5%) 71:92 (30%)	view
19	x 2	optimum/neuron/models/training/llama optimum/neuron/models/training/qwen3	modeling_llama.py modeling_qwen3.py	757:776 (2%) 182:201 (11%)	view
19	x 2	optimum/neuron/models/inference/llama optimum/neuron/models/inference/qwen2	modeling_llama.py modeling_qwen2.py	442:463 (5%) 75:96 (24%)	view
19	x 2	optimum/neuron/models/inference/qwen2 optimum/neuron/models/inference/qwen3	modeling_qwen2.py modeling_qwen3.py	75:96 (24%) 71:92 (30%)	view
18	x 2	optimum/neuron/models/inference/granite optimum/neuron/models/inference/llama	modeling_granite.py modeling_llama.py	93:112 (16%) 409:428 (4%)	view
18	x 2	optimum/neuron optimum/neuron/models/bert	modeling.py model.py	317:335 (3%) 120:138 (7%)	view
18	x 2	optimum/neuron optimum/neuron	modeling.py modeling.py	317:335 (3%) 364:383 (3%)	view
18	x 2	optimum/neuron optimum/neuron/models/bert	modeling.py model.py	269:287 (3%) 207:225 (7%)	view
18	x 2	notebooks/sagemaker notebooks/sagemaker	deploy-llama-3-3-70b.ipynb deploy-mixtral-8x7b.ipynb	295:312 (4%) 354:371 (4%)	view
18	x 2	optimum/neuron/models/bert optimum/neuron/models/bert	model.py model.py	120:138 (7%) 294:313 (7%)	view
18	x 2	optimum/neuron optimum/neuron/models/bert	modeling.py model.py	364:383 (3%) 294:313 (7%)	view
18	x 2	optimum/neuron optimum/neuron	modeling.py modeling.py	223:241 (3%) 412:431 (3%)	view
18	x 2	notebooks/sagemaker notebooks/sagemaker	deploy-llama-3-3-70b.ipynb deploy-mixtral-8x7b.ipynb	220:237 (4%) 278:295 (4%)	view
18	x 2	optimum/neuron/models/bert optimum/neuron/models/bert	model.py model.py	120:138 (7%) 207:225 (7%)	view
18	x 2	optimum/neuron/models/bert optimum/neuron/models/bert	model.py model.py	250:269 (7%) 294:313 (7%)	view