huggingface / optimum-benchmark Duplication

Intro

For duplication, we look at places in code where there are 6 or more lines of code that are exactly the same.
Before duplication is calculated, the code is cleaned to remove empty lines, comments, and frequently duplicated constructs such as imports.
You should aim at having as little as possible (<5%) of duplicated code as high-level of duplication can lead to maintenance difficulties, poor factoring, and logical contradictions.

Learn more...

Duplication Overall

20% duplication:

6,814 cleaned lines of cleaned code (without empty lines, comments, and frequently duplicated constructs such as imports)
1,375 duplicated lines

336 duplicates

dependency graphs: 2D graph | 3D graph | 3D graph (with duplicates)...

Duplication per Extension

Duplication per Component (primary)

Longest Duplicates

The list of 50 longest duplicates.

See data for all 336 duplicates...

Size	#	Folders	Files	Lines	Code
29	x 2	optimum_benchmark/launchers/process optimum_benchmark/launchers/torchrun	launcher.py launcher.py	35:73 (29%) 55:93 (18%)	view
28	x 2	optimum_benchmark/preprocessors optimum_benchmark/preprocessors	dataset_preprocessor.py dataset_preprocessor.py	51:85 (7%) 89:123 (7%)	view
28	x 2	optimum_benchmark/preprocessors optimum_benchmark/preprocessors	dataset_preprocessor.py dataset_preprocessor.py	13:47 (7%) 51:85 (7%)	view
28	x 2	optimum_benchmark/preprocessors optimum_benchmark/preprocessors	dataset_preprocessor.py dataset_preprocessor.py	13:47 (7%) 89:123 (7%)	view
23	x 2	optimum_benchmark/scenarios/energy_star optimum_benchmark/scenarios/inference	scenario.py scenario.py	16:39 (8%) 23:46 (6%)	view
22	x 2	optimum_benchmark/backends/tensorrt_llm optimum_benchmark/backends/vllm	backend.py backend.py	25:50 (26%) 26:51 (24%)	view
20	x 2	optimum_benchmark/scenarios/energy_star optimum_benchmark/scenarios/inference	config.py config.py	80:101 (28%) 81:102 (25%)	view
18	x 2	optimum_benchmark/backends/py_txi optimum_benchmark/backends/vllm	backend.py backend.py	19:39 (16%) 26:46 (20%)	view
18	x 2	optimum_benchmark/backends/py_txi optimum_benchmark/backends/tensorrt_llm	backend.py backend.py	19:39 (16%) 25:45 (21%)	view
15	x 2	optimum_benchmark/trackers optimum_benchmark/trackers	latency.py latency.py	338:355 (3%) 515:532 (3%)	view
15	x 2	optimum_benchmark/trackers optimum_benchmark/trackers	latency.py latency.py	245:262 (3%) 338:355 (3%)	view
15	x 2	optimum_benchmark/trackers optimum_benchmark/trackers	latency.py latency.py	270:286 (3%) 634:650 (3%)	view
15	x 2	optimum_benchmark/trackers optimum_benchmark/trackers	latency.py latency.py	245:262 (3%) 515:532 (3%)	view
14	x 2	optimum_benchmark/launchers/process optimum_benchmark/launchers/torchrun	launcher.py launcher.py	108:122 (14%) 139:153 (9%)	view
14	x 2	optimum_benchmark/backends optimum_benchmark/backends	timm_utils.py transformers_utils.py	54:68 (29%) 145:159 (8%)	view
14	x 2	optimum_benchmark/launchers/process optimum_benchmark/launchers/torchrun	launcher.py launcher.py	89:106 (14%) 118:135 (9%)	view
14	x 2	optimum_benchmark/scenarios/energy_star optimum_benchmark/scenarios/inference	scenario.py scenario.py	90:103 (5%) 109:123 (3%)	view
13	x 2	optimum_benchmark/preprocessors optimum_benchmark/preprocessors	dataset_preprocessor.py dataset_preprocessor.py	89:105 (3%) 267:283 (3%)	view
13	x 2	optimum_benchmark/preprocessors optimum_benchmark/preprocessors	dataset_preprocessor.py dataset_preprocessor.py	51:67 (3%) 267:283 (3%)	view
13	x 2	optimum_benchmark/preprocessors optimum_benchmark/preprocessors	dataset_preprocessor.py dataset_preprocessor.py	13:29 (3%) 267:283 (3%)	view
12	x 2	optimum_benchmark/preprocessors optimum_benchmark/preprocessors	dataset_preprocessor.py dataset_preprocessor.py	110:123 (3%) 154:167 (3%)	view
12	x 2	optimum_benchmark/preprocessors optimum_benchmark/preprocessors	dataset_preprocessor.py dataset_preprocessor.py	110:123 (3%) 433:446 (3%)	view
12	x 2	optimum_benchmark/preprocessors optimum_benchmark/preprocessors	dataset_preprocessor.py dataset_preprocessor.py	34:47 (3%) 154:167 (3%)	view
12	x 2	optimum_benchmark/preprocessors optimum_benchmark/preprocessors	dataset_preprocessor.py dataset_preprocessor.py	34:47 (3%) 433:446 (3%)	view
12	x 2	optimum_benchmark/generators optimum_benchmark/generators	task_generator.py task_generator.py	108:125 (3%) 136:153 (3%)	view
12	x 2	optimum_benchmark/preprocessors optimum_benchmark/preprocessors	dataset_preprocessor.py dataset_preprocessor.py	72:85 (3%) 433:446 (3%)	view
12	x 2	optimum_benchmark/preprocessors optimum_benchmark/preprocessors	dataset_preprocessor.py dataset_preprocessor.py	72:85 (3%) 154:167 (3%)	view
12	x 2	optimum_benchmark/preprocessors optimum_benchmark/preprocessors	dataset_preprocessor.py dataset_preprocessor.py	154:167 (3%) 433:446 (3%)	view
11	x 2	optimum_benchmark/trackers optimum_benchmark/trackers	energy.py memory.py	161:171 (6%) 129:139 (3%)	view
10	x 2	optimum_benchmark/preprocessors optimum_benchmark/preprocessors	dataset_preprocessor.py dataset_preprocessor.py	74:85 (2%) 292:303 (2%)	view
10	x 2	energy_star energy_star	t5_summarization.yaml t5_text_classification.yaml	11:22 (34%) 11:22 (34%)	view
10	x 2	optimum_benchmark/preprocessors optimum_benchmark/preprocessors	dataset_preprocessor.py dataset_preprocessor.py	156:167 (2%) 292:303 (2%)	view
10	x 2	optimum_benchmark/preprocessors optimum_benchmark/preprocessors	dataset_preprocessor.py dataset_preprocessor.py	112:123 (2%) 292:303 (2%)	view
10	x 2	optimum_benchmark/preprocessors optimum_benchmark/preprocessors	dataset_preprocessor.py dataset_preprocessor.py	292:303 (2%) 435:446 (2%)	view
10	x 2	energy_star energy_star	t5_question_answering.yaml t5_text_classification.yaml	11:22 (37%) 11:22 (34%)	view
10	x 2	optimum_benchmark/preprocessors optimum_benchmark/preprocessors	dataset_preprocessor.py dataset_preprocessor.py	36:47 (2%) 292:303 (2%)	view
10	x 2	energy_star energy_star	t5_question_answering.yaml t5_summarization.yaml	11:22 (37%) 11:22 (34%)	view
10	x 2	energy_star energy_star	t5_text_classification.yaml t5_text_generation.yaml	11:22 (34%) 11:22 (34%)	view
10	x 2	energy_star energy_star	t5_summarization.yaml t5_text_generation.yaml	11:22 (34%) 11:22 (34%)	view
10	x 2	energy_star energy_star	t5_question_answering.yaml t5_text_generation.yaml	11:22 (37%) 11:22 (34%)	view
9	x 2	optimum_benchmark/trackers optimum_benchmark/trackers	latency.py latency.py	213:224 (1%) 600:611 (1%)	view
9	x 2	optimum_benchmark/generators optimum_benchmark/generators	task_generator.py task_generator.py	110:122 (2%) 214:226 (2%)	view
9	x 2	optimum_benchmark/trackers optimum_benchmark/trackers	energy.py energy.py	85:95 (5%) 132:142 (5%)	view
9	x 2	optimum_benchmark/trackers optimum_benchmark/trackers	energy.py latency.py	85:95 (5%) 150:160 (1%)	view
9	x 2	optimum_benchmark/generators optimum_benchmark/generators	task_generator.py task_generator.py	138:150 (2%) 214:226 (2%)	view
9	x 2	optimum_benchmark/generators optimum_benchmark/generators	task_generator.py task_generator.py	101:111 (2%) 282:292 (2%)	view
9	x 2	optimum_benchmark/preprocessors optimum_benchmark/preprocessors	dataset_preprocessor.py dataset_preprocessor.py	139:151 (2%) 418:430 (2%)	view
9	x 2	optimum_benchmark/backends/ipex optimum_benchmark/backends/openvino	backend.py backend.py	84:96 (15%) 115:127 (10%)	view
9	x 2	optimum_benchmark/preprocessors optimum_benchmark/preprocessors	dataset_preprocessor.py dataset_preprocessor.py	307:317 (2%) 334:344 (2%)	view
9	x 2	optimum_benchmark/trackers optimum_benchmark/trackers	energy.py latency.py	132:142 (5%) 150:160 (1%)	view