huggingface / lighteval Duplication

Intro

For duplication, we look at places in code where there are 6 or more lines of code that are exactly the same.
Before duplication is calculated, the code is cleaned to remove empty lines, comments, and frequently duplicated constructs such as imports.
You should aim at having as little as possible (<5%) of duplicated code as high-level of duplication can lead to maintenance difficulties, poor factoring, and logical contradictions.

Learn more...

Duplication Overall

19% duplication:

22,959 cleaned lines of cleaned code (without empty lines, comments, and frequently duplicated constructs such as imports)
4,514 duplicated lines

2,461 duplicates

dependency graphs: 2D graph | 3D graph | 3D graph (with duplicates)...

Duplication per Extension

Duplication per Component (primary)

Duplication Between Components (50+ lines)

Download: SVG DOT (open online Graphviz editor)

From Component --> To Component	Duplicated Lines	File Pairs	Details
community_tasks (1%) --> src (<1%)	54	2 file pairs	details...

Open 2D force graph... Open 3D force graph...

Show more details on duplication between components...

Longest Duplicates

The list of 50 longest duplicates.

See data for all 2,461 duplicates...

Size	#	Folders	Files	Lines	Code
63	x 2	src/lighteval src/lighteval	main_endpoint.py main_sglang.py	163:238 (16%) 41:115 (62%)	view
62	x 2	src/lighteval src/lighteval	main_accelerate.py main_endpoint.py	50:126 (55%) 51:122 (15%)	view
62	x 2	src/lighteval src/lighteval	main_sglang.py main_vllm.py	44:117 (61%) 47:120 (59%)	view
62	x 2	src/lighteval src/lighteval	main_accelerate.py main_vllm.py	51:127 (55%) 47:120 (59%)	view
62	x 2	src/lighteval src/lighteval	main_accelerate.py main_sglang.py	51:127 (55%) 44:117 (61%)	view
61	x 2	src/lighteval src/lighteval	main_endpoint.py main_vllm.py	166:238 (15%) 47:118 (58%)	view
61	x 2	src/lighteval src/lighteval	main_endpoint.py main_vllm.py	53:122 (15%) 47:118 (58%)	view
61	x 2	src/lighteval src/lighteval	main_accelerate.py main_endpoint.py	51:126 (54%) 166:238 (15%)	view
61	x 2	src/lighteval src/lighteval	main_endpoint.py main_sglang.py	53:122 (15%) 44:115 (60%)	view
61	x 2	src/lighteval src/lighteval	main_endpoint.py main_endpoint.py	53:122 (15%) 166:238 (15%)	view
51	x 2	src/lighteval src/lighteval	main_endpoint.py main_sglang.py	282:335 (13%) 40:93 (50%)	view
50	x 2	src/lighteval src/lighteval	main_endpoint.py main_endpoint.py	163:215 (12%) 283:335 (12%)	view
48	x 2	src/lighteval src/lighteval	main_endpoint.py main_vllm.py	286:335 (12%) 47:96 (45%)	view
48	x 2	src/lighteval src/lighteval	main_accelerate.py main_endpoint.py	51:100 (43%) 286:335 (12%)	view
48	x 2	src/lighteval src/lighteval	main_endpoint.py main_endpoint.py	53:102 (12%) 286:335 (12%)	view
44	x 2	src/lighteval/models/transformers src/lighteval/models/transformers	transformers_model.py vlm_transformers_model.py	316:364 (6%) 200:249 (16%)	view
39	x 2	src/lighteval src/lighteval	main_endpoint.py main_vllm.py	420:459 (10%) 60:99 (37%)	view
39	x 2	src/lighteval src/lighteval	main_endpoint.py main_sglang.py	420:459 (10%) 57:96 (38%)	view
39	x 2	src/lighteval src/lighteval	main_accelerate.py main_endpoint.py	64:103 (35%) 420:459 (10%)	view
39	x 2	src/lighteval src/lighteval	main_endpoint.py main_endpoint.py	179:218 (10%) 420:459 (10%)	view
39	x 2	src/lighteval src/lighteval	main_endpoint.py main_endpoint.py	66:105 (10%) 420:459 (10%)	view
36	x 2	src/lighteval src/lighteval	main_endpoint.py main_endpoint.py	299:335 (9%) 420:456 (9%)	view
34	x 2	src/lighteval/tasks/multilingual src/lighteval/tasks/multilingual	tasks.py tasks.py	1752:1785 (<1%) 2354:2387 (<1%)	view
34	x 2	src/lighteval/tasks/multilingual src/lighteval/tasks/multilingual	tasks.py tasks.py	642:675 (<1%) 2661:2694 (<1%)	view
27	x 2	src/lighteval/tasks/multilingual src/lighteval/tasks/multilingual	tasks.py tasks.py	3019:3045 (<1%) 3678:3704 (<1%)	view
21	x 2	src/lighteval src/lighteval	main_endpoint.py main_endpoint.py	131:158 (5%) 248:275 (5%)	view
19	x 2	src/lighteval/models/sglang src/lighteval/models/vllm	sglang_model.py vllm_model.py	209:239 (7%) 282:313 (4%)	view
19	x 2	src/lighteval/tasks/multilingual src/lighteval/tasks/multilingual	tasks.py tasks.py	82:101 (<1%) 137:156 (<1%)	view
18	x 2	src/lighteval/tasks/multilingual src/lighteval/tasks/multilingual	tasks.py tasks.py	2550:2567 (<1%) 3223:3240 (<1%)	view
18	x 2	src/lighteval/models/nanotron src/lighteval/models/nanotron	nanotron_model.py nanotron_model.py	670:691 (1%) 908:928 (1%)	view
18	x 2	src/lighteval/tasks/multilingual src/lighteval/tasks/multilingual	tasks.py tasks.py	3178:3196 (<1%) 3635:3653 (<1%)	view
18	x 2	src/lighteval/tasks/multilingual src/lighteval/tasks/multilingual	tasks.py tasks.py	1337:1354 (<1%) 3068:3085 (<1%)	view
18	x 2	src/lighteval/tasks/multilingual src/lighteval/tasks/multilingual	tasks.py tasks.py	2400:2417 (<1%) 3107:3124 (<1%)	view
16	x 2	src/lighteval/tasks/multilingual src/lighteval/tasks/multilingual	tasks.py tasks.py	3072:3088 (<1%) 3180:3196 (<1%)	view
16	x 2	src/lighteval/tasks/multilingual src/lighteval/tasks/multilingual	tasks.py tasks.py	3178:3193 (<1%) 3562:3577 (<1%)	view
16	x 2	src/lighteval/models/endpoints src/lighteval/models	inference_providers_model.py litellm_model.py	226:247 (13%) 291:312 (9%)	view
16	x 2	src/lighteval/tasks/multilingual src/lighteval/tasks/multilingual	tasks.py tasks.py	2013:2028 (<1%) 2045:2060 (<1%)	view
16	x 2	src/lighteval/tasks/multilingual src/lighteval/tasks/multilingual	tasks.py tasks.py	2525:2540 (<1%) 2591:2606 (<1%)	view
16	x 2	src/lighteval/tasks/multilingual src/lighteval/tasks/multilingual	tasks.py tasks.py	3072:3088 (<1%) 3637:3653 (<1%)	view
16	x 2	src/lighteval/metrics src/lighteval/metrics	metrics_sample.py metrics_sample.py	1195:1212 (2%) 1326:1343 (2%)	view
16	x 2	src/lighteval/tasks/multilingual src/lighteval/tasks/multilingual	tasks.py tasks.py	3562:3577 (<1%) 3635:3650 (<1%)	view
16	x 2	src/lighteval/tasks/multilingual src/lighteval/tasks/multilingual	tasks.py tasks.py	2489:2504 (<1%) 2591:2606 (<1%)	view
16	x 2	src/lighteval/tasks/multilingual src/lighteval/tasks/multilingual	tasks.py tasks.py	2489:2504 (<1%) 2525:2540 (<1%)	view
15	x 2	src/lighteval/tasks/multilingual src/lighteval/tasks/multilingual	tasks.py tasks.py	3110:3124 (<1%) 3144:3158 (<1%)	view
15	x 2	src/lighteval/tasks/multilingual src/lighteval/tasks/multilingual	tasks.py tasks.py	2403:2417 (<1%) 3144:3158 (<1%)	view
14	x 2	src/lighteval/tasks/multilingual src/lighteval/tasks/multilingual	tasks.py tasks.py	3072:3085 (<1%) 3145:3158 (<1%)	view
14	x 2	src/lighteval/tasks/multilingual src/lighteval/tasks/multilingual	tasks.py tasks.py	3145:3158 (<1%) 3637:3650 (<1%)	view
14	x 2	src/lighteval/tasks/multilingual src/lighteval/tasks/multilingual	tasks.py tasks.py	1341:1354 (<1%) 3564:3577 (<1%)	view
14	x 2	src/lighteval/tasks/multilingual src/lighteval/tasks/multilingual	tasks.py tasks.py	3145:3158 (<1%) 3564:3577 (<1%)	view
14	x 2	src/lighteval/tasks/multilingual src/lighteval/tasks/multilingual	tasks.py tasks.py	3111:3124 (<1%) 3637:3650 (<1%)	view

Duplicated Units

The list of top 1 duplicated units.

See data for all 1 unit duplicate

Size	#	Folders	Files	Lines	Code
37	x 2	src/lighteval/models/transformers src/lighteval/models/transformers	vlm_transformers_model.py transformers_model.py	0:0 0:0	view