huggingface / dataset-viewer Duplication

Intro

For duplication, we look at places in code where there are 6 or more lines of code that are exactly the same.
Before duplication is calculated, the code is cleaned to remove empty lines, comments, and frequently duplicated constructs such as imports.
You should aim at having as little as possible (<5%) of duplicated code as high-level of duplication can lead to maintenance difficulties, poor factoring, and logical contradictions.

Learn more...

Duplication Overall

19% duplication:

23,427 cleaned lines of cleaned code (without empty lines, comments, and frequently duplicated constructs such as imports)
4,655 duplicated lines

720 duplicates

dependency graphs: 2D graph | 3D graph | 3D graph (with duplicates)...

Duplication per Extension

Duplication per Component (primary)

Duplication Between Components (50+ lines)

Download: SVG DOT (open online Graphviz editor)

From Component --> To Component	Duplicated Lines	File Pairs	Details
libs (1%) --> services (1%)	303	21 file pairs	details...
jobs (3%) --> services (<1%)	151	14 file pairs	details...
jobs (2%) --> libs (<1%)	129	10 file pairs	details...

Open 2D force graph... Open 3D force graph...

Show more details on duplication between components...

Longest Duplicates

The list of 50 longest duplicates.

See data for all 720 duplicates...

Size	#	Folders	Files	Lines	Code
53	x 2	chart/env chart/env	prod.yaml staging.yaml	17:69 (10%) 14:66 (18%)	view
53	x 2	libs/libcommon/src/libcommon libs/libcommon/src/libcommon	parquet_utils.py parquet_utils.py	231:287 (11%) 363:419 (11%)	view
52	x 2	services/rows/src/rows services/search/src/search	app.py app.py	35:94 (61%) 40:97 (46%)	view
47	x 2	chart/env chart	prod.yaml values.yaml	20:66 (9%) 32:78 (10%)	view
47	x 2	chart/env chart	staging.yaml values.yaml	17:63 (16%) 32:78 (10%)	view
46	x 2	services/api services/webhook	pyproject.toml pyproject.toml	14:66 (79%) 13:65 (80%)	view
40	x 2	services/search/src/search/routes services/search/src/search/routes	filter.py search.py	108:149 (23%) 163:204 (19%)	view
38	x 2	services/api/src/api services/webhook/src/webhook	app.py app.py	45:87 (45%) 34:76 (55%)	view
34	x 2	services/worker/src/worker services/worker/src/worker	main.py start_worker_loop.py	29:67 (69%) 28:66 (68%)	view
33	x 2	chart/env chart/env	prod.yaml staging.yaml	82:120 (6%) 79:117 (11%)	view
29	x 2	chart/templates/services/api chart/templates/services/api	ingress-internal.yaml ingress.yaml	13:41 (70%) 13:41 (70%)	view
26	x 2	services/sse-api services/webhook	pyproject.toml pyproject.toml	41:69 (42%) 37:65 (45%)	view
26	x 2	services/worker/src/worker/job_runners/config services/worker/src/worker/job_runners/dataset	opt_in_out_urls_count.py opt_in_out_urls_count.py	43:70 (43%) 51:78 (38%)	view
26	x 2	services/api services/sse-api	pyproject.toml pyproject.toml	38:66 (44%) 41:69 (42%)	view
24	x 2	services/api services/search	pyproject.toml pyproject.toml	6:33 (41%) 6:33 (43%)	view
23	x 2	chart chart	values.yaml values.yaml	474:503 (5%) 600:629 (5%)	view
23	x 2	chart chart	values.yaml values.yaml	441:470 (5%) 474:503 (5%)	view
23	x 2	chart chart	values.yaml values.yaml	441:470 (5%) 600:629 (5%)	view
21	x 2	chart chart	values.yaml values.yaml	400:424 (4%) 605:629 (4%)	view
21	x 2	chart chart	values.yaml values.yaml	446:470 (4%) 545:569 (4%)	view
21	x 2	libs/libcommon/src/libcommon libs/libcommon/src/libcommon	state.py state.py	280:300 (7%) 350:370 (7%)	view
21	x 2	chart chart	values.yaml values.yaml	400:424 (4%) 446:470 (4%)	view
21	x 2	chart chart	values.yaml values.yaml	400:424 (4%) 479:503 (4%)	view
21	x 2	chart chart	values.yaml values.yaml	479:503 (4%) 545:569 (4%)	view
21	x 2	chart chart	values.yaml values.yaml	400:424 (4%) 545:569 (4%)	view
21	x 2	chart chart	values.yaml values.yaml	545:569 (4%) 605:629 (4%)	view
20	x 2	services/rows services/search	pyproject.toml pyproject.toml	21:44 (33%) 19:42 (36%)	view
20	x 2	chart/templates/services/admin chart/templates/services/admin	ingress-internal.yaml ingress.yaml	13:32 (62%) 13:32 (62%)	view
19	x 2	libs/libcommon/src/libcommon libs/libcommon/src/libcommon	statistics_utils.py statistics_utils.py	397:417 (2%) 443:463 (2%)	view
18	x 2	services/api/src/api services/rows/src/rows	app.py app.py	38:56 (21%) 39:57 (21%)	view
18	x 2	services/search/src/search services/search/src/search	app.py app.py	100:117 (16%) 121:138 (16%)	view
18	x 2	services/worker/src/worker/job_runners/dataset services/worker/src/worker/job_runners/dataset	compatible_libraries.py compatible_libraries.py	420:437 (2%) 485:502 (2%)	view
18	x 2	services/api/src/api services/search/src/search	app.py app.py	38:56 (21%) 44:62 (16%)	view
17	x 2	services/worker/src/worker/job_runners/dataset services/worker/src/worker/job_runners/dataset	compatible_libraries.py compatible_libraries.py	397:413 (2%) 530:546 (2%)	view
17	x 2	chart chart	values.yaml values.yaml	408:424 (3%) 524:540 (3%)	view
17	x 2	services/worker/src/worker/job_runners/dataset services/worker/src/worker/job_runners/dataset	compatible_libraries.py compatible_libraries.py	397:413 (2%) 463:479 (2%)	view
17	x 2	chart chart	values.yaml values.yaml	454:470 (3%) 524:540 (3%)	view
17	x 2	services/admin services/worker	pyproject.toml pyproject.toml	49:67 (29%) 77:95 (19%)	view
17	x 2	services/worker/src/worker/job_runners/dataset services/worker/src/worker/job_runners/dataset	compatible_libraries.py compatible_libraries.py	463:479 (2%) 530:546 (2%)	view
17	x 2	chart chart	values.yaml values.yaml	524:540 (3%) 613:629 (3%)	view
17	x 2	chart chart	values.yaml values.yaml	487:503 (3%) 524:540 (3%)	view
17	x 2	jobs/cache_maintenance services/admin	pyproject.toml pyproject.toml	32:50 (40%) 49:67 (29%)	view
17	x 2	jobs/cache_maintenance services/worker	pyproject.toml pyproject.toml	32:50 (40%) 77:95 (19%)	view
17	x 2	chart chart	values.yaml values.yaml	524:540 (3%) 553:569 (3%)	view
17	x 2	services/search services/webhook	pyproject.toml pyproject.toml	14:33 (30%) 13:32 (29%)	view
16	x 2	services/api/src/api services/webhook/src/webhook	config.py config.py	13:31 (21%) 13:30 (43%)	view
16	x 2	jobs/mongodb_migration/s...db_migration/migrations jobs/mongodb_migration/s...db_migration/migrations	_20230703110100_cache_add_partial_fie... _20230703110100_cache_add_partial_fie...	22:37 (32%) 45:60 (32%)	view
15	x 2	services/rows services/search	pyproject.toml pyproject.toml	51:67 (25%) 47:63 (27%)	view
15	x 2	services/search services/webhook	pyproject.toml pyproject.toml	47:63 (27%) 49:65 (26%)	view
15	x 2	services/worker/src/worker services/worker/src/worker	job_manager.py job_manager.py	211:225 (7%) 234:248 (7%)	view