mozilla / smart-tab-grouping

Intro

For duplication, we look at places in code where there are 6 or more lines of code that are exactly the same.
Before duplication is calculated, the code is cleaned to remove empty lines, comments, and frequently duplicated constructs such as imports.
You should aim at having as little as possible (<5%) of duplicated code as high-level of duplication can lead to maintenance difficulties, poor factoring, and logical contradictions.

Learn more...

Duplication Overall

52% duplication:

12,675 cleaned lines of cleaned code (without empty lines, comments, and frequently duplicated constructs such as imports)
6,665 duplicated lines

6,991 duplicates

dependency graphs: 2D graph | 3D graph | 3D graph (with duplicates)...

Duplication per Extension

Duplication per Component (primary)

Longest Duplicates

The list of 50 longest duplicates.

See data for all 6,991 duplicates...

Size	#	Folders	Files	Lines	Code
243	x 2	src/jobs/util src/util	grouping_pipeline.py grouping_pipeline.py	29:363 (100%) 29:363 (100%)	view
164	x 2	src/jobs/util src/util	tab_titles.py tab_titles.py	17:226 (100%) 17:226 (100%)	view
108	x 2	src/jobs/util src/util	labeled_data_utils.py labeled_data_utils.py	6:148 (100%) 6:148 (100%)	view
106	x 2	src/jobs/util src/util	language_model.py language_model.py	16:172 (100%) 16:172 (100%)	view
91	x 2	src/jobs/util src/util	shorten_topic_length.py shorten_topic_length.py	7:112 (100%) 7:112 (100%)	view
86	x 2	notebooks notebooks	Common Crawl.ipynb Common Crawl.ipynb	572:657 (3%) 2107:2192 (3%)	view
70	x 2	notebooks notebooks	Common Crawl.ipynb Common Crawl.ipynb	1097:1166 (2%) 2140:2209 (2%)	view
69	x 2	src/jobs/util src/util	key_document_finder.py key_document_finder.py	10:87 (100%) 10:87 (100%)	view
62	x 2	src/jobs/util src/util	silhouette.py silhouette.py	9:127 (100%) 9:127 (100%)	view
62	x 2	src/jobs/util src/util	secrets.py secrets.py	9:98 (100%) 9:98 (100%)	view
53	x 2	notebooks notebooks	Common Crawl.ipynb Common Crawl.ipynb	605:657 (1%) 1097:1149 (1%)	view
53	x 2	notebooks notebooks	Common Crawl.ipynb Common Crawl.ipynb	553:605 (1%) 2545:2597 (1%)	view
52	x 2	notebooks notebooks	Common Crawl.ipynb Common Crawl.ipynb	1167:1218 (1%) 2229:2280 (1%)	view
46	x 2	notebooks notebooks	Common Crawl.ipynb Common Crawl.ipynb	1321:1366 (1%) 2365:2410 (1%)	view
41	x 2	src/jobs/util src/util	evaluate.py evaluate.py	10:57 (100%) 10:57 (100%)	view
39	x 2	notebooks notebooks	Analyze Clusters.ipynb Analyze Clusters.ipynb	222:260 (1%) 305:343 (1%)	view
39	x 2	notebooks notebooks	Analyze Clusters.ipynb Analyze Clusters.ipynb	305:343 (1%) 390:428 (1%)	view
39	x 2	notebooks notebooks	Analyze Clusters.ipynb Analyze Clusters.ipynb	222:260 (1%) 390:428 (1%)	view
38	x 2	notebooks notebooks	AnalyzeTrainingData.ipynb Directed Training Clusters.ipynb	158:195 (19%) 322:359 (10%)	view
36	x 2	notebooks notebooks	Extractive Test.ipynb Extractive Test.ipynb	247:282 (4%) 620:655 (4%)	view
35	x 2	notebooks notebooks	Common Crawl.ipynb Common Crawl.ipynb	528:562 (1%) 1461:1495 (1%)	view
35	x 2	notebooks notebooks	Common Crawl.ipynb Common Crawl.ipynb	1771:1805 (1%) 2281:2315 (1%)	view
34	x 2	notebooks notebooks	Extractive Test.ipynb Extractive Test.ipynb	53:86 (4%) 402:435 (4%)	view
34	x 2	src/jobs/util src/util	topic_utils.py topic_utils.py	7:46 (100%) 7:46 (100%)	view
34	x 2	src/jobs src/jobs	tune_bart.py tune_gpt2.py	90:126 (23%) 75:111 (28%)	view
34	x 2	notebooks notebooks	Common Crawl.ipynb Common Crawl.ipynb	2107:2140 (1%) 2564:2597 (1%)	view
32	x 2	notebooks notebooks	Common Crawl.ipynb Common Crawl.ipynb	335:366 (1%) 2364:2395 (1%)	view
32	x 2	notebooks notebooks	Common Crawl.ipynb Common Crawl.ipynb	913:944 (1%) 2477:2508 (1%)	view
31	x 2	notebooks notebooks	Common Crawl.ipynb Common Crawl.ipynb	401:431 (1%) 2430:2460 (1%)	view
31	x 2	notebooks notebooks	Common Crawl.ipynb Common Crawl.ipynb	336:366 (1%) 1321:1351 (1%)	view
30	x 2	notebooks notebooks	Common Crawl.ipynb Common Crawl.ipynb	1511:1540 (1%) 2556:2585 (1%)	view
30	x 2	notebooks notebooks	Common Crawl.ipynb Common Crawl.ipynb	564:593 (1%) 1511:1540 (1%)	view
29	x 2	notebooks notebooks	Analyze Clusters.ipynb Analyze Clusters.ipynb	1978:2006 (1%) 2383:2411 (1%)	view
29	x 2	notebooks notebooks	Common Crawl.ipynb Common Crawl.ipynb	314:342 (1%) 1822:1850 (1%)	view
28	x 2	notebooks notebooks	Common Crawl.ipynb Common Crawl.ipynb	284:311 (<1%) 2274:2301 (<1%)	view
28	x 2	notebooks notebooks	Analyze Clusters.ipynb Analyze Clusters.ipynb	1318:1345 (1%) 1863:1890 (1%)	view
27	x 2	notebooks notebooks	Benchmarking.ipynb Common Crawl.ipynb	498:524 (5%) 2779:2805 (<1%)	view
27	x 2	notebooks notebooks	AnalyzeTrainingData.ipynb Extractive Test.ipynb	169:195 (13%) 734:760 (3%)	view
27	x 2	notebooks notebooks	Benchmarking.ipynb Onnx.ipynb	498:524 (5%) 31:57 (47%)	view
27	x 2	notebooks notebooks	Common Crawl.ipynb Common Crawl.ipynb	777:803 (<1%) 1322:1348 (<1%)	view
27	x 2	notebooks notebooks	Directed Training Clusters.ipynb Extractive Test.ipynb	333:359 (7%) 734:760 (3%)	view
27	x 2	notebooks notebooks	Common Crawl.ipynb Onnx.ipynb	2779:2805 (<1%) 31:57 (47%)	view
27	x 2	notebooks notebooks	Analyze Clusters.ipynb Analyze Clusters.ipynb	1435:1461 (1%) 1863:1889 (1%)	view
27	x 2	notebooks notebooks	Analyze Clusters.ipynb Analyze Clusters.ipynb	1318:1344 (1%) 1435:1461 (1%)	view
27	x 2	notebooks notebooks	Extractive Test.ipynb Gen Dissimilar Titles & Keywords.ipynb	734:760 (3%) 1259:1285 (2%)	view
27	x 2	notebooks notebooks	Common Crawl.ipynb Common Crawl.ipynb	777:803 (<1%) 2366:2392 (<1%)	view
27	x 2	notebooks notebooks	AnalyzeTrainingData.ipynb Gen Dissimilar Titles & Keywords.ipynb	169:195 (13%) 1259:1285 (2%)	view
27	x 2	notebooks notebooks	Directed Training Clusters.ipynb Gen Dissimilar Titles & Keywords.ipynb	333:359 (7%) 1259:1285 (2%)	view
27	x 2	notebooks notebooks	Common Crawl.ipynb Common Crawl.ipynb	337:363 (<1%) 777:803 (<1%)	view
27	x 2	notebooks notebooks	Analyze Clusters.ipynb Directed Training Clusters.ipynb	2613:2639 (1%) 333:359 (7%)	view

Duplicated Units

The list of top 11 duplicated units.

See data for all 11 unit duplicates...

Size	#	Folders	Files	Lines	Code
58	x 2	src/jobs/util src/util	labeled_data_utils.py labeled_data_utils.py	0:0 0:0	view
63	x 2	src/jobs/util src/util	grouping_pipeline.py grouping_pipeline.py	0:0 0:0	view
45	x 2	src/jobs/util src/util	shorten_topic_length.py shorten_topic_length.py	0:0 0:0	view
30	x 2	src/jobs/util src/util	language_model.py language_model.py	0:0 0:0	view
25	x 2	src/jobs/util src/util	tab_titles.py tab_titles.py	0:0 0:0	view
24	x 2	src/jobs/util src/util	evaluate.py evaluate.py	0:0 0:0	view
41	x 2	src/jobs/util src/util	silhouette.py silhouette.py	0:0 0:0	view
19	x 2	src/jobs/util src/util	topic_utils.py topic_utils.py	0:0 0:0	view
14	x 2	src/jobs/util src/util	key_document_finder.py key_document_finder.py	0:0 0:0	view
14	x 2	src/jobs/util src/util	storage.py storage.py	0:0 0:0	view
11	x 2	src/jobs/util src/util	secrets.py secrets.py	0:0 0:0	view