openai / deeptype - Duplication

Intro

For duplication, we look at places in code where there are 6 or more lines of code that are exactly the same.
Before duplication is calculated, the code is cleaned to remove empty lines, comments, and frequently duplicated constructs such as imports.
You should aim at having as little as possible (<5%) of duplicated code as high-level of duplication can lead to maintenance difficulties, poor factoring, and logical contradictions.

Learn more...

Duplication Overall

6% duplication:

10,666 cleaned lines of cleaned code (without empty lines, comments, and frequently duplicated constructs such as imports)
659 duplicated lines

57 duplicates

dependency graphs: 2D graph | 3D graph | 3D graph (with duplicates)...

Duplication per Extension

Duplication per Component (primary)

Duplication Between Components (50+ lines)

Download: SVG DOT (open online Graphviz editor)

From Component --> To Component	Duplicated Lines	File Pairs	Details
extraction (<1%) --> learning (1%)	82	1 file pair	details...

Open 2D force graph... Open 3D force graph...

Show more details on duplication between components...

Longest Duplicates

The list of 50 longest duplicates.

See data for all 57 duplicates...

Size	#	Folders	Files	Lines	Code
21	x 2	learning learning	batchifier.py batchifier.py	201:224 (4%) 372:395 (4%)	view
20	x 2	extraction learning	TypeSystemToNeuralTypeSystem.ipynb SentencePredictions.ipynb	233:252 (7%) 277:296 (6%)	view
19	x 2	extraction extraction	evaluate_type_system.py produce_wikidata_tsv.py	326:346 (5%) 261:280 (7%)	view
16	x 2	wikidata_linker_utils_sr...n/wikidata_linker_utils wikidata_linker_utils_sr...n/wikidata_linker_utils	fast_disambiguate.pyx fast_disambiguate.pyx	423:439 (2%) 615:631 (2%)	view
15	x 2	wikidata_linker_utils_sr...n/wikidata_linker_utils wikidata_linker_utils_sr...n/wikidata_linker_utils	fast_disambiguate.pyx fast_disambiguate.pyx	289:303 (2%) 615:629 (2%)	view
15	x 2	wikidata_linker_utils_sr...n/wikidata_linker_utils wikidata_linker_utils_sr...n/wikidata_linker_utils	fast_disambiguate.pyx fast_disambiguate.pyx	289:303 (2%) 423:437 (2%)	view
12	x 2	learning learning	train_type.py train_type.py	2396:2407 (<1%) 2414:2425 (<1%)	view
11	x 2	wikidata_linker_utils_sr...n/wikidata_linker_utils wikidata_linker_utils_sr...n/wikidata_linker_utils	fast_disambiguate.pyx fast_disambiguate.pyx	644:654 (1%) 663:673 (1%)	view
10	x 2	wikidata_linker_utils_sr...n/wikidata_linker_utils wikidata_linker_utils_sr...n/wikidata_linker_utils	successor_mask.pyx successor_mask.pyx	954:963 (1%) 1039:1048 (1%)	view
9	x 2	extraction extraction	TypeSystemToNeuralTypeSystem.ipynb TypeSystemToNeuralTypeSystem.ipynb	186:194 (3%) 225:233 (3%)	view
9	x 2	extraction extraction	get_redirection_category_links.py get_redirection_category_links.py	37:46 (7%) 60:69 (7%)	view
9	x 2	learning learning	dataset.py dataset.py	102:110 (6%) 158:166 (6%)	view
9	x 2	wikidata_linker_utils_sr...n/wikidata_linker_utils wikidata_linker_utils_sr...n/wikidata_linker_utils	successor_mask.pyx successor_mask.pyx	920:929 (<1%) 1004:1013 (<1%)	view
9	x 2	wikidata_linker_utils_sr...n/wikidata_linker_utils wikidata_linker_utils_sr...n/wikidata_linker_utils	fast_disambiguate.pyx fast_disambiguate.pyx	234:242 (1%) 480:488 (1%)	view
9	x 2	wikidata_linker_utils_sr...n/wikidata_linker_utils wikidata_linker_utils_sr...n/wikidata_linker_utils	successor_mask.pyx successor_mask.pyx	966:974 (<1%) 1054:1062 (<1%)	view
8	x 2	extraction extraction	evaluate_type_system.py project_graph.py	444:453 (2%) 173:183 (5%)	view
8	x 2	wikidata_linker_utils_sr...n/wikidata_linker_utils wikidata_linker_utils_sr...n/wikidata_linker_utils	successor_mask.pyx successor_mask.pyx	127:134 (<1%) 255:262 (<1%)	view
8	x 2	extraction/classifiers extraction/classifiers	country_classifier.py location_classifier.py	32:39 (<1%) 62:69 (8%)	view
8	x 2	extraction extraction	produce_wikidata_tsv.py produce_wikidata_tsv.py	116:123 (3%) 198:205 (3%)	view
8	x 2	wikidata_linker_utils_sr...n/wikidata_linker_utils wikidata_linker_utils_sr...n/wikidata_linker_utils	successor_mask.pyx successor_mask.pyx	945:952 (<1%) 1030:1037 (<1%)	view
8	x 2	wikidata_linker_utils_sr...n/wikidata_linker_utils wikidata_linker_utils_sr...n/wikidata_linker_utils	fast_disambiguate.pyx successor_mask.pyx	347:354 (1%) 711:721 (<1%)	view
7	x 2	extraction extraction	produce_wikidata_tsv.py produce_wikidata_tsv.py	198:204 (2%) 218:224 (2%)	view
7	x 2	extraction/classifiers extraction/classifiers	location_classifier.py time_classifier.py	6:22 (7%) 6:22 (23%)	view
7	x 2	learning learning	batchifier.py batchifier.py	409:415 (1%) 530:536 (1%)	view
7	x 2	extraction extraction	produce_wikidata_tsv.py produce_wikidata_tsv.py	116:122 (2%) 218:224 (2%)	view
7	x 2	learning learning	train_type.py train_type.py	2138:2144 (<1%) 2306:2312 (<1%)	view
7	x 2	extraction/classifiers extraction/classifiers	country_classifier.py location_classifier.py	6:21 (<1%) 6:22 (7%)	view
7	x 2	wikidata_linker_utils_sr...n/wikidata_linker_utils wikidata_linker_utils_sr...n/wikidata_linker_utils	fast_disambiguate.pyx fast_disambiguate.pyx	476:482 (1%) 584:590 (1%)	view
7	x 2	extraction/classifiers extraction/classifiers	country_classifier.py time_classifier.py	6:21 (<1%) 6:22 (23%)	view
6	x 2	wikidata_linker_utils_sr...n/wikidata_linker_utils wikidata_linker_utils_sr...n/wikidata_linker_utils	fast_disambiguate.pyx fast_disambiguate.pyx	585:590 (1%) 664:669 (1%)	view
6	x 2	wikidata_linker_utils_sr...n/wikidata_linker_utils wikidata_linker_utils_sr...n/wikidata_linker_utils	fast_disambiguate.pyx fast_disambiguate.pyx	477:482 (1%) 664:669 (1%)	view
6	x 2	learning learning	batchifier.py batchifier.py	468:473 (1%) 503:508 (1%)	view
6	x 2	extraction extraction	evaluate_type_system.py produce_wikidata_tsv.py	184:189 (1%) 219:224 (2%)	view
6	x 2	learning learning	train_type.py train_type.py	1101:1106 (<1%) 1177:1182 (<1%)	view
6	x 2	extraction learning	TypeSystemToNeuralTypeSystem.ipynb SentencePredictions.ipynb	16:21 (2%) 166:171 (1%)	view
6	x 2	extraction extraction	evaluate_type_system.py produce_wikidata_tsv.py	184:189 (1%) 117:122 (2%)	view
6	x 2	learning learning	SentencePredictions.ipynb SentencePredictions.ipynb	179:184 (1%) 228:233 (1%)	view
6	x 2	extraction extraction	evaluate_type_system.py produce_wikidata_tsv.py	184:189 (1%) 199:204 (2%)	view
6	x 2	wikidata_linker_utils_sr...n/wikidata_linker_utils wikidata_linker_utils_sr...n/wikidata_linker_utils	successor_mask.pyx successor_mask.pyx	57:62 (<1%) 956:961 (<1%)	view
6	x 2	learning learning	batchifier.py batchifier.py	428:433 (1%) 489:494 (1%)	view
6	x 2	wikidata_linker_utils_sr...n/wikidata_linker_utils wikidata_linker_utils_sr...n/wikidata_linker_utils	successor_mask.pyx successor_mask.pyx	57:62 (<1%) 1041:1046 (<1%)	view
6	x 2	learning learning	SentencePredictions.ipynb SentencePredictions.ipynb	27:32 (1%) 191:196 (1%)	view
6	x 2	learning learning	train_type.py train_type.py	2418:2423 (<1%) 2468:2473 (<1%)	view
6	x 2	extraction learning	TypeSystemToNeuralTypeSystem.ipynb SentencePredictions.ipynb	16:21 (2%) 191:196 (1%)	view
6	x 2	extraction learning	TypeSystemToNeuralTypeSystem.ipynb SentencePredictions.ipynb	16:21 (2%) 27:32 (1%)	view
6	x 2	learning learning	train_type.py train_type.py	2400:2405 (<1%) 2468:2473 (<1%)	view
6	x 2	wikidata_linker_utils_sr...n/wikidata_linker_utils wikidata_linker_utils_sr...n/wikidata_linker_utils	fast_disambiguate.pyx fast_disambiguate.pyx	251:256 (1%) 495:500 (1%)	view
6	x 2	learning learning	train_type.py train_type.py	645:650 (<1%) 655:660 (<1%)	view
6	x 2	learning learning	SentencePredictions.ipynb SentencePredictions.ipynb	166:171 (1%) 191:196 (1%)	view
6	x 2	extraction learning	TypeSystemToNeuralTypeSystem.ipynb SentencePredictions.ipynb	205:210 (2%) 228:233 (1%)	view