Intro

For duplication, we look at places in code where there are 6 or more lines of code that are exactly the same.
Before duplication is calculated, the code is cleaned to remove empty lines, comments, and frequently duplicated constructs such as imports.
You should aim at having as little as possible (<5%) of duplicated code as high-level of duplication can lead to maintenance difficulties, poor factoring, and logical contradictions.

Learn more...

Duplication Overall

11% duplication:

6,118 cleaned lines of cleaned code (without empty lines, comments, and frequently duplicated constructs such as imports)
723 duplicated lines

43 duplicates

Duplication per Extension

Duplication per Component (primary)

Longest Duplicates

The list of 20 longest duplicates.

See data for all 43 duplicates...

Size	#	Folders	Files	Lines	Code
35	x 2	torchtext/datasets torchtext/datasets	iwslt2016.py iwslt2016.py	42:76 (17%) 79:113 (17%)	view
24	x 2	torchtext/datasets torchtext/datasets	iwslt2017.py iwslt2017.py	47:70 (14%) 73:96 (14%)	view
19	x 2	torchtext/datasets torchtext/datasets	yahooanswers.py yelpreviewpolarity.py	38:61 (47%) 38:62 (47%)	view
17	x 2	torchtext/datasets torchtext/datasets	iwslt2016.py iwslt2017.py	241:260 (8%) 202:221 (10%)	view
15	x 2	torchtext/datasets torchtext/datasets	iwslt2016.py iwslt2017.py	188:207 (7%) 161:180 (9%)	view
14	x 2	torchtext/datasets torchtext/datasets	wikitext103.py wikitext2.py	37:52 (38%) 37:52 (38%)	view
14	x 2	torchtext/csrc torchtext/csrc	vocab.cpp vocab.cpp	252:273 (6%) 298:317 (6%)	view
13	x 2	torchtext/datasets torchtext/experimental/datasets/raw	iwslt2017.py wmt14.py	98:114 (7%) 42:58 (13%)	view
13	x 2	torchtext/datasets torchtext/experimental/datasets/raw	iwslt2016.py wmt14.py	126:142 (6%) 42:58 (13%)	view
13	x 2	torchtext/datasets torchtext/datasets	iwslt2016.py iwslt2017.py	126:142 (6%) 98:114 (7%)	view
12	x 2	torchtext/datasets torchtext/datasets	amazonreviewfull.py dbpedia.py	43:59 (32%) 39:55 (36%)	view
11	x 2	torchtext/datasets torchtext/datasets	squad1.py squad2.py	38:50 (33%) 38:50 (33%)	view
10	x 2	torchtext/models/roberta torchtext/models/roberta	bundler.py bundler.py	171:182 (5%) 201:212 (5%)	view
10	x 2	torchtext/models/roberta torchtext/models/roberta	bundler.py bundler.py	236:247 (5%) 282:293 (5%)	view
10	x 2	torchtext/datasets torchtext/datasets	iwslt2016.py iwslt2017.py	217:228 (5%) 182:193 (6%)	view
9	x 2	torchtext/data torchtext/data	datasets_utils.py datasets_utils.py	208:217 (3%) 244:253 (3%)	view
9	x 2	torchtext/datasets torchtext/datasets	iwslt2016.py iwslt2017.py	263:274 (4%) 223:234 (5%)	view
9	x 2	torchtext/csrc torchtext/csrc	register_pybindings.cpp register_torchbindings.cpp	70:78 (6%) 44:52 (7%)	view
8	x 2	torchtext/models/roberta torchtext/models/roberta	bundler.py bundler.py	218:225 (4%) 264:271 (4%)	view
8	x 2	torchtext/experimental torchtext/experimental	transforms.py transforms.py	110:126 (5%) 144:160 (5%)	view

Duplicated Units

The list of top 4 duplicated units.

See data for all 4 unit duplicates...

Size	#	Folders	Files	Lines	Code
19	x 2	torchtext/datasets torchtext/datasets	yelpreviewpolarity.py yahooanswers.py	0:0 0:0	view
16	x 2	torchtext/datasets torchtext/datasets	wikitext103.py wikitext2.py	0:0 0:0	view
12	x 2	torchtext/datasets torchtext/datasets	squad1.py squad2.py	0:0 0:0	view
6	x 3	torchtext/experimental/datasets/raw torchtext/datasets torchtext/datasets	wmt14.py iwslt2016.py iwslt2017.py	0:0 0:0 0:0	view