G

Intro

For duplication, we look at places in code where there are 6 or more lines of code that are exactly the same.
Before duplication is calculated, the code is cleaned to remove empty lines, comments, and frequently duplicated constructs such as imports.
You should aim at having as little as possible (<5%) of duplicated code as high-level of duplication can lead to maintenance difficulties, poor factoring, and logical contradictions.

Learn more...

Duplication Overall

15% duplication:

14,250 cleaned lines of cleaned code (without empty lines, comments, and frequently duplicated constructs such as imports)
2,219 duplicated lines

160 duplicates

Duplication per Extension

Duplication per Component (primary)

Duplication Between Components (50+ lines)

Download: SVG DOT (open online Graphviz editor)

From Component --> To Component	Duplicated Lines	File Pairs	Details
XLM/src/model (8%) --> pytorch-transformers/pytorch_transformers (<1%)	117	2 file pairs	details...
XLM/src/evaluation (3%) --> pytorch-transformers (3%)	72	1 file pair	details...

Open 3D force graph...

Show more details on duplication between components...

Longest Duplicates

The list of 20 longest duplicates.

See data for all 160 duplicates...

Size	#	Folders	Files	Lines	Code
42	x 2	pytorch-transformers/pytorch_transformers pytorch-transformers/pytorch_transformers	tokenization_openai.py tokenization_xlm.py	111:156 (30%) 611:656 (6%)	view
33	x 2	pytorch-transformers/pytorch_transformers pytorch-transformers/pytorch_transformers	modeling_utils.py modeling_xlnet.py	780:831 (6%) 1231:1282 (4%)	view
30	x 2	pytorch-transformers/pytorch_transformers pytorch-transformers/pytorch_transformers	tokenization_xlm.py tokenization_xlm.py	99:128 (4%) 130:159 (4%)	view
28	x 2	XLM/src/model pytorch-transformers/pytorch_transformers	transformer.py modeling_xlm.py	180:215 (6%) 311:346 (5%)	view
26	x 2	pytorch-transformers/pytorch_transformers pytorch-transformers/pytorch_transformers	modeling_gpt2.py modeling_openai.py	189:221 (6%) 206:237 (6%)	view
26	x 2	XLM/src/evaluation pytorch-transformers	evaluator.py add_umt_subqs_subas_to_q_squad_format...	639:664 (5%) 44:69 (6%)	view
26	x 2	pytorch-transformers/pytorch_transformers pytorch-transformers/pytorch_transformers	tokenization_gpt2.py tokenization_xlm.py	146:172 (17%) 626:652 (3%)	view
26	x 2	pytorch-transformers/pytorch_transformers pytorch-transformers/pytorch_transformers	tokenization_gpt2.py tokenization_openai.py	146:172 (17%) 126:152 (18%)	view
23	x 2	pytorch-transformers/pytorch_transformers pytorch-transformers/pytorch_transformers	modeling_transfo_xl.py modeling_xlnet.py	152:175 (2%) 153:176 (3%)	view
21	x 2	pytorch-transformers/pytorch_transformers pytorch-transformers/pytorch_transformers	modeling_bert.py modeling_roberta.py	1209:1235 (3%) 413:439 (12%)	view
20	x 2	pytorch-transformers/pytorch_transformers pytorch-transformers/pytorch_transformers	modeling_transfo_xl_utilities.py modeling_transfo_xl_utilities.py	121:144 (8%) 215:237 (8%)	view
19	x 2	pytorch-transformers/pytorch_transformers pytorch-transformers/pytorch_transformers	tokenization_gpt2.py tokenization_openai.py	202:224 (13%) 186:208 (13%)	view
19	x 2	pytorch-transformers/pytorch_transformers pytorch-transformers/pytorch_transformers	modeling_distilbert.py modeling_xlm.py	177:198 (4%) 274:295 (3%)	view
19	x 2	pytorch-transformers/pytorch_transformers pytorch-transformers/pytorch_transformers	modeling_gpt2.py modeling_openai.py	268:291 (5%) 282:305 (5%)	view
17	x 2	pytorch-transformers/pytorch_transformers pytorch-transformers/pytorch_transformers	modeling_gpt2.py modeling_openai.py	732:751 (4%) 694:713 (4%)	view
16	x 2	pytorch-transformers/pytorch_transformers pytorch-transformers/pytorch_transformers	modeling_transfo_xl.py modeling_transfo_xl.py	598:626 (1%) 690:717 (1%)	view
15	x 2	pytorch-transformers/pytorch_transformers pytorch-transformers/pytorch_transformers	modeling_transfo_xl.py modeling_xlnet.py	294:312 (1%) 315:333 (2%)	view
15	x 2	XLM/src/evaluation XLM/src/evaluation	evaluator.py evaluator.py	335:354 (3%) 410:429 (3%)	view
13	x 2	pytorch-transformers/pytorch_transformers pytorch-transformers/pytorch_transformers	modeling_gpt2.py modeling_openai.py	246:263 (3%) 261:276 (3%)	view
13	x 2	pytorch-transformers/pytorch_transformers pytorch-transformers/pytorch_transformers	tokenization_gpt2.py tokenization_roberta.py	17:45 (8%) 17:45 (28%)	view

Duplicated Units

The list of top 7 duplicated units.

See data for all 7 unit duplicates...

Size	#	Folders	Files	Lines	Code
38	x 2	pytorch-transformers/pytorch_transformers pytorch-transformers/pytorch_transformers	tokenization_xlm.py tokenization_openai.py	0:0 0:0	view
19	x 2	pytorch-transformers/pytorch_transformers pytorch-transformers/pytorch_transformers	tokenization_gpt2.py tokenization_openai.py	0:0 0:0	view
19	x 2	pytorch-transformers/pytorch_transformers pytorch-transformers/pytorch_transformers	modeling_distilbert.py modeling_xlm.py	0:0 0:0	view
8	x 3	pytorch-transformers/pytorch_transformers pytorch-transformers/pytorch_transformers XLM/src/model	modeling_distilbert.py modeling_xlm.py transformer.py	0:0 0:0 0:0	view
11	x 2	pytorch-transformers/pytorch_transformers pytorch-transformers/pytorch_transformers	modeling_gpt2.py modeling_openai.py	0:0 0:0	view
10	x 2	pytorch-transformers/pytorch_transformers pytorch-transformers/pytorch_transformers	tokenization_xlm.py tokenization_openai.py	0:0 0:0	view
6	x 3	pytorch-transformers pytorch-transformers XLM/src/evaluation	add_umt_subqs_subas_to_q_squad_format... split_umt_dev.py evaluator.py	0:0 0:0 0:0	view