facebookresearch / reconsider
Duplication

Places in code with 6 or more lines that are exactly the same.

Intro
  • For duplication, we look at places in code where there are 6 or more lines of code that are exactly the same.
  • Before duplication is calculated, the code is cleaned to remove empty lines, comments, and frequently duplicated constructs such as imports.
  • You should aim at having as little as possible (<5%) of duplicated code as high-level of duplication can lead to maintenance difficulties, poor factoring, and logical contradictions.
Learn more...
Duplication Overall
  • 18% duplication:
    • 8,295 cleaned lines of cleaned code (without empty lines, comments, and frequently duplicated constructs such as imports)
    • 1,550 duplicated lines
  • 106 duplicates
system18% (1,550 lines)
Duplication per Extension
py18% (1,550 lines)
Duplication per Component (primary)
pytorch_transformers20% (1,550 lines)
ROOT0% (0 lines)
Longest Duplicates
The list of 20 longest duplicates.
See data for all 106 duplicates...
Size#FoldersFilesLinesCode
62 x 2 pytorch_transformers
pytorch_transformers
tokenization_openai.py
tokenization_xlm.py
 108:184 (45%)
144:220 (36%)
 view
62 x 2 pytorch_transformers
pytorch_transformers
tokenization_gpt2.py
tokenization_roberta.py
 119:195 (43%)
88:164 (46%)
 view
33 x 2 pytorch_transformers
pytorch_transformers
modeling_utils.py
modeling_xlnet.py
 758:809 (7%)
1244:1295 (4%)
 view
26 x 2 pytorch_transformers
pytorch_transformers
tokenization_gpt2.py
tokenization_openai.py
 142:168 (18%)
126:152 (18%)
 view
26 x 2 pytorch_transformers
pytorch_transformers
modeling_gpt2.py
modeling_openai.py
 189:221 (6%)
206:237 (7%)
 view
26 x 2 pytorch_transformers
pytorch_transformers
tokenization_openai.py
tokenization_roberta.py
 126:152 (18%)
111:137 (19%)
 view
26 x 2 pytorch_transformers
pytorch_transformers
tokenization_roberta.py
tokenization_xlm.py
 111:137 (19%)
162:188 (15%)
 view
26 x 2 pytorch_transformers
pytorch_transformers
tokenization_gpt2.py
tokenization_xlm.py
 142:168 (18%)
162:188 (15%)
 view
23 x 2 pytorch_transformers
pytorch_transformers
modeling_transfo_xl.py
modeling_xlnet.py
 152:175 (2%)
153:176 (3%)
 view
21 x 2 pytorch_transformers
pytorch_transformers
modeling_gpt2.py
modeling_openai.py
 232:256 (5%)
248:272 (5%)
 view
20 x 2 pytorch_transformers
pytorch_transformers
modeling_transfo_xl_utilities.py
modeling_transfo_xl_utilities.py
 121:144 (8%)
215:237 (8%)
 view
19 x 2 pytorch_transformers
pytorch_transformers
tokenization_openai.py
tokenization_roberta.py
 186:208 (13%)
182:204 (14%)
 view
19 x 2 pytorch_transformers
pytorch_transformers
modeling_gpt2.py
modeling_openai.py
 261:284 (5%)
278:301 (5%)
 view
19 x 2 pytorch_transformers
pytorch_transformers
tokenization_gpt2.py
tokenization_roberta.py
 197:219 (13%)
182:204 (14%)
 view
19 x 2 pytorch_transformers
pytorch_transformers
tokenization_gpt2.py
tokenization_openai.py
 197:219 (13%)
186:208 (13%)
 view
18 x 2 pytorch_transformers
pytorch_transformers
tokenization_openai.py
tokenization_xlm.py
 48:75 (13%)
69:96 (10%)
 view
17 x 2 pytorch_transformers
pytorch_transformers
modeling_gpt2.py
modeling_openai.py
 708:727 (4%)
701:720 (4%)
 view
16 x 2 pytorch_transformers
pytorch_transformers
modeling_transfo_xl.py
modeling_transfo_xl.py
 598:626 (1%)
690:717 (1%)
 view
15 x 2 pytorch_transformers
pytorch_transformers
modeling_transfo_xl.py
modeling_xlnet.py
 294:312 (1%)
315:333 (2%)
 view
15 x 2 pytorch_transformers
pytorch_transformers
modeling_distilbert.py
modeling_xlm.py
 178:194 (3%)
269:285 (3%)
 view
Duplicated Units
The list of top 11 duplicated units.
See data for all 11 unit duplicates...
Size#FoldersFilesLinesCode
38 x 2 pytorch_transformers
pytorch_transformers
tokenization_xlm.py
tokenization_openai.py
 0:0 
0:0 
 view
36 x 2 pytorch_transformers
pytorch_transformers
tokenization_gpt2.py
tokenization_roberta.py
 0:0 
0:0 
 view
19 x 3 pytorch_transformers
pytorch_transformers
pytorch_transformers
tokenization_gpt2.py
tokenization_roberta.py
tokenization_openai.py
 0:0 
0:0 
0:0 
 view
16 x 2 pytorch_transformers
pytorch_transformers
modeling_distilbert.py
modeling_xlm.py
 0:0 
0:0 
 view
14 x 2 pytorch_transformers
pytorch_transformers
modeling_gpt2.py
modeling_openai.py
 0:0 
0:0 
 view
13 x 2 pytorch_transformers
pytorch_transformers
tokenization_xlm.py
tokenization_openai.py
 0:0 
0:0 
 view
13 x 2 pytorch_transformers
pytorch_transformers
tokenization_xlm.py
tokenization_openai.py
 0:0 
0:0 
 view
9 x 2 pytorch_transformers
pytorch_transformers
tokenization_gpt2.py
tokenization_roberta.py
 0:0 
0:0 
 view
8 x 2 pytorch_transformers
pytorch_transformers
modeling_distilbert.py
modeling_xlm.py
 0:0 
0:0 
 view
11 x 2 pytorch_transformers
pytorch_transformers
modeling_gpt2.py
modeling_openai.py
 0:0 
0:0 
 view
10 x 2 pytorch_transformers
pytorch_transformers
tokenization_xlm.py
tokenization_openai.py
 0:0 
0:0 
 view