facebookresearch / SpanBERT
Duplication

Places in code with 6 or more lines that are exactly the same.

Intro
  • For duplication, we look at places in code where there are 6 or more lines of code that are exactly the same.
  • Before duplication is calculated, the code is cleaned to remove empty lines, comments, and frequently duplicated constructs such as imports.
  • You should aim at having as little as possible (<5%) of duplicated code as high-level of duplication can lead to maintenance difficulties, poor factoring, and logical contradictions.
Learn more...
Duplication Overall
  • 37% duplication:
    • 11,047 cleaned lines of cleaned code (without empty lines, comments, and frequently duplicated constructs such as imports)
    • 4,148 duplicated lines
  • 259 duplicates
system37% (4,148 lines)
Duplication per Extension
py37% (4,148 lines)
Duplication per Component (primary)
code61% (1,887 lines)
pretraining/fairseq/models71% (916 lines)
code/pytorch_pretrained_bert34% (389 lines)
pretraining/fairseq/data26% (374 lines)
pretraining/fairseq/modules25% (233 lines)
pretraining/fairseq/criterions45% (194 lines)
pretraining/fairseq/optim21% (155 lines)
pretraining0% (0 lines)
pretraining/fairseq0% (0 lines)
pretraining/fairseq/tasks0% (0 lines)

Duplication Between Components (50+ lines)

G code/pytorch_pretrained_bert code/pytorch_pretrained_bert pretraining/fairseq/models pretraining/fairseq/models code/pytorch_pretrained_bert--pretraining/fairseq/models 850 pretraining/fairseq/optim pretraining/fairseq/optim code/pytorch_pretrained_bert--pretraining/fairseq/optim 147

Download: SVG DOT (open online Graphviz editor)

Open 3D force graph...

Show more details on duplication between components...
Longest Duplicates
The list of 20 longest duplicates.
See data for all 259 duplicates...
Size#FoldersFilesLinesCode
118 x 2 pretraining/fairseq/models
pretraining/fairseq/models
hf_bert.py
pair_bert.py
 391:598 (23%)
440:647 (21%)
 view
75 x 2 code
code
run_mrqa.py
run_squad.py
 470:565 (9%)
520:615 (7%)
 view
59 x 2 pretraining/fairseq/models
pretraining/fairseq/models
hf_bert.py
pair_bert.py
 26:129 (11%)
25:128 (10%)
 view
54 x 2 code/pytorch_pretrained_bert
pretraining/fairseq/models
modeling.py
pair_bert.py
 367:436 (9%)
258:327 (9%)
 view
54 x 2 code
code
run_mrqa.py
run_squad.py
 191:261 (6%)
206:269 (5%)
 view
51 x 2 pretraining/fairseq/modules
pretraining/fairseq/modules
adaptive_input.py
adaptive_inputs.py
 16:75 (100%)
16:75 (100%)
 view
47 x 2 pretraining/fairseq/models
pretraining/fairseq/models
hf_bert.py
pair_bert.py
 270:330 (9%)
269:329 (8%)
 view
45 x 2 code/pytorch_pretrained_bert
pretraining/fairseq/models
modeling.py
hf_bert.py
 378:436 (7%)
270:328 (8%)
 view
43 x 2 code
code
run_mrqa.py
run_squad.py
 739:784 (5%)
919:964 (4%)
 view
43 x 2 pretraining/fairseq/models
pretraining/fairseq/models
hf_bert.py
pair_bert.py
 156:207 (8%)
155:206 (7%)
 view
37 x 2 code
code
run_mrqa.py
run_squad.py
 631:677 (4%)
809:855 (3%)
 view
35 x 2 code
code
run_mrqa.py
run_squad.py
 402:438 (4%)
436:472 (3%)
 view
34 x 2 code/pytorch_pretrained_bert
pretraining/fairseq/optim
optimization.py
bert_adam.py
 71:112 (36%)
94:134 (28%)
 view
33 x 2 code/pytorch_pretrained_bert
pretraining/fairseq/models
modeling.py
pair_bert.py
 314:361 (5%)
207:254 (6%)
 view
33 x 2 code/pytorch_pretrained_bert
pretraining/fairseq/models
modeling.py
hf_bert.py
 443:489 (5%)
333:379 (6%)
 view
31 x 2 code
code
run_glue.py
run_tacred.py
 792:828 (4%)
384:419 (6%)
 view
30 x 2 code
code
run_mrqa.py
run_squad.py
 314:357 (3%)
326:369 (3%)
 view
29 x 2 pretraining/fairseq/models
pretraining/fairseq/models
hf_bert.py
pair_bert.py
 231:268 (5%)
230:267 (5%)
 view
29 x 2 code
code
run_mrqa.py
run_squad.py
 705:737 (3%)
884:916 (3%)
 view
27 x 2 code
code
run_mrqa.py
run_squad.py
 785:814 (3%)
967:996 (2%)
 view
Duplicated Units
The list of top 20 duplicated units.
See data for all 31 unit duplicates...
Size#FoldersFilesLinesCode
101 x 2 pretraining/fairseq/models
pretraining/fairseq/models
hf_bert.py
pair_bert.py
 0:0 
0:0 
 view
50 x 2 code
code
run_squad.py
run_mrqa.py
 0:0 
0:0 
 view
56 x 2 pretraining/fairseq/models
pretraining/fairseq/models
hf_bert.py
pair_bert.py
 0:0 
0:0 
 view
19 x 2 pretraining/fairseq/data
pretraining/fairseq/data
no_nsp_span_bert_dataset.py
span_bert_dataset.py
 0:0 
0:0 
 view
18 x 2 pretraining/fairseq/models
pretraining/fairseq/models
hf_bert.py
pair_bert.py
 0:0 
0:0 
 view
17 x 2 code
code
run_squad.py
run_mrqa.py
 0:0 
0:0 
 view
16 x 2 code
code
run_squad.py
run_mrqa.py
 0:0 
0:0 
 view
14 x 2 code
code
run_squad.py
run_mrqa.py
 0:0 
0:0 
 view
14 x 2 pretraining/fairseq/models
pretraining/fairseq/models
hf_bert.py
pair_bert.py
 0:0 
0:0 
 view
14 x 2 pretraining/fairseq/data
pretraining/fairseq/data
masking.py
masking.py
 0:0 
0:0 
 view
13 x 2 code/pytorch_pretrained_bert
pretraining/fairseq/optim
optimization.py
bert_adam.py
 0:0 
0:0 
 view
13 x 2 pretraining/fairseq/models
pretraining/fairseq/models
hf_bert.py
pair_bert.py
 0:0 
0:0 
 view
12 x 2 pretraining/fairseq/models
pretraining/fairseq/models
hf_bert.py
pair_bert.py
 0:0 
0:0 
 view
12 x 2 pretraining/fairseq/models
pretraining/fairseq/models
hf_bert.py
pair_bert.py
 0:0 
0:0 
 view
11 x 2 code
code
run_glue.py
run_glue.py
 0:0 
0:0 
 view
11 x 2 pretraining/fairseq/models
pretraining/fairseq/models
hf_bert.py
pair_bert.py
 0:0 
0:0 
 view
11 x 2 pretraining/fairseq/modules
pretraining/fairseq/modules
multihead_attention.py
bidirectional_multihead_attention.py
 0:0 
0:0 
 view
11 x 2 pretraining/fairseq/modules
pretraining/fairseq/modules
adaptive_input.py
adaptive_inputs.py
 0:0 
0:0 
 view
11 x 2 pretraining/fairseq/data
pretraining/fairseq/data
no_nsp_span_bert_dataset.py
span_bert_dataset.py
 0:0 
0:0 
 view
9 x 2 pretraining/fairseq/models
pretraining/fairseq/models
hf_bert.py
pair_bert.py
 0:0 
0:0 
 view