amazon-research / fact-check-summarization
Duplication

Places in code with 6 or more lines that are exactly the same.

Intro
  • For duplication, we look at places in code where there are 6 or more lines of code that are exactly the same.
  • Before duplication is calculated, the code is cleaned to remove empty lines, comments, and frequently duplicated constructs such as imports.
  • You should aim at having as little as possible (<5%) of duplicated code as high-level of duplication can lead to maintenance difficulties, poor factoring, and logical contradictions.
Learn more...
Duplication Overall
  • 19% duplication:
    • 32,109 cleaned lines of cleaned code (without empty lines, comments, and frequently duplicated constructs such as imports)
    • 6,114 duplicated lines
  • 619 duplicates
system19% (6,114 lines)
Duplication per Extension
py18% (5,686 lines)
cu31% (279 lines)
cpp21% (76 lines)
cuh63% (73 lines)
Duplication per Component (primary)
fairseq/models19% (1,461 lines)
fairseq/tasks32% (951 lines)
scripts56% (851 lines)
preprocess26% (796 lines)
fairseq/modules20% (785 lines)
fairseq/data5% (223 lines)
fairseq/optim13% (217 lines)
fairseq/criterions17% (197 lines)
fairseq/clib34% (192 lines)
fairseq_cli15% (184 lines)
fairseq/benchmark57% (128 lines)
fairseq2% (83 lines)
fairseq/logging8% (46 lines)
ROOT0% (0 lines)

Duplication Between Components (50+ lines)

G fairseq/models fairseq/models fairseq/modules fairseq/modules fairseq/models--fairseq/modules 52 fairseq/benchmark fairseq/benchmark fairseq/tasks fairseq/tasks fairseq/benchmark--fairseq/tasks 50

Download: SVG DOT (open online Graphviz editor)

Open 3D force graph...

Show more details on duplication between components...
Longest Duplicates
The list of 20 longest duplicates.
See data for all 619 duplicates...
Size#FoldersFilesLinesCode
100 x 2 fairseq/tasks
fairseq/tasks
translation.py
translation_with_unlikelihood.py
 309:436 (31%)
195:322 (45%)
 view
68 x 2 scripts
scripts
launch_multitask_cnndm.py
launch_multitask_xsum.py
 69:144 (60%)
69:145 (59%)
 view
60 x 2 scripts
scripts
launch_cnndm.py
launch_multitask_xsum.py
 76:144 (53%)
78:145 (52%)
 view
60 x 2 scripts
scripts
launch_cnndm.py
launch_multitask_cnndm.py
 76:144 (53%)
78:144 (53%)
 view
53 x 2 fairseq/benchmark
fairseq/benchmark
dummy_lm.py
dummy_masked_lm.py
 40:111 (75%)
51:122 (67%)
 view
53 x 2 fairseq/clib/libnat_cuda
fairseq/clib/libnat_cuda
edit_dist.cu
edit_dist.cu
 101:161 (18%)
181:241 (18%)
 view
47 x 2 preprocess
preprocess
sm_inference_asum.py
sm_inference_asum.py
 557:609 (4%)
687:740 (4%)
 view
38 x 2 fairseq/modules
fairseq/modules/dynamicconv_layer
dynamic_convolution.py
dynamicconv_layer.py
 136:183 (19%)
136:183 (23%)
 view
38 x 2 fairseq/models/nat
fairseq/models/nat
iterative_nonautoregressive_transform...
nonautoregressive_transformer.py
 151:191 (24%)
376:416 (12%)
 view
38 x 2 scripts
scripts
launch_multitask_cnndm.py
launch_multitask_newsroom.py
 36:75 (33%)
36:76 (33%)
 view
35 x 2 fairseq/models/nat
fairseq/models/nat
insertion_transformer.py
nonautoregressive_transformer.py
 242:277 (17%)
376:411 (11%)
 view
35 x 2 fairseq/models/nat
fairseq/models/nat
insertion_transformer.py
iterative_nonautoregressive_transform...
 242:277 (17%)
151:186 (22%)
 view
35 x 2 scripts
scripts
launch_sagemaker_unlikelihood_cnndm.py
launch_sagemaker_unlikelihood_xsum.py
 69:109 (31%)
69:109 (31%)
 view
34 x 2 fairseq/tasks
fairseq/tasks
translation.py
translation_with_unlikelihood.py
 216:253 (10%)
85:122 (15%)
 view
34 x 2 fairseq/models/nat
fairseq/models/nat
levenshtein_transformer.py
nonautoregressive_transformer.py
 402:436 (9%)
376:410 (10%)
 view
34 x 2 fairseq/models/nat
fairseq/models/nat
iterative_nonautoregressive_transform...
levenshtein_transformer.py
 151:185 (21%)
402:436 (9%)
 view
34 x 2 fairseq/models/nat
fairseq/models/nat
insertion_transformer.py
levenshtein_transformer.py
 242:276 (16%)
402:436 (9%)
 view
34 x 2 preprocess
preprocess
sm_inference_asum.py
sm_inference_asum.py
 446:480 (3%)
487:522 (3%)
 view
33 x 2 preprocess
preprocess
sm_inference_asum.py
sm_inference_asum.py
 277:309 (2%)
338:371 (2%)
 view
33 x 2 fairseq/models/bart
fairseq/models/roberta
model.py
model.py
 160:197 (15%)
153:190 (12%)
 view
Duplicated Units
The list of top 20 duplicated units.
See data for all 20 unit duplicates...
Size#FoldersFilesLinesCode
31 x 2 fairseq/tasks
fairseq/tasks
translation.py
translation_with_unlikelihood.py
 0:0 
0:0 
 view
26 x 2 fairseq/tasks
fairseq/tasks
masked_lm.py
multilingual_masked_lm.py
 0:0 
0:0 
 view
25 x 2 fairseq/tasks
fairseq/tasks
translation.py
translation_with_unlikelihood.py
 0:0 
0:0 
 view
23 x 2 fairseq/tasks
fairseq/tasks
translation.py
translation_with_unlikelihood.py
 0:0 
0:0 
 view
19 x 2 fairseq/benchmark
fairseq/benchmark
dummy_lm.py
dummy_masked_lm.py
 0:0 
0:0 
 view
11 x 3 fairseq/models/roberta
fairseq/models/roberta
fairseq/models/roberta
model_camembert.py
model_xlmr.py
model.py
 0:0 
0:0 
0:0 
 view
12 x 2 fairseq/tasks
fairseq/tasks
translation.py
translation_with_unlikelihood.py
 0:0 
0:0 
 view
11 x 2 fairseq/criterions
fairseq/criterions
sentence_prediction.py
sentence_ranking.py
 0:0 
0:0 
 view
8 x 2 fairseq/criterions
fairseq/criterions
label_smoothed_cross_entropy_with_unl...
label_smoothed_cross_entropy.py
 0:0 
0:0 
 view
6 x 2 fairseq
preprocess
options.py
sm_inference_asum.py
 0:0 
0:0 
 view
6 x 2 fairseq/data
fairseq/data
append_token_dataset.py
prepend_token_dataset.py
 0:0 
0:0 
 view
8 x 3 fairseq/data
fairseq/data/audio
fairseq/data
monolingual_dataset.py
raw_audio_dataset.py
subsample_dataset.py
 0:0 
0:0 
0:0 
 view
11 x 2 fairseq/optim
fairseq/optim
nag.py
sgd.py
 0:0 
0:0 
 view
6 x 2 fairseq/benchmark
fairseq/benchmark
dummy_lm.py
dummy_masked_lm.py
 0:0 
0:0 
 view
6 x 2 fairseq/benchmark
fairseq/benchmark
dummy_lm.py
dummy_masked_lm.py
 0:0 
0:0 
 view
8 x 2 fairseq/tasks
fairseq/tasks
multilingual_denoising.py
multilingual_masked_lm.py
 0:0 
0:0 
 view
7 x 2 fairseq/tasks
fairseq/tasks
sentence_prediction.py
sentence_ranking.py
 0:0 
0:0 
 view
8 x 2 fairseq/models
fairseq/models
fairseq_model.py
fairseq_model.py
 0:0 
0:0 
 view
7 x 2 fairseq/models
fairseq/models
fairseq_incremental_decoder.py
fairseq_decoder.py
 0:0 
0:0 
 view
6 x 11 fairseq/criterions
fairseq/criterions
fairseq/criterions
fairseq/criterions
fairseq/criterions
fairseq/criterions
fairseq/criterions
fairseq/criterions
fairseq/criterions
fairseq/criterions
...
sentence_prediction.py
label_smoothed_cross_entropy_with_unl...
label_smoothed_cross_entropy_with_mul...
label_smoothed_cross_entropy.py
label_smoothed_cross_entropy_with_ali...
nat_loss.py
adaptive_loss.py
masked_lm.py
legacy_masked_lm.py
sentence_ranking.py
...
 0:0 
0:0 
0:0 
0:0 
0:0 
0:0 
0:0 
0:0 
0:0 
0:0 
...
 view