duplicated block id: 1 size: 58 cleaned lines of code in 2 files: - src/fairseq/fairseq/benchmark/dummy_lm.py (42:118) - src/fairseq/fairseq/benchmark/dummy_masked_lm.py (51:127) duplicated block id: 2 size: 53 cleaned lines of code in 2 files: - src/fairseq/fairseq/clib/libnat_cuda/edit_dist.cu (101:161) - src/fairseq/fairseq/clib/libnat_cuda/edit_dist.cu (181:241) duplicated block id: 3 size: 38 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/nat/iterative_nonautoregressive_transformer.py (151:191) - src/fairseq/fairseq/models/nat/nonautoregressive_transformer.py (376:416) duplicated block id: 4 size: 38 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/dynamic_convolution.py (136:183) - src/fairseq/fairseq/modules/dynamicconv_layer/dynamicconv_layer.py (136:183) duplicated block id: 5 size: 38 cleaned lines of code in 2 files: - src/graph_construction.py (35:75) - src/prepare_data.py (92:132) duplicated block id: 6 size: 35 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/nat/insertion_transformer.py (242:277) - src/fairseq/fairseq/models/nat/iterative_nonautoregressive_transformer.py (151:186) duplicated block id: 7 size: 35 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/nat/insertion_transformer.py (242:277) - src/fairseq/fairseq/models/nat/nonautoregressive_transformer.py (376:411) duplicated block id: 8 size: 34 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/nat/iterative_nonautoregressive_transformer.py (151:185) - src/fairseq/fairseq/models/nat/levenshtein_transformer.py (402:436) duplicated block id: 9 size: 34 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/nat/insertion_transformer.py (242:276) - src/fairseq/fairseq/models/nat/levenshtein_transformer.py (402:436) duplicated block id: 10 size: 34 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/nat/levenshtein_transformer.py (402:436) - src/fairseq/fairseq/models/nat/nonautoregressive_transformer.py (376:410) duplicated block id: 11 size: 31 cleaned lines of code in 2 files: - src/fairseq/fairseq/clib/libnat/edit_dist.cpp (58:98) - src/fairseq/fairseq/clib/libnat/edit_dist.cpp (130:170) duplicated block id: 12 size: 31 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/nat/nonautoregressive_transformer.py (375:405) - src/fairseq/fairseq/models/transformer.py (1087:1117) duplicated block id: 13 size: 30 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/nat/insertion_transformer.py (242:271) - src/fairseq/fairseq/models/transformer.py (1088:1117) duplicated block id: 14 size: 30 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/nat/levenshtein_transformer.py (402:431) - src/fairseq/fairseq/models/transformer.py (1088:1117) duplicated block id: 15 size: 30 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/nat/iterative_nonautoregressive_transformer.py (151:180) - src/fairseq/fairseq/models/transformer.py (1088:1117) duplicated block id: 16 size: 29 cleaned lines of code in 2 files: - src/fairseq/fairseq/tasks/masked_lm.py (173:202) - src/fairseq/fairseq/tasks/multilingual_masked_lm.py (264:293) duplicated block id: 17 size: 28 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/bart/model.py (170:199) - src/fairseq/fairseq/models/roberta/model.py (191:220) duplicated block id: 18 size: 26 cleaned lines of code in 2 files: - src/fairseq/fairseq/model_parallel/modules/multihead_attention.py (108:138) - src/fairseq/fairseq/modules/multihead_attention.py (177:207) duplicated block id: 19 size: 25 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/longformer_multihead_attention.py (136:180) - src/fairseq/fairseq/modules/multihead_attention.py (98:141) duplicated block id: 20 size: 25 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/nat/cmlm_transformer.py (107:131) - src/fairseq/fairseq/models/nat/insertion_transformer.py (242:266) duplicated block id: 21 size: 25 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/nat/cmlm_transformer.py (107:131) - src/fairseq/fairseq/models/transformer.py (1088:1112) duplicated block id: 22 size: 25 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/nat/cmlm_transformer.py (107:131) - src/fairseq/fairseq/models/nat/levenshtein_transformer.py (402:426) duplicated block id: 23 size: 25 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/nat/cmlm_transformer.py (107:131) - src/fairseq/fairseq/models/nat/iterative_nonautoregressive_transformer.py (151:175) duplicated block id: 24 size: 25 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/dynamic_convolution.py (189:219) - src/fairseq/fairseq/modules/dynamicconv_layer/dynamicconv_layer.py (184:214) duplicated block id: 25 size: 25 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/nat/cmlm_transformer.py (107:131) - src/fairseq/fairseq/models/nat/nonautoregressive_transformer.py (376:400) duplicated block id: 26 size: 24 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/longformer_multihead_attention.py (499:528) - src/fairseq/fairseq/modules/multihead_attention.py (446:475) duplicated block id: 27 size: 21 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/longformer_multihead_attention.py (52:76) - src/fairseq/fairseq/modules/multihead_attention.py (39:63) duplicated block id: 28 size: 20 cleaned lines of code in 2 files: - src/fairseq/fairseq/tasks/masked_lm.py (37:57) - src/fairseq/fairseq/tasks/multilingual_masked_lm.py (40:60) duplicated block id: 29 size: 20 cleaned lines of code in 2 files: - src/fairseq/fairseq/tasks/multilingual_denoising.py (200:222) - src/fairseq/fairseq/tasks/multilingual_masked_lm.py (240:265) duplicated block id: 30 size: 19 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/quantization/scalar/modules/qconv.py (96:123) - src/fairseq/fairseq/modules/quantization/scalar/modules/qlinear.py (69:96) duplicated block id: 31 size: 19 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/quantization/scalar/modules/qconv.py (96:123) - src/fairseq/fairseq/modules/quantization/scalar/modules/qemb.py (84:111) duplicated block id: 32 size: 19 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/quantization/scalar/modules/qemb.py (84:111) - src/fairseq/fairseq/modules/quantization/scalar/modules/qlinear.py (69:96) duplicated block id: 33 size: 17 cleaned lines of code in 2 files: - src/fairseq/fairseq/model_parallel/models/roberta/model.py (199:236) - src/fairseq/fairseq/models/roberta/model.py (318:355) duplicated block id: 34 size: 17 cleaned lines of code in 2 files: - src/fairseq/fairseq/logging/progress_bar.py (165:184) - src/fairseq/fairseq/logging/progress_bar.py (237:256) duplicated block id: 35 size: 17 cleaned lines of code in 2 files: - src/fairseq/fairseq/model_parallel/modules/multihead_attention.py (295:313) - src/fairseq/fairseq/modules/multihead_attention.py (423:441) duplicated block id: 36 size: 16 cleaned lines of code in 2 files: - src/fairseq/fairseq/criterions/sentence_prediction.py (70:95) - src/fairseq/fairseq/criterions/sentence_ranking.py (91:116) duplicated block id: 37 size: 16 cleaned lines of code in 2 files: - src/fairseq/fairseq/model_parallel/modules/multihead_attention.py (247:267) - src/fairseq/fairseq/modules/multihead_attention.py (370:390) duplicated block id: 38 size: 15 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/bart/hub_interface.py (164:181) - src/fairseq/fairseq/models/roberta/hub_interface.py (86:103) duplicated block id: 39 size: 15 cleaned lines of code in 2 files: - src/fairseq/fairseq/optim/fp16_optimizer.py (165:183) - src/fairseq/fairseq/optim/fp16_optimizer.py (369:387) duplicated block id: 40 size: 15 cleaned lines of code in 2 files: - src/fairseq/fairseq/tasks/masked_lm.py (152:166) - src/fairseq/fairseq/tasks/multilingual_masked_lm.py (183:197) duplicated block id: 41 size: 14 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/lightconv_layer/lightconv_cuda_kernel.cu (51:67) - src/fairseq/fairseq/modules/lightconv_layer/lightconv_cuda_kernel.cu (116:132) duplicated block id: 42 size: 14 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/longformer_multihead_attention.py (37:50) - src/fairseq/fairseq/modules/multihead_attention.py (25:38) duplicated block id: 43 size: 14 cleaned lines of code in 2 files: - src/fairseq/fairseq/data/append_token_dataset.py (26:42) - src/fairseq/fairseq/data/prepend_token_dataset.py (26:42) duplicated block id: 44 size: 14 cleaned lines of code in 2 files: - src/fairseq/fairseq/optim/fp16_optimizer.py (234:250) - src/fairseq/fairseq/optim/fp16_optimizer.py (431:447) duplicated block id: 45 size: 14 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/nat/cmlm_transformer.py (133:149) - src/fairseq/fairseq/models/nat/iterative_nonautoregressive_transformer.py (177:193) duplicated block id: 46 size: 14 cleaned lines of code in 2 files: - src/fairseq/fairseq/tasks/multilingual_denoising.py (182:196) - src/fairseq/fairseq/tasks/multilingual_masked_lm.py (224:238) duplicated block id: 47 size: 14 cleaned lines of code in 2 files: - src/fairseq/fairseq/criterions/cross_entropy.py (52:73) - src/fairseq/fairseq/model_parallel/criterions/vocab_parallel_cross_entropy.py (53:74) duplicated block id: 48 size: 14 cleaned lines of code in 2 files: - src/fairseq/fairseq/optim/adam.py (127:149) - src/fairseq/fairseq/optim/adamax.py (87:109) duplicated block id: 49 size: 14 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/nat/levenshtein_transformer.py (348:366) - src/fairseq/fairseq/models/nat/nonautoregressive_transformer.py (282:300) duplicated block id: 50 size: 14 cleaned lines of code in 2 files: - src/fairseq/fairseq/tasks/sentence_prediction.py (209:229) - src/fairseq/fairseq/tasks/sentence_ranking.py (172:192) duplicated block id: 51 size: 13 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/roberta/model_camembert.py (29:43) - src/fairseq/fairseq/models/roberta/model_xlmr.py (23:37) duplicated block id: 52 size: 13 cleaned lines of code in 2 files: - src/fairseq/fairseq/model_parallel/models/roberta/model.py (240:254) - src/fairseq/fairseq/models/roberta/model.py (359:373) duplicated block id: 53 size: 13 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/dynamic_convolution.py (217:233) - src/fairseq/fairseq/modules/lightweight_convolution.py (232:248) duplicated block id: 54 size: 13 cleaned lines of code in 2 files: - src/fairseq/fairseq/tasks/translation.py (255:275) - src/fairseq/fairseq/tasks/translation_lev.py (32:52) duplicated block id: 55 size: 13 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/nat/levenshtein_transformer.py (198:211) - src/fairseq/fairseq/models/nat/nonautoregressive_ensembles.py (183:195) duplicated block id: 56 size: 13 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/longformer_multihead_attention.py (101:120) - src/fairseq/fairseq/modules/multihead_attention.py (73:92) duplicated block id: 57 size: 12 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/dynamicconv_layer/dynamicconv_cuda.cuh (39:50) - src/fairseq/fairseq/modules/dynamicconv_layer/dynamicconv_cuda_kernel.cu (73:84) duplicated block id: 58 size: 12 cleaned lines of code in 2 files: - src/fairseq/fairseq/tasks/masked_lm.py (69:83) - src/fairseq/fairseq/tasks/multilingual_masked_lm.py (68:82) duplicated block id: 59 size: 12 cleaned lines of code in 2 files: - src/fairseq/fairseq/optim/adafactor.py (145:163) - src/fairseq/fairseq/optim/adam.py (135:153) duplicated block id: 60 size: 12 cleaned lines of code in 2 files: - src/fairseq/fairseq/model_parallel/models/roberta/model.py (112:128) - src/fairseq/fairseq/models/roberta/model.py (238:254) duplicated block id: 61 size: 12 cleaned lines of code in 2 files: - src/fairseq/fairseq/tasks/masked_lm.py (132:143) - src/fairseq/fairseq/tasks/multilingual_masked_lm.py (168:179) duplicated block id: 62 size: 12 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/quantization/pq/modules/qemb.py (36:47) - src/fairseq/fairseq/modules/quantization/scalar/modules/qemb.py (50:61) duplicated block id: 63 size: 12 cleaned lines of code in 2 files: - src/fairseq/fairseq/data/denoising_dataset.py (45:60) - src/fairseq/fairseq/data/language_pair_dataset.py (66:81) duplicated block id: 64 size: 12 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/sparse_transformer_sentence_encoder.py (28:39) - src/fairseq/fairseq/modules/transformer_sentence_encoder.py (86:97) duplicated block id: 65 size: 12 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/quantization/pq/modules/qemb.py (73:84) - src/fairseq/fairseq/modules/quantization/scalar/modules/qemb.py (119:130) duplicated block id: 66 size: 12 cleaned lines of code in 2 files: - src/fairseq/fairseq/model_parallel/models/roberta/model.py (178:189) - src/fairseq/fairseq/models/roberta/model.py (293:304) duplicated block id: 67 size: 12 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/nat/cmlm_transformer.py (133:147) - src/fairseq/fairseq/models/nat/nonautoregressive_transformer.py (402:416) duplicated block id: 68 size: 11 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/nat/insertion_transformer.py (68:79) - src/fairseq/fairseq/models/nat/levenshtein_utils.py (112:123) duplicated block id: 69 size: 11 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/lightconv_lm.py (38:48) - src/fairseq/fairseq/models/transformer_lm.py (61:71) duplicated block id: 70 size: 11 cleaned lines of code in 2 files: - src/fairseq/fairseq/optim/adamax.py (87:106) - src/fairseq/fairseq/optim/nag.py (48:67) duplicated block id: 71 size: 11 cleaned lines of code in 2 files: - src/fairseq/fairseq/optim/adam.py (127:146) - src/fairseq/fairseq/optim/nag.py (48:67) duplicated block id: 72 size: 11 cleaned lines of code in 2 files: - src/fairseq/fairseq_cli/eval_lm.py (126:137) - src/fairseq/fairseq_cli/generate.py (101:113) duplicated block id: 73 size: 11 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/lightconv_lm.py (73:83) - src/fairseq/fairseq/models/transformer_lm.py (98:108) duplicated block id: 74 size: 11 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/transformer.py (96:108) - src/fairseq/fairseq/models/transformer_lm.py (48:60) duplicated block id: 75 size: 11 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/lstm.py (56:66) - src/fairseq/fairseq/models/lstm_lm.py (31:41) duplicated block id: 76 size: 11 cleaned lines of code in 2 files: - src/fairseq/fairseq/tasks/language_modeling.py (176:187) - src/fairseq/fairseq/tasks/masked_lm.py (105:117) duplicated block id: 77 size: 11 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/nat/levenshtein_transformer.py (258:269) - src/fairseq/fairseq/models/nat/nonautoregressive_transformer.py (151:162) duplicated block id: 78 size: 11 cleaned lines of code in 2 files: - src/fairseq/fairseq/model_parallel/models/roberta/model.py (85:96) - src/fairseq/fairseq/models/roberta/model.py (133:144) duplicated block id: 79 size: 11 cleaned lines of code in 2 files: - src/fairseq/fairseq/data/denoising_dataset.py (32:44) - src/fairseq/fairseq/data/language_pair_dataset.py (52:64) duplicated block id: 80 size: 11 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/sparse_transformer_sentence_encoder.py (17:27) - src/fairseq/fairseq/modules/transformer_sentence_encoder.py (74:84) duplicated block id: 81 size: 10 cleaned lines of code in 2 files: - src/fairseq/fairseq/data/base_wrapper_dataset.py (33:46) - src/fairseq/fairseq/data/transform_eos_lang_pair_dataset.py (75:88) duplicated block id: 82 size: 10 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/lightconv_layer/lightconv_cuda_kernel.cu (21:33) - src/fairseq/fairseq/modules/lightconv_layer/lightconv_cuda_kernel.cu (81:93) duplicated block id: 83 size: 10 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/bart/model.py (144:153) - src/fairseq/fairseq/models/roberta/model.py (135:144) duplicated block id: 84 size: 10 cleaned lines of code in 2 files: - src/fairseq/fairseq/data/append_token_dataset.py (14:24) - src/fairseq/fairseq/data/prepend_token_dataset.py (14:24) duplicated block id: 85 size: 10 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/transformer_layer.py (143:153) - src/fairseq/fairseq/modules/transformer_layer.py (370:380) duplicated block id: 86 size: 10 cleaned lines of code in 2 files: - src/fairseq/fairseq/tasks/masked_lm.py (115:126) - src/fairseq/fairseq/tasks/multilingual_masked_lm.py (155:166) duplicated block id: 87 size: 10 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/nat/levenshtein_utils.py (51:61) - src/fairseq/fairseq/models/nat/levenshtein_utils.py (113:123) duplicated block id: 88 size: 10 cleaned lines of code in 2 files: - src/fairseq/fairseq/model_parallel/models/roberta/model.py (87:96) - src/fairseq/fairseq/models/bart/model.py (144:153) duplicated block id: 89 size: 10 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/nat/nat_crf_transformer.py (87:97) - src/fairseq/fairseq/models/nat/nonautoregressive_transformer.py (120:130) duplicated block id: 90 size: 10 cleaned lines of code in 2 files: - src/fairseq/fairseq/model_parallel/modules/transformer_sentence_encoder.py (43:52) - src/fairseq/fairseq/modules/transformer_sentence_encoder.py (194:203) duplicated block id: 91 size: 10 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/transformer.py (168:180) - src/fairseq/fairseq/models/transformer_lm.py (116:128) duplicated block id: 92 size: 10 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/roberta/model.py (162:171) - src/fairseq/fairseq/models/roberta/model_xlmr.py (28:37) duplicated block id: 93 size: 10 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/dynamicconv_layer/dynamicconv_layer.py (114:126) - src/fairseq/fairseq/modules/lightweight_convolution.py (234:246) duplicated block id: 94 size: 10 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/dynamic_convolution.py (126:136) - src/fairseq/fairseq/modules/dynamic_convolution.py (180:189) duplicated block id: 95 size: 10 cleaned lines of code in 2 files: - src/fairseq/fairseq/model_parallel/modules/transformer_layer.py (29:40) - src/fairseq/fairseq/model_parallel/modules/transformer_layer.py (53:64) duplicated block id: 96 size: 10 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/transformer.py (925:934) - src/fairseq/fairseq/models/transformer.py (938:947) duplicated block id: 97 size: 10 cleaned lines of code in 2 files: - src/fairseq/fairseq/model_parallel/modules/transformer_sentence_encoder_layer.py (61:70) - src/fairseq/fairseq/modules/transformer_sentence_encoder_layer.py (114:123) duplicated block id: 98 size: 10 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/roberta/model.py (162:171) - src/fairseq/fairseq/models/roberta/model_camembert.py (34:43) duplicated block id: 99 size: 10 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/lightconv_layer/lightconv_cuda.cuh (45:54) - src/fairseq/fairseq/modules/lightconv_layer/lightconv_cuda_kernel.cu (136:145) duplicated block id: 100 size: 10 cleaned lines of code in 2 files: - src/fairseq/fairseq/optim/nag.py (23:40) - src/fairseq/fairseq/optim/sgd.py (22:39) duplicated block id: 101 size: 10 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/nat/nat_crf_transformer.py (36:50) - src/fairseq/fairseq/models/nat/nonautoregressive_transformer.py (77:91) duplicated block id: 102 size: 10 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/dynamic_convolution.py (219:231) - src/fairseq/fairseq/modules/dynamicconv_layer/dynamicconv_layer.py (114:126) duplicated block id: 103 size: 10 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/nat/insertion_transformer.py (69:79) - src/fairseq/fairseq/models/nat/levenshtein_utils.py (51:61) duplicated block id: 104 size: 10 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/sparse_transformer_sentence_encoder_layer.py (15:24) - src/fairseq/fairseq/modules/transformer_sentence_encoder_layer.py (25:34) duplicated block id: 105 size: 10 cleaned lines of code in 2 files: - src/fairseq/fairseq/model_parallel/modules/multihead_attention.py (193:207) - src/fairseq/fairseq/modules/multihead_attention.py (278:292) duplicated block id: 106 size: 9 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/bart/model.py (130:138) - src/fairseq/fairseq/models/roberta/model.py (162:170) duplicated block id: 107 size: 9 cleaned lines of code in 2 files: - src/fairseq/fairseq/criterions/adaptive_loss.py (87:101) - src/fairseq/fairseq/criterions/cross_entropy.py (59:73) duplicated block id: 108 size: 9 cleaned lines of code in 2 files: - src/fairseq/fairseq/model_parallel/modules/multihead_attention.py (174:182) - src/fairseq/fairseq/modules/multihead_attention.py (259:267) duplicated block id: 109 size: 9 cleaned lines of code in 2 files: - src/fairseq/fairseq/model_parallel/modules/multihead_attention.py (64:74) - src/fairseq/fairseq/modules/multihead_attention.py (47:57) duplicated block id: 110 size: 9 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/bart/model.py (130:138) - src/fairseq/fairseq/models/roberta/model_xlmr.py (28:36) duplicated block id: 111 size: 9 cleaned lines of code in 2 files: - src/fairseq/fairseq/tasks/translation_from_pretrained_bart.py (63:74) - src/fairseq/fairseq/tasks/translation_lev.py (39:50) duplicated block id: 112 size: 9 cleaned lines of code in 2 files: - src/fairseq/fairseq_cli/generate.py (76:84) - src/fairseq/fairseq_cli/interactive.py (104:112) duplicated block id: 113 size: 9 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/lightconv_layer/lightconv_cuda.cuh (65:73) - src/fairseq/fairseq/modules/lightconv_layer/lightconv_cuda_kernel.cu (264:272) duplicated block id: 114 size: 9 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/dynamic_convolution.py (58:67) - src/fairseq/fairseq/modules/lightweight_convolution.py (115:124) duplicated block id: 115 size: 9 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/bart/hub_interface.py (82:90) - src/fairseq/fairseq/models/roberta/hub_interface.py (67:75) duplicated block id: 116 size: 9 cleaned lines of code in 2 files: - src/fairseq/fairseq/tasks/cross_lingual_lm.py (84:94) - src/fairseq/fairseq/tasks/legacy_masked_lm.py (61:71) duplicated block id: 117 size: 9 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/nat/cmlm_transformer.py (133:142) - src/fairseq/fairseq/models/nat/insertion_transformer.py (268:277) duplicated block id: 118 size: 9 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/dynamicconv_layer/dynamicconv_cuda.cuh (1:11) - src/fairseq/fairseq/modules/lightconv_layer/lightconv_cuda.cuh (1:11) duplicated block id: 119 size: 9 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/dynamic_convolution.py (141:149) - src/fairseq/fairseq/modules/lightweight_convolution.py (178:186) duplicated block id: 120 size: 9 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/dynamic_convolution.py (112:121) - src/fairseq/fairseq/modules/dynamicconv_layer/dynamicconv_layer.py (88:98) duplicated block id: 121 size: 9 cleaned lines of code in 2 files: - src/fairseq/fairseq/tasks/multilingual_denoising.py (120:129) - src/fairseq/fairseq/tasks/multilingual_masked_lm.py (143:152) duplicated block id: 122 size: 9 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/fconv.py (45:55) - src/fairseq/fairseq/models/lightconv.py (50:61) duplicated block id: 123 size: 9 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/dynamic_convolution.py (221:231) - src/fairseq/fairseq/modules/lightconv_layer/lightconv_layer.py (114:124) duplicated block id: 124 size: 9 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/dynamicconv_layer/dynamicconv_layer.py (141:149) - src/fairseq/fairseq/modules/lightweight_convolution.py (178:186) duplicated block id: 125 size: 9 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/lightconv.py (398:411) - src/fairseq/fairseq/models/nat/levenshtein_transformer.py (330:341) duplicated block id: 126 size: 9 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/dynamicconv_layer/dynamicconv_layer.py (116:126) - src/fairseq/fairseq/modules/lightconv_layer/lightconv_layer.py (114:124) duplicated block id: 127 size: 9 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/bart/model.py (130:138) - src/fairseq/fairseq/models/roberta/model_camembert.py (34:42) duplicated block id: 128 size: 9 cleaned lines of code in 2 files: - src/fairseq/fairseq/criterions/adaptive_loss.py (87:101) - src/fairseq/fairseq/model_parallel/criterions/vocab_parallel_cross_entropy.py (60:74) duplicated block id: 129 size: 9 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/lightconv.py (677:687) - src/fairseq/fairseq/models/transformer.py (1071:1081) duplicated block id: 130 size: 9 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/nat/cmlm_transformer.py (55:65) - src/fairseq/fairseq/models/nat/nonautoregressive_transformer.py (97:106) duplicated block id: 131 size: 9 cleaned lines of code in 2 files: - src/fairseq/fairseq/tasks/translation.py (262:273) - src/fairseq/fairseq/tasks/translation_from_pretrained_bart.py (63:74) duplicated block id: 132 size: 9 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/dynamicconv_layer/dynamicconv_cuda.cuh (13:23) - src/fairseq/fairseq/modules/lightconv_layer/lightconv_cuda.cuh (12:22) duplicated block id: 133 size: 9 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/__init__.py (123:131) - src/fairseq/fairseq/models/huggingface/__init__.py (11:19) duplicated block id: 134 size: 9 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/lightconv_layer/lightconv_layer.py (114:124) - src/fairseq/fairseq/modules/lightweight_convolution.py (236:246) duplicated block id: 135 size: 9 cleaned lines of code in 2 files: - src/fairseq/fairseq/model_parallel/models/transformer.py (24:34) - src/fairseq/fairseq/model_parallel/modules/transformer_sentence_encoder.py (20:30) duplicated block id: 136 size: 9 cleaned lines of code in 2 files: - src/fairseq/fairseq/model_parallel/modules/multihead_attention.py (64:74) - src/fairseq/fairseq/modules/longformer_multihead_attention.py (60:70) duplicated block id: 137 size: 8 cleaned lines of code in 2 files: - src/fairseq/fairseq/model_parallel/modules/transformer_sentence_encoder.py (56:63) - src/fairseq/fairseq/modules/transformer_sentence_encoder.py (208:215) duplicated block id: 138 size: 8 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/fconv.py (65:74) - src/fairseq/fairseq/models/lstm.py (32:41) duplicated block id: 139 size: 8 cleaned lines of code in 2 files: - src/prepare_data.py (283:291) - src/prepare_data.py (372:380) duplicated block id: 140 size: 8 cleaned lines of code in 2 files: - src/fairseq/fairseq/tasks/denoising.py (105:119) - src/fairseq/fairseq/tasks/masked_lm.py (83:97) duplicated block id: 141 size: 8 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/transformer.py (355:362) - src/fairseq/fairseq/models/transformer.py (723:730) duplicated block id: 142 size: 8 cleaned lines of code in 2 files: - src/fairseq/fairseq/model_parallel/models/roberta/model.py (148:155) - src/fairseq/fairseq/models/roberta/model.py (272:279) duplicated block id: 143 size: 8 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/nat/insertion_transformer.py (198:206) - src/fairseq/fairseq/models/nat/nat_crf_transformer.py (89:97) duplicated block id: 144 size: 8 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/nat/cmlm_transformer.py (54:61) - src/fairseq/fairseq/models/nat/iterative_nonautoregressive_transformer.py (137:144) duplicated block id: 145 size: 8 cleaned lines of code in 2 files: - src/fairseq/fairseq/model_parallel/modules/multihead_attention.py (34:41) - src/fairseq/fairseq/modules/multihead_attention.py (25:32) duplicated block id: 146 size: 8 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/nat/levenshtein_transformer.py (241:249) - src/fairseq/fairseq/models/nat/nonautoregressive_ensembles.py (127:134) duplicated block id: 147 size: 8 cleaned lines of code in 2 files: - src/prepare_data.py (223:231) - src/prepare_data.py (326:334) duplicated block id: 148 size: 8 cleaned lines of code in 2 files: - src/fairseq/fairseq/optim/fused_adam.py (283:290) - src/fairseq/fairseq/optim/fused_adam.py (297:304) duplicated block id: 149 size: 8 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/lightconv.py (167:175) - src/fairseq/fairseq/models/multilingual_transformer.py (78:86) duplicated block id: 150 size: 8 cleaned lines of code in 2 files: - src/fairseq/fairseq/model_parallel/modules/transformer_layer.py (9:20) - src/fairseq/fairseq/model_parallel/modules/transformer_sentence_encoder_layer.py (12:21) duplicated block id: 151 size: 8 cleaned lines of code in 2 files: - src/fairseq/fairseq/tasks/masked_lm.py (96:103) - src/fairseq/fairseq/tasks/multilingual_denoising.py (122:129) duplicated block id: 152 size: 8 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/nat/cmlm_transformer.py (94:102) - src/fairseq/fairseq/models/nat/nat_crf_transformer.py (89:97) duplicated block id: 153 size: 8 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/nat/cmlm_transformer.py (94:102) - src/fairseq/fairseq/models/nat/nonautoregressive_transformer.py (122:130) duplicated block id: 154 size: 8 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/fconv.py (45:53) - src/fairseq/fairseq/models/transformer.py (58:67) duplicated block id: 155 size: 8 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/nat/insertion_transformer.py (133:144) - src/fairseq/fairseq/models/nat/levenshtein_transformer.py (73:84) duplicated block id: 156 size: 8 cleaned lines of code in 2 files: - src/fairseq/fairseq/optim/adafactor.py (145:159) - src/fairseq/fairseq/optim/adamax.py (95:109) duplicated block id: 157 size: 8 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/lightconv.py (146:154) - src/fairseq/fairseq/models/lightconv_lm.py (90:98) duplicated block id: 158 size: 8 cleaned lines of code in 2 files: - src/fairseq/fairseq/tasks/masked_lm.py (96:103) - src/fairseq/fairseq/tasks/multilingual_masked_lm.py (145:152) duplicated block id: 159 size: 8 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/nat/cmlm_transformer.py (94:102) - src/fairseq/fairseq/models/nat/insertion_transformer.py (198:206) duplicated block id: 160 size: 8 cleaned lines of code in 2 files: - src/fairseq/fairseq/data/transform_eos_dataset.py (109:121) - src/fairseq/fairseq/data/transform_eos_lang_pair_dataset.py (79:89) duplicated block id: 161 size: 8 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/dynamicconv_layer/dynamicconv_layer.py (176:183) - src/fairseq/fairseq/modules/lightweight_convolution.py (206:213) duplicated block id: 162 size: 8 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/nat/insertion_transformer.py (198:206) - src/fairseq/fairseq/models/nat/nonautoregressive_transformer.py (122:130) duplicated block id: 163 size: 8 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/nat/levenshtein_transformer.py (273:280) - src/fairseq/fairseq/models/nat/nonautoregressive_transformer.py (192:199) duplicated block id: 164 size: 8 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/lightconv_lm.py (135:142) - src/fairseq/fairseq/models/transformer_lm.py (188:195) duplicated block id: 165 size: 8 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/lightconv.py (103:110) - src/fairseq/fairseq/models/transformer.py (119:126) duplicated block id: 166 size: 8 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/nat/levenshtein_transformer.py (458:465) - src/fairseq/fairseq/models/transformer.py (1152:1159) duplicated block id: 167 size: 8 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/lightconv.py (638:645) - src/fairseq/fairseq/modules/transformer_layer.py (353:361) duplicated block id: 168 size: 8 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/multihead_attention.py (213:220) - src/fairseq/fairseq/modules/multihead_attention.py (299:306) duplicated block id: 169 size: 8 cleaned lines of code in 2 files: - src/fairseq/fairseq/benchmark/dummy_lm.py (25:33) - src/fairseq/fairseq/benchmark/dummy_masked_lm.py (25:33) duplicated block id: 170 size: 8 cleaned lines of code in 2 files: - src/fairseq/fairseq/tasks/fairseq_task.py (342:350) - src/fairseq/fairseq/tasks/translation_lev.py (151:159) duplicated block id: 171 size: 8 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/nat/cmlm_transformer.py (133:141) - src/fairseq/fairseq/models/nat/levenshtein_transformer.py (428:436) duplicated block id: 172 size: 8 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/nat/nonautoregressive_transformer.py (144:152) - src/fairseq/fairseq/models/nat/nonautoregressive_transformer.py (174:182) duplicated block id: 173 size: 8 cleaned lines of code in 2 files: - src/fairseq/fairseq/model_parallel/modules/multihead_attention.py (165:172) - src/fairseq/fairseq/modules/multihead_attention.py (250:257) duplicated block id: 174 size: 8 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/sparse_transformer_sentence_encoder.py (57:64) - src/fairseq/fairseq/modules/transformer_sentence_encoder.py (154:161) duplicated block id: 175 size: 8 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/dynamicconv_layer/dynamicconv_layer.py (39:46) - src/fairseq/fairseq/modules/lightconv_layer/lightconv_layer.py (38:45) duplicated block id: 176 size: 8 cleaned lines of code in 2 files: - src/fairseq/fairseq/criterions/cross_entropy.py (34:42) - src/fairseq/fairseq/criterions/label_smoothed_cross_entropy.py (62:70) duplicated block id: 177 size: 8 cleaned lines of code in 2 files: - src/fairseq/fairseq/optim/lr_scheduler/fixed_schedule.py (13:23) - src/fairseq/fairseq/optim/lr_scheduler/polynomial_decay_schedule.py (13:23) duplicated block id: 178 size: 8 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/lightconv_layer/lightconv_cuda_kernel.cu (251:260) - src/fairseq/fairseq/modules/lightconv_layer/lightconv_cuda_kernel.cu (366:375) duplicated block id: 179 size: 8 cleaned lines of code in 2 files: - src/fairseq/fairseq/model_parallel/modules/multihead_attention.py (34:41) - src/fairseq/fairseq/modules/longformer_multihead_attention.py (37:44) duplicated block id: 180 size: 8 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/lightconv.py (50:59) - src/fairseq/fairseq/models/transformer.py (58:67) duplicated block id: 181 size: 8 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/dynamic_convolution.py (176:183) - src/fairseq/fairseq/modules/lightweight_convolution.py (206:213) duplicated block id: 182 size: 8 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/fconv_lm.py (17:26) - src/fairseq/fairseq/models/lstm_lm.py (18:28) duplicated block id: 183 size: 8 cleaned lines of code in 2 files: - src/fairseq/fairseq/model_parallel/models/roberta/model.py (73:83) - src/fairseq/fairseq/models/roberta/model.py (113:123) duplicated block id: 184 size: 8 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/nat/levenshtein_transformer.py (175:182) - src/fairseq/fairseq/models/nat/nonautoregressive_ensembles.py (160:167) duplicated block id: 185 size: 7 cleaned lines of code in 2 files: - src/fairseq/fairseq/model_parallel/modules/multihead_attention.py (83:89) - src/fairseq/fairseq/modules/multihead_attention.py (106:112) duplicated block id: 186 size: 7 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/dynamic_convolution.py (142:148) - src/fairseq/fairseq/modules/lightconv_layer/lightconv_layer.py (81:87) duplicated block id: 187 size: 7 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/roberta/model.py (91:98) - src/fairseq/fairseq/models/transformer_lm.py (117:124) duplicated block id: 188 size: 7 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/nat/cmlm_transformer.py (39:48) - src/fairseq/fairseq/models/nat/nonautoregressive_transformer.py (81:91) duplicated block id: 189 size: 7 cleaned lines of code in 2 files: - src/fairseq/fairseq/tasks/sentence_prediction.py (13:19) - src/fairseq/fairseq/tasks/sentence_ranking.py (13:19) duplicated block id: 190 size: 7 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/dynamic_crf_layer.py (111:117) - src/fairseq/fairseq/modules/dynamic_crf_layer.py (138:144) duplicated block id: 191 size: 7 cleaned lines of code in 2 files: - src/fairseq/fairseq/data/audio/raw_audio_dataset.py (108:118) - src/fairseq/fairseq/data/subsample_dataset.py (60:68) duplicated block id: 192 size: 7 cleaned lines of code in 2 files: - src/fairseq/fairseq/optim/adam.py (157:167) - src/fairseq/fairseq/optim/adamax.py (114:123) duplicated block id: 193 size: 7 cleaned lines of code in 2 files: - src/fairseq/fairseq/data/encoders/gpt2_bpe.py (42:49) - src/fairseq/fairseq/data/encoders/hf_byte_bpe.py (39:46) duplicated block id: 194 size: 7 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/quantization/pq/modules/qconv.py (70:76) - src/fairseq/fairseq/modules/quantization/pq/modules/qlinear.py (42:48) duplicated block id: 195 size: 7 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/nat/iterative_nonautoregressive_transformer.py (138:144) - src/fairseq/fairseq/models/nat/nonautoregressive_transformer.py (97:103) duplicated block id: 196 size: 7 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/huggingface/hf_gpt2.py (44:50) - src/fairseq/fairseq/models/huggingface/hf_gpt2.py (90:96) duplicated block id: 197 size: 7 cleaned lines of code in 2 files: - src/fairseq/fairseq/tasks/sentence_prediction.py (115:124) - src/fairseq/fairseq/tasks/sentence_ranking.py (92:102) duplicated block id: 198 size: 7 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/fairseq_model.py (46:53) - src/fairseq/fairseq/models/transformer.py (291:298) duplicated block id: 199 size: 7 cleaned lines of code in 2 files: - src/fairseq/fairseq/model_parallel/modules/multihead_attention.py (83:89) - src/fairseq/fairseq/modules/longformer_multihead_attention.py (145:151) duplicated block id: 200 size: 7 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/nat/cmlm_transformer.py (56:63) - src/fairseq/fairseq/models/nat/nat_crf_transformer.py (65:72) duplicated block id: 201 size: 7 cleaned lines of code in 2 files: - src/fairseq/fairseq/tasks/multilingual_denoising.py (160:167) - src/fairseq/fairseq/tasks/multilingual_masked_lm.py (201:209) duplicated block id: 202 size: 7 cleaned lines of code in 2 files: - src/fairseq/fairseq/optim/fp16_optimizer.py (226:232) - src/fairseq/fairseq/optim/fp16_optimizer.py (423:429) duplicated block id: 203 size: 7 cleaned lines of code in 2 files: - src/fairseq/fairseq/optim/lr_scheduler/fixed_schedule.py (45:54) - src/fairseq/fairseq/optim/lr_scheduler/polynomial_decay_schedule.py (48:57) duplicated block id: 204 size: 7 cleaned lines of code in 2 files: - src/prepare_data.py (210:216) - src/prepare_data.py (298:304) duplicated block id: 205 size: 7 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/bart/model.py (127:134) - src/fairseq/fairseq/models/fairseq_model.py (203:232) duplicated block id: 206 size: 7 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/lstm.py (97:103) - src/fairseq/fairseq/models/lstm_lm.py (67:73) duplicated block id: 207 size: 7 cleaned lines of code in 2 files: - src/fairseq/fairseq/tasks/masked_lm.py (14:20) - src/fairseq/fairseq/tasks/multilingual_masked_lm.py (17:23) duplicated block id: 208 size: 7 cleaned lines of code in 2 files: - src/fairseq/fairseq/tasks/sentence_prediction.py (79:91) - src/fairseq/fairseq/tasks/sentence_ranking.py (67:79) duplicated block id: 209 size: 7 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/lstm.py (173:179) - src/fairseq/fairseq/models/lstm_lm.py (104:110) duplicated block id: 210 size: 7 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/dynamicconv_layer/dynamicconv_layer.py (142:148) - src/fairseq/fairseq/modules/lightconv_layer/lightconv_layer.py (81:87) duplicated block id: 211 size: 7 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/dynamicconv_layer/dynamicconv_cuda.cpp (40:47) - src/fairseq/fairseq/modules/lightconv_layer/lightconv_cuda.cpp (39:46) duplicated block id: 212 size: 7 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/lightconv.py (485:492) - src/fairseq/fairseq/models/lightconv.py (573:580) duplicated block id: 213 size: 7 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/dynamicconv_layer/dynamicconv_layer.py (203:211) - src/fairseq/fairseq/modules/lightweight_convolution.py (222:229) duplicated block id: 214 size: 7 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/nat/nat_crf_transformer.py (65:72) - src/fairseq/fairseq/models/nat/nonautoregressive_transformer.py (98:105) duplicated block id: 215 size: 7 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/nat/cmlm_transformer.py (39:48) - src/fairseq/fairseq/models/nat/nat_crf_transformer.py (40:50) duplicated block id: 216 size: 7 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/fconv.py (152:159) - src/fairseq/fairseq/models/fconv_self_att.py (175:182) duplicated block id: 217 size: 7 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/model_utils.py (84:90) - src/fairseq/fairseq/models/nat/levenshtein_utils.py (278:284) duplicated block id: 218 size: 7 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/lightconv_lm.py (52:58) - src/fairseq/fairseq/models/transformer_lm.py (77:83) duplicated block id: 219 size: 7 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/nat/insertion_transformer.py (168:176) - src/fairseq/fairseq/models/nat/levenshtein_transformer.py (137:145) duplicated block id: 220 size: 7 cleaned lines of code in 2 files: - src/fairseq/fairseq/data/audio/raw_audio_dataset.py (108:118) - src/fairseq/fairseq/data/monolingual_dataset.py (185:193) duplicated block id: 221 size: 7 cleaned lines of code in 2 files: - src/fairseq/fairseq/data/monolingual_dataset.py (185:193) - src/fairseq/fairseq/data/subsample_dataset.py (60:68) duplicated block id: 222 size: 7 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/lightconv_layer/lightconv_layer.py (81:87) - src/fairseq/fairseq/modules/lightweight_convolution.py (179:185) duplicated block id: 223 size: 7 cleaned lines of code in 2 files: - src/fairseq/fairseq_cli/train.py (187:193) - src/fairseq/fairseq_cli/train.py (289:295) duplicated block id: 224 size: 7 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/dynamic_convolution.py (208:216) - src/fairseq/fairseq/modules/lightweight_convolution.py (222:229) duplicated block id: 225 size: 7 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/transformer.py (69:76) - src/fairseq/fairseq/models/transformer_lm.py (30:37) duplicated block id: 226 size: 7 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/lightconv.py (168:175) - src/fairseq/fairseq/models/transformer.py (230:238) duplicated block id: 227 size: 7 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/quantization/pq/modules/qconv.py (40:46) - src/fairseq/fairseq/modules/quantization/scalar/modules/qconv.py (36:42) duplicated block id: 228 size: 7 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/lightconv_layer/lightconv_cuda_kernel.cu (221:228) - src/fairseq/fairseq/modules/lightconv_layer/lightconv_cuda_kernel.cu (337:344) duplicated block id: 229 size: 7 cleaned lines of code in 2 files: - src/fairseq/fairseq/data/denoising_dataset.py (23:30) - src/fairseq/fairseq/data/language_pair_dataset.py (21:28) duplicated block id: 230 size: 7 cleaned lines of code in 2 files: - src/fairseq/fairseq/data/encoders/bytes.py (14:22) - src/fairseq/fairseq/data/encoders/characters.py (15:23) duplicated block id: 231 size: 7 cleaned lines of code in 2 files: - src/fairseq/fairseq_cli/preprocess.py (149:155) - src/fairseq/fairseq_cli/preprocess.py (207:213) duplicated block id: 232 size: 7 cleaned lines of code in 2 files: - src/fairseq/fairseq/data/encoders/utils.py (18:24) - src/fairseq/fairseq/tasks/multilingual_masked_lm.py (95:102) duplicated block id: 233 size: 7 cleaned lines of code in 2 files: - src/fairseq/fairseq/sequence_generator.py (665:671) - src/fairseq/fairseq/sequence_generator.py (684:690) duplicated block id: 234 size: 7 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/lightconv_layer/lightconv_layer.py (93:102) - src/fairseq/fairseq/modules/lightweight_convolution.py (195:203) duplicated block id: 235 size: 7 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/dynamic_convolution.py (123:129) - src/fairseq/fairseq/modules/dynamicconv_layer/dynamicconv_layer.py (128:134) duplicated block id: 236 size: 7 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/lstm.py (132:138) - src/fairseq/fairseq/models/lstm_lm.py (75:81) duplicated block id: 237 size: 7 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/multilingual_transformer.py (79:86) - src/fairseq/fairseq/models/transformer.py (230:238) duplicated block id: 238 size: 7 cleaned lines of code in 2 files: - src/fairseq/fairseq_cli/preprocess.py (123:129) - src/fairseq/fairseq_cli/preprocess.py (181:187) duplicated block id: 239 size: 7 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/roberta/model.py (91:98) - src/fairseq/fairseq/models/transformer.py (169:176) duplicated block id: 240 size: 7 cleaned lines of code in 2 files: - src/fairseq/fairseq/data/base_wrapper_dataset.py (37:46) - src/fairseq/fairseq/data/transform_eos_dataset.py (109:120) duplicated block id: 241 size: 7 cleaned lines of code in 2 files: - src/fairseq/fairseq_cli/generate.py (189:195) - src/fairseq/fairseq_cli/interactive.py (177:183) duplicated block id: 242 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/logging/meters.py (101:106) - src/fairseq/fairseq/logging/meters.py (154:159) duplicated block id: 243 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/model_parallel/megatron_trainer.py (30:35) - src/fairseq/fairseq/model_parallel/models/transformer.py (47:52) duplicated block id: 244 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/benchmark/dummy_lm.py (77:83) - src/fairseq/fairseq/tasks/sentence_ranking.py (202:208) duplicated block id: 245 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq_cli/preprocess.py (137:142) - src/fairseq/fairseq_cli/preprocess.py (194:199) duplicated block id: 246 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/optim/fused_adam.py (91:97) - src/fairseq/fairseq/optim/fused_adam.py (215:221) duplicated block id: 247 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/transformer.py (127:132) - src/fairseq/fairseq/models/transformer_lm.py (67:72) duplicated block id: 248 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/fconv.py (259:269) - src/fairseq/fairseq/models/fconv_self_att.py (252:262) duplicated block id: 249 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/lightconv.py (77:84) - src/fairseq/fairseq/models/lstm.py (29:37) duplicated block id: 250 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/model_parallel/models/transformer_lm.py (42:50) - src/fairseq/fairseq/models/transformer_lm.py (132:140) duplicated block id: 251 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/tasks/multilingual_masked_lm.py (311:317) - src/fairseq/fairseq/tasks/sentence_prediction.py (239:245) duplicated block id: 252 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/optim/adamax.py (87:93) - src/fairseq/fairseq/optim/fused_adam.py (91:97) duplicated block id: 253 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/quantization/pq/modules/qconv.py (95:100) - src/fairseq/fairseq/modules/quantization/scalar/modules/qconv.py (89:94) duplicated block id: 254 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/tasks/language_modeling.py (71:76) - src/fairseq/fairseq/tasks/masked_lm.py (43:48) duplicated block id: 255 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq_cli/generate.py (103:108) - src/fairseq/fairseq_cli/validate.py (89:94) duplicated block id: 256 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/data/encoders/byte_bpe.py (24:31) - src/fairseq/fairseq/data/encoders/sentencepiece_bpe.py (22:29) duplicated block id: 257 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq_cli/preprocess.py (23:28) - src/fairseq/fairseq_cli/train.py (33:38) duplicated block id: 258 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/benchmark/dummy_model.py (85:90) - src/fairseq/fairseq/models/roberta/model.py (125:131) duplicated block id: 259 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/lightconv.py (112:117) - src/fairseq/fairseq/models/transformer.py (126:131) duplicated block id: 260 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/tasks/masked_lm.py (62:67) - src/fairseq/fairseq/tasks/sentence_ranking.py (54:59) duplicated block id: 261 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/data/denoising_dataset.py (357:370) - src/fairseq/fairseq/data/monolingual_dataset.py (175:188) duplicated block id: 262 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/lightconv_layer/lightconv_cuda.cuh (48:53) - src/fairseq/fairseq/modules/lightconv_layer/lightconv_cuda.cuh (68:73) duplicated block id: 263 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/tasks/sentence_prediction.py (239:245) - src/fairseq/fairseq/tasks/sentence_ranking.py (202:208) duplicated block id: 264 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/transformer.py (1060:1065) - src/fairseq/fairseq/modules/transformer_layer.py (405:410) duplicated block id: 265 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/fconv.py (14:19) - src/fairseq/fairseq/models/lightconv.py (14:19) duplicated block id: 266 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/model_parallel/models/roberta/model.py (180:185) - src/fairseq/fairseq/models/masked_lm.py (140:145) duplicated block id: 267 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/tasks/cross_lingual_lm.py (130:135) - src/fairseq/fairseq/tasks/legacy_masked_lm.py (127:132) duplicated block id: 268 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/cuda_utils.cu (1:6) - src/fairseq/fairseq/modules/dynamicconv_layer/dynamicconv_cuda_kernel.cu (1:6) duplicated block id: 269 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/nat/iterative_nonautoregressive_transformer.py (139:144) - src/fairseq/fairseq/models/nat/nat_crf_transformer.py (65:70) duplicated block id: 270 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/fconv_self_att.py (202:207) - src/fairseq/fairseq/models/fconv_self_att.py (340:345) duplicated block id: 271 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/optim/adamax.py (87:93) - src/fairseq/fairseq/optim/fused_adam.py (215:221) duplicated block id: 272 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/fconv_lm.py (25:30) - src/fairseq/fairseq/models/fconv_self_att.py (79:84) duplicated block id: 273 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/lightconv_layer/lightconv_cuda.cuh (26:31) - src/fairseq/fairseq/modules/lightconv_layer/lightconv_cuda_kernel.cu (13:18) duplicated block id: 274 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq_cli/train.py (33:38) - src/fairseq/fairseq_cli/validate.py (18:23) duplicated block id: 275 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/optim/adagrad.py (21:34) - src/fairseq/fairseq/optim/sgd.py (23:36) duplicated block id: 276 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/tasks/language_modeling.py (91:96) - src/fairseq/fairseq/tasks/sentence_ranking.py (54:59) duplicated block id: 277 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/model_parallel/modules/multihead_attention.py (18:23) - src/fairseq/fairseq/model_parallel/modules/transformer_sentence_encoder_layer.py (16:21) duplicated block id: 278 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/sparse_transformer_sentence_encoder.py (22:27) - src/fairseq/fairseq/modules/transformer_sentence_encoder_layer.py (27:32) duplicated block id: 279 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/transformer_from_pretrained_xlm.py (124:129) - src/fairseq/fairseq/models/transformer_from_pretrained_xlm.py (142:148) duplicated block id: 280 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/criterions/adaptive_loss.py (74:80) - src/fairseq/fairseq/criterions/legacy_masked_lm.py (123:129) duplicated block id: 281 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/nat/insertion_transformer.py (213:219) - src/fairseq/fairseq/models/nat/nonautoregressive_transformer.py (194:199) duplicated block id: 282 size: 6 cleaned lines of code in 2 files: - src/prepare_data.py (140:145) - src/prepare_data.py (210:215) duplicated block id: 283 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/tasks/masked_lm.py (204:210) - src/fairseq/fairseq/tasks/sentence_ranking.py (202:208) duplicated block id: 284 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/file_io.py (39:44) - src/fairseq/fairseq/file_io.py (47:52) duplicated block id: 285 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/lightconv_layer/lightconv_cuda_kernel.cu (230:237) - src/fairseq/fairseq/modules/lightconv_layer/lightconv_cuda_kernel.cu (346:352) duplicated block id: 286 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/benchmark/dummy_masked_lm.py (86:92) - src/fairseq/fairseq/tasks/masked_lm.py (204:210) duplicated block id: 287 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/tasks/language_modeling.py (71:76) - src/fairseq/fairseq/tasks/multilingual_masked_lm.py (46:51) duplicated block id: 288 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq_cli/eval_lm.py (128:133) - src/fairseq/fairseq_cli/validate.py (89:94) duplicated block id: 289 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/benchmark/dummy_lm.py (77:83) - src/fairseq/fairseq/tasks/sentence_prediction.py (239:245) duplicated block id: 290 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/tasks/language_modeling.py (156:168) - src/fairseq/fairseq/tasks/masked_lm.py (85:96) duplicated block id: 291 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/tasks/denoising.py (163:171) - src/fairseq/fairseq/tasks/sentence_prediction.py (239:245) duplicated block id: 292 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/model_parallel/models/roberta/model.py (263:268) - src/fairseq/fairseq/models/roberta/model.py (382:387) duplicated block id: 293 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/quantization/pq/modules/qconv.py (37:42) - src/fairseq/fairseq/modules/quantization/pq/utils.py (126:131) duplicated block id: 294 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/longformer_multihead_attention.py (84:90) - src/fairseq/fairseq/modules/multihead_attention.py (65:71) duplicated block id: 295 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/optim/adagrad.py (21:34) - src/fairseq/fairseq/optim/nag.py (24:37) duplicated block id: 296 size: 6 cleaned lines of code in 2 files: - src/convert_model_to_long.py (50:56) - src/convert_model_to_long.py (120:126) duplicated block id: 297 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/transformer_layer.py (41:47) - src/fairseq/fairseq/modules/transformer_layer.py (192:198) duplicated block id: 298 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/sparse_transformer_sentence_encoder_layer.py (17:22) - src/fairseq/fairseq/modules/transformer_sentence_encoder.py (79:84) duplicated block id: 299 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/nat/insertion_transformer.py (213:219) - src/fairseq/fairseq/models/nat/levenshtein_transformer.py (275:280) duplicated block id: 300 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/gumbel_vector_quantizer.py (129:134) - src/fairseq/fairseq/modules/kmeans_vector_quantizer.py (107:112) duplicated block id: 301 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/dynamicconv_layer/dynamicconv_layer.py (30:37) - src/fairseq/fairseq/modules/lightconv_layer/lightconv_layer.py (29:36) duplicated block id: 302 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/lightconv_layer/lightconv_cuda_kernel.cu (131:137) - src/fairseq/fairseq/modules/lightconv_layer/lightconv_cuda_kernel.cu (259:265) duplicated block id: 303 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/sparse_transformer_sentence_encoder.py (39:45) - src/fairseq/fairseq/modules/sparse_transformer_sentence_encoder_layer.py (24:30) duplicated block id: 304 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/dynamicconv_layer/dynamicconv_cuda_kernel.cu (1:6) - src/fairseq/fairseq/modules/lightconv_layer/lightconv_cuda.cuh (1:6) duplicated block id: 305 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/tasks/multilingual_denoising.py (122:127) - src/fairseq/fairseq/tasks/sentence_ranking.py (101:106) duplicated block id: 306 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/fconv.py (65:72) - src/fairseq/fairseq/models/fconv_self_att.py (69:76) duplicated block id: 307 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/dynamicconv_layer/dynamicconv_cuda.cuh (1:6) - src/fairseq/fairseq/modules/lightconv_layer/lightconv_cuda_kernel.cu (1:6) duplicated block id: 308 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/data/legacy/masked_lm_dataset.py (312:318) - src/fairseq/fairseq/data/monolingual_dataset.py (190:196) duplicated block id: 309 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/optim/fused_adam.py (215:221) - src/fairseq/fairseq/optim/nag.py (48:54) duplicated block id: 310 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/trainer.py (376:383) - src/fairseq/fairseq/trainer.py (572:577) duplicated block id: 311 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/transformer.py (1078:1083) - src/fairseq/fairseq/modules/transformer_layer.py (418:423) duplicated block id: 312 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/tasks/language_modeling.py (91:96) - src/fairseq/fairseq/tasks/sentence_prediction.py (57:62) duplicated block id: 313 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/model_parallel/models/roberta/model.py (39:44) - src/fairseq/fairseq/model_parallel/modules/transformer_sentence_encoder_layer.py (16:21) duplicated block id: 314 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/model_parallel/modules/multihead_attention.py (184:189) - src/fairseq/fairseq/modules/multihead_attention.py (269:274) duplicated block id: 315 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/benchmark/dummy_masked_lm.py (84:91) - src/fairseq/fairseq/tasks/language_modeling.py (270:279) duplicated block id: 316 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/optim/fused_lamb.py (29:42) - src/fairseq/fairseq/optim/sgd.py (23:36) duplicated block id: 317 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/model_parallel/models/transformer_lm.py (75:80) - src/fairseq/fairseq/model_parallel/models/transformer_lm.py (87:92) duplicated block id: 318 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/fconv_self_att.py (69:76) - src/fairseq/fairseq/models/lstm.py (32:39) duplicated block id: 319 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/fairseq_decoder.py (58:63) - src/fairseq/fairseq/models/transformer.py (291:296) duplicated block id: 320 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/optim/fp16_optimizer.py (92:105) - src/fairseq/fairseq/optim/fp16_optimizer.py (299:312) duplicated block id: 321 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/fairseq_decoder.py (58:63) - src/fairseq/fairseq/models/fairseq_model.py (46:51) duplicated block id: 322 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/fconv.py (14:19) - src/fairseq/fairseq/models/lstm.py (12:17) duplicated block id: 323 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/model_parallel/models/transformer.py (47:52) - src/fairseq/fairseq/model_parallel/modules/multihead_attention.py (46:51) duplicated block id: 324 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/tasks/multilingual_masked_lm.py (311:317) - src/fairseq/fairseq/tasks/sentence_ranking.py (202:208) duplicated block id: 325 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/tasks/masked_lm.py (204:210) - src/fairseq/fairseq/tasks/multilingual_masked_lm.py (311:317) duplicated block id: 326 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq_cli/interactive.py (23:28) - src/fairseq/fairseq_cli/train.py (33:38) duplicated block id: 327 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/model_parallel/megatron_trainer.py (30:35) - src/fairseq/fairseq/model_parallel/modules/multihead_attention.py (46:51) duplicated block id: 328 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/benchmark/dummy_masked_lm.py (86:92) - src/fairseq/fairseq/tasks/multilingual_masked_lm.py (311:317) duplicated block id: 329 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/tasks/multilingual_masked_lm.py (145:150) - src/fairseq/fairseq/tasks/sentence_ranking.py (101:106) duplicated block id: 330 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/tasks/multilingual_translation.py (81:86) - src/fairseq/fairseq/tasks/translation.py (184:189) duplicated block id: 331 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/transformer_sentence_encoder.py (79:84) - src/fairseq/fairseq/modules/transformer_sentence_encoder_layer.py (27:32) duplicated block id: 332 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/lightconv_lm.py (122:127) - src/fairseq/fairseq/models/transformer_lm.py (156:161) duplicated block id: 333 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/lightconv_layer/lightconv_cuda_kernel.cu (139:144) - src/fairseq/fairseq/modules/lightconv_layer/lightconv_cuda_kernel.cu (267:272) duplicated block id: 334 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/tasks/masked_lm.py (96:101) - src/fairseq/fairseq/tasks/sentence_ranking.py (101:106) duplicated block id: 335 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/model_parallel/criterions/vocab_parallel_cross_entropy.py (24:29) - src/fairseq/fairseq/model_parallel/models/transformer.py (47:52) duplicated block id: 336 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/tasks/denoising.py (163:171) - src/fairseq/fairseq/tasks/masked_lm.py (204:210) duplicated block id: 337 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/adaptive_input.py (27:33) - src/fairseq/fairseq/modules/adaptive_softmax.py (59:65) duplicated block id: 338 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/quantization/pq/pq.py (53:58) - src/fairseq/fairseq/modules/quantization/pq/utils.py (83:88) duplicated block id: 339 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/lightconv.py (441:448) - src/fairseq/fairseq/models/transformer.py (994:1001) duplicated block id: 340 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/masked_lm.py (60:65) - src/fairseq/fairseq/models/transformer.py (113:118) duplicated block id: 341 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq_cli/interactive.py (23:28) - src/fairseq/fairseq_cli/preprocess.py (23:28) duplicated block id: 342 size: 6 cleaned lines of code in 2 files: - src/prepare_data.py (140:145) - src/prepare_data.py (298:303) duplicated block id: 343 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/lightconv.py (113:118) - src/fairseq/fairseq/models/lightconv_lm.py (44:49) duplicated block id: 344 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/lightconv_layer/cuda_function_gen.py (98:103) - src/fairseq/fairseq/modules/lightconv_layer/cuda_function_gen.py (107:112) duplicated block id: 345 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/quantization/pq/modules/qlinear.py (52:57) - src/fairseq/fairseq/modules/quantization/pq/pq.py (114:119) duplicated block id: 346 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/lightconv.py (14:19) - src/fairseq/fairseq/models/lstm.py (12:17) duplicated block id: 347 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/masked_lm.py (140:145) - src/fairseq/fairseq/models/roberta/model.py (295:300) duplicated block id: 348 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/fconv.py (81:86) - src/fairseq/fairseq/models/fconv_self_att.py (81:86) duplicated block id: 349 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/tasks/denoising.py (107:118) - src/fairseq/fairseq/tasks/language_modeling.py (156:168) duplicated block id: 350 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/quantization/scalar/modules/qemb.py (72:78) - src/fairseq/fairseq/modules/quantization/scalar/modules/qlinear.py (57:63) duplicated block id: 351 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/fconv.py (647:652) - src/fairseq/fairseq/models/fconv.py (663:668) duplicated block id: 352 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/tasks/masked_lm.py (204:210) - src/fairseq/fairseq/tasks/sentence_prediction.py (239:245) duplicated block id: 353 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/tasks/denoising.py (163:171) - src/fairseq/fairseq/tasks/sentence_ranking.py (202:208) duplicated block id: 354 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/fconv.py (178:183) - src/fairseq/fairseq/models/fconv.py (390:395) duplicated block id: 355 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/tasks/denoising.py (163:171) - src/fairseq/fairseq/tasks/multilingual_masked_lm.py (311:317) duplicated block id: 356 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/model_parallel/criterions/vocab_parallel_cross_entropy.py (24:29) - src/fairseq/fairseq/model_parallel/megatron_trainer.py (30:35) duplicated block id: 357 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/optim/adagrad.py (21:34) - src/fairseq/fairseq/optim/fused_lamb.py (29:42) duplicated block id: 358 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/lstm.py (165:170) - src/fairseq/fairseq/models/lstm_lm.py (96:101) duplicated block id: 359 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/benchmark/dummy_masked_lm.py (86:92) - src/fairseq/fairseq/tasks/sentence_ranking.py (202:208) duplicated block id: 360 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/model_parallel/modules/multihead_attention.py (18:23) - src/fairseq/fairseq/model_parallel/modules/transformer_layer.py (15:20) duplicated block id: 361 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/criterions/sentence_prediction.py (60:65) - src/fairseq/fairseq/criterions/sentence_ranking.py (82:87) duplicated block id: 362 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/data/audio/raw_audio_dataset.py (20:25) - src/fairseq/fairseq/data/audio/raw_audio_dataset.py (125:130) duplicated block id: 363 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/tasks/sentence_prediction.py (57:62) - src/fairseq/fairseq/tasks/sentence_ranking.py (54:59) duplicated block id: 364 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/model_parallel/modules/multihead_attention.py (279:284) - src/fairseq/fairseq/modules/multihead_attention.py (403:408) duplicated block id: 365 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/transformer_layer.py (68:73) - src/fairseq/fairseq/modules/transformer_layer.py (79:84) duplicated block id: 366 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/lightconv.py (684:689) - src/fairseq/fairseq/modules/dynamic_convolution.py (34:39) duplicated block id: 367 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/benchmark/dummy_lm.py (75:82) - src/fairseq/fairseq/tasks/language_modeling.py (270:279) duplicated block id: 368 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/data/indexed_dataset.py (165:172) - src/fairseq/fairseq/data/indexed_dataset.py (274:281) duplicated block id: 369 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/cuda_utils.cu (1:6) - src/fairseq/fairseq/modules/lightconv_layer/lightconv_cuda_kernel.cu (1:6) duplicated block id: 370 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/optim/adam.py (127:133) - src/fairseq/fairseq/optim/fused_adam.py (215:221) duplicated block id: 371 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/benchmark/dummy_lm.py (77:83) - src/fairseq/fairseq/tasks/masked_lm.py (204:210) duplicated block id: 372 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/dynamicconv_layer/dynamicconv_layer.py (129:134) - src/fairseq/fairseq/modules/lightweight_convolution.py (170:175) duplicated block id: 373 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/model_parallel/models/roberta/model.py (39:44) - src/fairseq/fairseq/model_parallel/modules/transformer_layer.py (15:20) duplicated block id: 374 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/tasks/multilingual_denoising.py (79:89) - src/fairseq/fairseq/tasks/multilingual_masked_lm.py (109:119) duplicated block id: 375 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/model_parallel/modules/multihead_attention.py (156:163) - src/fairseq/fairseq/modules/multihead_attention.py (241:248) duplicated block id: 376 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/tasks/sentence_prediction.py (151:156) - src/fairseq/fairseq/tasks/sentence_ranking.py (130:135) duplicated block id: 377 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/optim/fused_adam.py (91:97) - src/fairseq/fairseq/optim/nag.py (48:54) duplicated block id: 378 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/tasks/sentence_prediction.py (95:100) - src/fairseq/fairseq/tasks/sentence_ranking.py (84:89) duplicated block id: 379 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/bart/model.py (277:282) - src/fairseq/fairseq/models/roberta/model.py (274:279) duplicated block id: 380 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/benchmark/dummy_masked_lm.py (86:92) - src/fairseq/fairseq/tasks/sentence_prediction.py (239:245) duplicated block id: 381 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/clib/libnat_cuda/edit_dist.cu (86:91) - src/fairseq/fairseq/clib/libnat_cuda/edit_dist.cu (165:170) duplicated block id: 382 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/model_parallel/models/roberta/model.py (98:103) - src/fairseq/fairseq/models/bart/model.py (155:160) duplicated block id: 383 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/lightweight_convolution.py (62:68) - src/fairseq/fairseq/modules/lightweight_convolution.py (132:138) duplicated block id: 384 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/sparse_transformer_sentence_encoder.py (22:27) - src/fairseq/fairseq/modules/sparse_transformer_sentence_encoder_layer.py (17:22) duplicated block id: 385 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/benchmark/dummy_lm.py (77:83) - src/fairseq/fairseq/tasks/denoising.py (163:171) duplicated block id: 386 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/lightconv.py (355:360) - src/fairseq/fairseq/models/transformer.py (783:788) duplicated block id: 387 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/fconv.py (366:372) - src/fairseq/fairseq/models/fconv_self_att.py (321:327) duplicated block id: 388 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/dynamic_convolution.py (124:129) - src/fairseq/fairseq/modules/lightweight_convolution.py (170:175) duplicated block id: 389 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/tasks/cross_lingual_lm.py (41:47) - src/fairseq/fairseq/tasks/legacy_masked_lm.py (38:44) duplicated block id: 390 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/optim/fused_lamb.py (29:42) - src/fairseq/fairseq/optim/nag.py (24:37) duplicated block id: 391 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/model_parallel/megatron_trainer.py (30:35) - src/fairseq/fairseq/model_parallel/models/transformer_lm.py (34:39) duplicated block id: 392 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/criterions/masked_lm.py (68:75) - src/fairseq/fairseq/model_parallel/criterions/vocab_parallel_cross_entropy.py (49:56) duplicated block id: 393 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/lightconv.py (96:101) - src/fairseq/fairseq/models/transformer.py (112:117) duplicated block id: 394 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/data/legacy/masked_lm_dictionary.py (14:19) - src/fairseq/fairseq/data/legacy/masked_lm_dictionary.py (36:41) duplicated block id: 395 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/model_parallel/models/transformer.py (47:52) - src/fairseq/fairseq/model_parallel/models/transformer_lm.py (34:39) duplicated block id: 396 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/data/denoising_dataset.py (61:66) - src/fairseq/fairseq/data/language_pair_dataset.py (83:88) duplicated block id: 397 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq_cli/generate.py (120:125) - src/fairseq/fairseq_cli/interactive.py (128:133) duplicated block id: 398 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/benchmark/dummy_lm.py (77:83) - src/fairseq/fairseq/tasks/multilingual_masked_lm.py (311:317) duplicated block id: 399 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/dynamicconv_layer/dynamicconv_cuda.cuh (1:6) - src/fairseq/fairseq/modules/dynamicconv_layer/dynamicconv_cuda_kernel.cu (1:6) duplicated block id: 400 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/model_parallel/models/roberta/model.py (150:155) - src/fairseq/fairseq/models/bart/model.py (277:282) duplicated block id: 401 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/benchmark/dummy_masked_lm.py (86:92) - src/fairseq/fairseq/tasks/denoising.py (163:171) duplicated block id: 402 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/nat/insertion_transformer.py (133:139) - src/fairseq/fairseq/models/nat/nonautoregressive_transformer.py (73:79) duplicated block id: 403 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/model_parallel/models/roberta/model.py (39:44) - src/fairseq/fairseq/model_parallel/modules/multihead_attention.py (18:23) duplicated block id: 404 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/clib/libnat_cuda/edit_dist.cu (309:314) - src/fairseq/fairseq/clib/libnat_cuda/edit_dist.cu (321:326) duplicated block id: 405 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/lightconv_layer/lightconv_cuda.cuh (48:53) - src/fairseq/fairseq/modules/lightconv_layer/lightconv_cuda_kernel.cu (267:272) duplicated block id: 406 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/model_parallel/criterions/vocab_parallel_cross_entropy.py (24:29) - src/fairseq/fairseq/model_parallel/modules/multihead_attention.py (46:51) duplicated block id: 407 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/models/nat/levenshtein_transformer.py (73:79) - src/fairseq/fairseq/models/nat/nonautoregressive_transformer.py (73:79) duplicated block id: 408 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/tasks/language_modeling.py (91:96) - src/fairseq/fairseq/tasks/masked_lm.py (62:67) duplicated block id: 409 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/lightconv_layer/lightconv_cuda.cuh (68:73) - src/fairseq/fairseq/modules/lightconv_layer/lightconv_cuda_kernel.cu (139:144) duplicated block id: 410 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq_cli/interactive.py (23:28) - src/fairseq/fairseq_cli/validate.py (18:23) duplicated block id: 411 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/tasks/masked_lm.py (62:67) - src/fairseq/fairseq/tasks/sentence_prediction.py (57:62) duplicated block id: 412 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/optim/adam.py (127:133) - src/fairseq/fairseq/optim/fused_adam.py (91:97) duplicated block id: 413 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/model_parallel/criterions/vocab_parallel_cross_entropy.py (24:29) - src/fairseq/fairseq/model_parallel/models/transformer_lm.py (34:39) duplicated block id: 414 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq_cli/preprocess.py (23:28) - src/fairseq/fairseq_cli/validate.py (18:23) duplicated block id: 415 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/lightconv_layer/lightconv_cuda.cuh (1:6) - src/fairseq/fairseq/modules/lightconv_layer/lightconv_cuda_kernel.cu (1:6) duplicated block id: 416 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/cuda_utils.cu (1:6) - src/fairseq/fairseq/modules/lightconv_layer/lightconv_cuda.cuh (1:6) duplicated block id: 417 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/dynamicconv_layer/dynamicconv_cuda_kernel.cu (1:6) - src/fairseq/fairseq/modules/lightconv_layer/lightconv_cuda_kernel.cu (1:6) duplicated block id: 418 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/model_parallel/models/transformer_lm.py (34:39) - src/fairseq/fairseq/model_parallel/modules/multihead_attention.py (46:51) duplicated block id: 419 size: 6 cleaned lines of code in 2 files: - src/fairseq/fairseq/modules/cuda_utils.cu (1:6) - src/fairseq/fairseq/modules/dynamicconv_layer/dynamicconv_cuda.cuh (1:6)