duplicated block id: 1 size: 100 cleaned lines of code in 2 files: - fairseq/tasks/translation.py (309:436) - fairseq/tasks/translation_with_unlikelihood.py (195:322) duplicated block id: 2 size: 68 cleaned lines of code in 2 files: - scripts/launch_multitask_cnndm.py (69:144) - scripts/launch_multitask_xsum.py (69:145) duplicated block id: 3 size: 60 cleaned lines of code in 2 files: - scripts/launch_cnndm.py (76:144) - scripts/launch_multitask_xsum.py (78:145) duplicated block id: 4 size: 60 cleaned lines of code in 2 files: - scripts/launch_cnndm.py (76:144) - scripts/launch_multitask_cnndm.py (78:144) duplicated block id: 5 size: 53 cleaned lines of code in 2 files: - fairseq/benchmark/dummy_lm.py (40:111) - fairseq/benchmark/dummy_masked_lm.py (51:122) duplicated block id: 6 size: 53 cleaned lines of code in 2 files: - fairseq/clib/libnat_cuda/edit_dist.cu (101:161) - fairseq/clib/libnat_cuda/edit_dist.cu (181:241) duplicated block id: 7 size: 47 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (557:609) - preprocess/sm_inference_asum.py (687:740) duplicated block id: 8 size: 38 cleaned lines of code in 2 files: - fairseq/modules/dynamic_convolution.py (136:183) - fairseq/modules/dynamicconv_layer/dynamicconv_layer.py (136:183) duplicated block id: 9 size: 38 cleaned lines of code in 2 files: - fairseq/models/nat/iterative_nonautoregressive_transformer.py (151:191) - fairseq/models/nat/nonautoregressive_transformer.py (376:416) duplicated block id: 10 size: 38 cleaned lines of code in 2 files: - scripts/launch_multitask_cnndm.py (36:75) - scripts/launch_multitask_newsroom.py (36:76) duplicated block id: 11 size: 35 cleaned lines of code in 2 files: - fairseq/models/nat/insertion_transformer.py (242:277) - fairseq/models/nat/nonautoregressive_transformer.py (376:411) duplicated block id: 12 size: 35 cleaned lines of code in 2 files: - fairseq/models/nat/insertion_transformer.py (242:277) - fairseq/models/nat/iterative_nonautoregressive_transformer.py (151:186) duplicated block id: 13 size: 35 cleaned lines of code in 2 files: - scripts/launch_sagemaker_unlikelihood_cnndm.py (69:109) - scripts/launch_sagemaker_unlikelihood_xsum.py (69:109) duplicated block id: 14 size: 34 cleaned lines of code in 2 files: - fairseq/tasks/translation.py (216:253) - fairseq/tasks/translation_with_unlikelihood.py (85:122) duplicated block id: 15 size: 34 cleaned lines of code in 2 files: - fairseq/models/nat/levenshtein_transformer.py (402:436) - fairseq/models/nat/nonautoregressive_transformer.py (376:410) duplicated block id: 16 size: 34 cleaned lines of code in 2 files: - fairseq/models/nat/iterative_nonautoregressive_transformer.py (151:185) - fairseq/models/nat/levenshtein_transformer.py (402:436) duplicated block id: 17 size: 34 cleaned lines of code in 2 files: - fairseq/models/nat/insertion_transformer.py (242:276) - fairseq/models/nat/levenshtein_transformer.py (402:436) duplicated block id: 18 size: 34 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (446:480) - preprocess/sm_inference_asum.py (487:522) duplicated block id: 19 size: 33 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (277:309) - preprocess/sm_inference_asum.py (338:371) duplicated block id: 20 size: 33 cleaned lines of code in 2 files: - fairseq/models/bart/model.py (160:197) - fairseq/models/roberta/model.py (153:190) duplicated block id: 21 size: 32 cleaned lines of code in 2 files: - scripts/launch_multitask_newsroom.py (79:116) - scripts/launch_multitask_xsum.py (78:116) duplicated block id: 22 size: 32 cleaned lines of code in 2 files: - scripts/launch_multitask_cnndm.py (78:115) - scripts/launch_multitask_newsroom.py (79:116) duplicated block id: 23 size: 32 cleaned lines of code in 2 files: - scripts/launch_cnndm.py (76:115) - scripts/launch_multitask_newsroom.py (79:116) duplicated block id: 24 size: 31 cleaned lines of code in 2 files: - fairseq/models/nat/nonautoregressive_transformer.py (375:405) - fairseq/models/transformer.py (928:958) duplicated block id: 25 size: 31 cleaned lines of code in 2 files: - fairseq/clib/libnat/edit_dist.cpp (58:98) - fairseq/clib/libnat/edit_dist.cpp (130:170) duplicated block id: 26 size: 30 cleaned lines of code in 2 files: - scripts/launch_multitask_cnndm.py (114:144) - scripts/launch_xsum.py (92:122) duplicated block id: 27 size: 30 cleaned lines of code in 2 files: - fairseq/models/nat/iterative_nonautoregressive_transformer.py (151:180) - fairseq/models/transformer.py (929:958) duplicated block id: 28 size: 30 cleaned lines of code in 2 files: - fairseq/models/nat/levenshtein_transformer.py (402:431) - fairseq/models/transformer.py (929:958) duplicated block id: 29 size: 30 cleaned lines of code in 2 files: - scripts/launch_cnndm.py (114:144) - scripts/launch_xsum.py (92:122) duplicated block id: 30 size: 30 cleaned lines of code in 2 files: - scripts/launch_multitask_xsum.py (115:145) - scripts/launch_xsum.py (92:122) duplicated block id: 31 size: 30 cleaned lines of code in 2 files: - fairseq/models/nat/insertion_transformer.py (242:271) - fairseq/models/transformer.py (929:958) duplicated block id: 32 size: 30 cleaned lines of code in 2 files: - scripts/launch_sagemaker_unlikelihood_cnndm.py (32:62) - scripts/launch_sagemaker_unlikelihood_xsum.py (32:62) duplicated block id: 33 size: 29 cleaned lines of code in 2 files: - fairseq/tasks/masked_lm.py (157:186) - fairseq/tasks/multilingual_masked_lm.py (264:293) duplicated block id: 34 size: 28 cleaned lines of code in 2 files: - scripts/launch_sagemaker_unlikelihood_cnndm.py (111:139) - scripts/launch_sagemaker_unlikelihood_xsum.py (111:139) duplicated block id: 35 size: 28 cleaned lines of code in 2 files: - scripts/launch_newsroom.py (28:57) - scripts/launch_xsum.py (26:57) duplicated block id: 36 size: 27 cleaned lines of code in 2 files: - scripts/launch_multitask_newsroom.py (118:145) - scripts/launch_multitask_xsum.py (118:145) duplicated block id: 37 size: 27 cleaned lines of code in 2 files: - scripts/launch_cnndm.py (117:144) - scripts/launch_multitask_newsroom.py (118:145) duplicated block id: 38 size: 27 cleaned lines of code in 2 files: - scripts/launch_multitask_newsroom.py (118:145) - scripts/launch_xsum.py (95:122) duplicated block id: 39 size: 27 cleaned lines of code in 2 files: - scripts/launch_multitask_cnndm.py (117:144) - scripts/launch_multitask_newsroom.py (118:145) duplicated block id: 40 size: 26 cleaned lines of code in 2 files: - scripts/launch_cnndm.py (35:62) - scripts/launch_xsum.py (26:54) duplicated block id: 41 size: 26 cleaned lines of code in 2 files: - scripts/launch_cnndm.py (35:62) - scripts/launch_newsroom.py (28:55) duplicated block id: 42 size: 26 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (337:363) - preprocess/sm_inference_asum.py (486:512) duplicated block id: 43 size: 25 cleaned lines of code in 2 files: - fairseq/models/nat/cmlm_transformer.py (107:131) - fairseq/models/nat/levenshtein_transformer.py (402:426) duplicated block id: 44 size: 25 cleaned lines of code in 2 files: - fairseq/models/nat/cmlm_transformer.py (107:131) - fairseq/models/nat/nonautoregressive_transformer.py (376:400) duplicated block id: 45 size: 25 cleaned lines of code in 2 files: - fairseq/modules/dynamic_convolution.py (189:219) - fairseq/modules/dynamicconv_layer/dynamicconv_layer.py (184:214) duplicated block id: 46 size: 25 cleaned lines of code in 2 files: - fairseq/models/nat/cmlm_transformer.py (107:131) - fairseq/models/transformer.py (929:953) duplicated block id: 47 size: 25 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (338:363) - preprocess/sm_inference_asum.py (446:470) duplicated block id: 48 size: 25 cleaned lines of code in 2 files: - fairseq/models/nat/cmlm_transformer.py (107:131) - fairseq/models/nat/insertion_transformer.py (242:266) duplicated block id: 49 size: 25 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (277:301) - preprocess/sm_inference_asum.py (487:512) duplicated block id: 50 size: 25 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (277:301) - preprocess/sm_inference_asum.py (446:470) duplicated block id: 51 size: 25 cleaned lines of code in 2 files: - fairseq/models/nat/cmlm_transformer.py (107:131) - fairseq/models/nat/iterative_nonautoregressive_transformer.py (151:175) duplicated block id: 52 size: 23 cleaned lines of code in 2 files: - scripts/launch_newsroom.py (67:92) - scripts/launch_xsum.py (68:93) duplicated block id: 53 size: 22 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (129:152) - preprocess/sm_inference_asum.py (159:182) duplicated block id: 54 size: 21 cleaned lines of code in 2 files: - scripts/launch_cnndm.py (72:96) - scripts/launch_xsum.py (64:87) duplicated block id: 55 size: 20 cleaned lines of code in 2 files: - fairseq/tasks/masked_lm.py (36:56) - fairseq/tasks/multilingual_masked_lm.py (40:60) duplicated block id: 56 size: 20 cleaned lines of code in 2 files: - fairseq/tasks/multilingual_denoising.py (199:221) - fairseq/tasks/multilingual_masked_lm.py (240:265) duplicated block id: 57 size: 20 cleaned lines of code in 2 files: - fairseq/tasks/translation.py (262:292) - fairseq/tasks/translation_with_unlikelihood.py (138:168) duplicated block id: 58 size: 19 cleaned lines of code in 2 files: - scripts/launch_multitask_cnndm.py (53:73) - scripts/launch_newsroom.py (43:63) duplicated block id: 59 size: 19 cleaned lines of code in 2 files: - scripts/launch_multitask_newsroom.py (76:98) - scripts/launch_newsroom.py (65:86) duplicated block id: 60 size: 19 cleaned lines of code in 2 files: - scripts/launch_multitask_newsroom.py (54:74) - scripts/launch_newsroom.py (43:63) duplicated block id: 61 size: 18 cleaned lines of code in 2 files: - scripts/launch_newsroom.py (103:121) - scripts/launch_xsum.py (104:122) duplicated block id: 62 size: 18 cleaned lines of code in 2 files: - scripts/launch_multitask_cnndm.py (126:144) - scripts/launch_newsroom.py (103:121) duplicated block id: 63 size: 18 cleaned lines of code in 2 files: - scripts/launch_cnndm.py (126:144) - scripts/launch_sagemaker_unlikelihood_xsum.py (121:139) duplicated block id: 64 size: 18 cleaned lines of code in 2 files: - scripts/launch_newsroom.py (103:121) - scripts/launch_sagemaker_unlikelihood_cnndm.py (121:139) duplicated block id: 65 size: 18 cleaned lines of code in 2 files: - scripts/launch_multitask_xsum.py (127:145) - scripts/launch_newsroom.py (103:121) duplicated block id: 66 size: 18 cleaned lines of code in 2 files: - scripts/launch_newsroom.py (103:121) - scripts/launch_sagemaker_unlikelihood_xsum.py (121:139) duplicated block id: 67 size: 18 cleaned lines of code in 2 files: - scripts/launch_multitask_newsroom.py (127:145) - scripts/launch_sagemaker_unlikelihood_cnndm.py (121:139) duplicated block id: 68 size: 18 cleaned lines of code in 2 files: - scripts/launch_cnndm.py (126:144) - scripts/launch_sagemaker_unlikelihood_cnndm.py (121:139) duplicated block id: 69 size: 18 cleaned lines of code in 2 files: - scripts/launch_multitask_cnndm.py (126:144) - scripts/launch_sagemaker_unlikelihood_xsum.py (121:139) duplicated block id: 70 size: 18 cleaned lines of code in 2 files: - scripts/launch_multitask_cnndm.py (126:144) - scripts/launch_sagemaker_unlikelihood_cnndm.py (121:139) duplicated block id: 71 size: 18 cleaned lines of code in 2 files: - fairseq/tasks/masked_lm.py (89:110) - fairseq/tasks/multilingual_masked_lm.py (145:166) duplicated block id: 72 size: 18 cleaned lines of code in 2 files: - scripts/launch_multitask_xsum.py (127:145) - scripts/launch_sagemaker_unlikelihood_xsum.py (121:139) duplicated block id: 73 size: 18 cleaned lines of code in 2 files: - scripts/launch_cnndm.py (126:144) - scripts/launch_newsroom.py (103:121) duplicated block id: 74 size: 18 cleaned lines of code in 2 files: - scripts/launch_multitask_xsum.py (127:145) - scripts/launch_sagemaker_unlikelihood_cnndm.py (121:139) duplicated block id: 75 size: 18 cleaned lines of code in 2 files: - scripts/launch_sagemaker_unlikelihood_xsum.py (121:139) - scripts/launch_xsum.py (104:122) duplicated block id: 76 size: 18 cleaned lines of code in 2 files: - scripts/launch_multitask_newsroom.py (127:145) - scripts/launch_newsroom.py (103:121) duplicated block id: 77 size: 18 cleaned lines of code in 2 files: - scripts/launch_sagemaker_unlikelihood_cnndm.py (121:139) - scripts/launch_xsum.py (104:122) duplicated block id: 78 size: 18 cleaned lines of code in 2 files: - scripts/launch_multitask_newsroom.py (127:145) - scripts/launch_sagemaker_unlikelihood_xsum.py (121:139) duplicated block id: 79 size: 17 cleaned lines of code in 2 files: - scripts/launch_multitask_xsum.py (78:98) - scripts/launch_xsum.py (68:87) duplicated block id: 80 size: 17 cleaned lines of code in 2 files: - scripts/launch_multitask_xsum.py (78:98) - scripts/launch_newsroom.py (67:86) duplicated block id: 81 size: 17 cleaned lines of code in 2 files: - scripts/launch_multitask_cnndm.py (78:97) - scripts/launch_xsum.py (68:87) duplicated block id: 82 size: 17 cleaned lines of code in 2 files: - scripts/launch_multitask_cnndm.py (78:97) - scripts/launch_newsroom.py (67:86) duplicated block id: 83 size: 17 cleaned lines of code in 2 files: - fairseq/logging/progress_bar.py (165:184) - fairseq/logging/progress_bar.py (237:256) duplicated block id: 84 size: 17 cleaned lines of code in 2 files: - scripts/launch_cnndm.py (76:96) - scripts/launch_newsroom.py (67:86) duplicated block id: 85 size: 17 cleaned lines of code in 2 files: - scripts/launch_multitask_newsroom.py (79:98) - scripts/launch_xsum.py (68:87) duplicated block id: 86 size: 16 cleaned lines of code in 2 files: - fairseq/criterions/label_smoothed_cross_entropy.py (72:96) - fairseq/criterions/label_smoothed_cross_entropy_with_unlikelihood.py (146:170) duplicated block id: 87 size: 16 cleaned lines of code in 2 files: - fairseq/optim/adam.py (123:147) - fairseq/optim/adamax.py (87:111) duplicated block id: 88 size: 16 cleaned lines of code in 2 files: - scripts/launch_multitask_cnndm.py (36:51) - scripts/launch_multitask_xsum.py (36:51) duplicated block id: 89 size: 16 cleaned lines of code in 2 files: - scripts/launch_multitask_newsroom.py (36:51) - scripts/launch_multitask_xsum.py (36:51) duplicated block id: 90 size: 16 cleaned lines of code in 2 files: - fairseq/sequence_generator.py (583:598) - fairseq/sequence_generator.py (700:715) duplicated block id: 91 size: 16 cleaned lines of code in 2 files: - fairseq/criterions/sentence_prediction.py (77:102) - fairseq/criterions/sentence_ranking.py (94:119) duplicated block id: 92 size: 15 cleaned lines of code in 2 files: - fairseq/models/bart/hub_interface.py (199:216) - fairseq/models/roberta/hub_interface.py (86:103) duplicated block id: 93 size: 15 cleaned lines of code in 2 files: - fairseq/tasks/masked_lm.py (136:150) - fairseq/tasks/multilingual_masked_lm.py (183:197) duplicated block id: 94 size: 15 cleaned lines of code in 2 files: - preprocess/evaluate_hypo.py (87:101) - preprocess/evaluate_hypo.py (255:269) duplicated block id: 95 size: 14 cleaned lines of code in 2 files: - scripts/launch_cnndm.py (35:48) - scripts/launch_multitask_cnndm.py (36:49) duplicated block id: 96 size: 14 cleaned lines of code in 2 files: - fairseq/models/nat/levenshtein_transformer.py (348:366) - fairseq/models/nat/nonautoregressive_transformer.py (282:300) duplicated block id: 97 size: 14 cleaned lines of code in 2 files: - fairseq/modules/sparse_transformer_sentence_encoder.py (28:41) - fairseq/modules/transformer_sentence_encoder.py (84:97) duplicated block id: 98 size: 14 cleaned lines of code in 2 files: - fairseq/optim/fp16_optimizer.py (162:178) - fairseq/optim/fp16_optimizer.py (355:372) duplicated block id: 99 size: 14 cleaned lines of code in 2 files: - fairseq/modules/lightconv_layer/lightconv_cuda_kernel.cu (51:67) - fairseq/modules/lightconv_layer/lightconv_cuda_kernel.cu (116:132) duplicated block id: 100 size: 14 cleaned lines of code in 2 files: - scripts/launch_multitask_xsum.py (36:49) - scripts/launch_newsroom.py (28:41) duplicated block id: 101 size: 14 cleaned lines of code in 2 files: - scripts/launch_multitask_cnndm.py (36:49) - scripts/launch_xsum.py (26:39) duplicated block id: 102 size: 14 cleaned lines of code in 2 files: - scripts/launch_cnndm.py (35:48) - scripts/launch_multitask_xsum.py (36:49) duplicated block id: 103 size: 14 cleaned lines of code in 2 files: - scripts/launch_multitask_cnndm.py (36:49) - scripts/launch_newsroom.py (28:41) duplicated block id: 104 size: 14 cleaned lines of code in 2 files: - scripts/launch_multitask_newsroom.py (36:49) - scripts/launch_newsroom.py (28:41) duplicated block id: 105 size: 14 cleaned lines of code in 2 files: - scripts/launch_multitask_xsum.py (36:49) - scripts/launch_xsum.py (26:39) duplicated block id: 106 size: 14 cleaned lines of code in 2 files: - fairseq/data/append_token_dataset.py (26:42) - fairseq/data/prepend_token_dataset.py (26:42) duplicated block id: 107 size: 14 cleaned lines of code in 2 files: - scripts/launch_cnndm.py (35:48) - scripts/launch_multitask_newsroom.py (36:49) duplicated block id: 108 size: 14 cleaned lines of code in 2 files: - fairseq/tasks/multilingual_denoising.py (181:195) - fairseq/tasks/multilingual_masked_lm.py (224:238) duplicated block id: 109 size: 14 cleaned lines of code in 2 files: - fairseq/models/nat/cmlm_transformer.py (133:149) - fairseq/models/nat/iterative_nonautoregressive_transformer.py (177:193) duplicated block id: 110 size: 14 cleaned lines of code in 2 files: - fairseq/tasks/sentence_prediction.py (193:213) - fairseq/tasks/sentence_ranking.py (160:180) duplicated block id: 111 size: 14 cleaned lines of code in 2 files: - scripts/launch_multitask_newsroom.py (36:49) - scripts/launch_xsum.py (26:39) duplicated block id: 112 size: 13 cleaned lines of code in 2 files: - fairseq/modules/dynamic_convolution.py (217:233) - fairseq/modules/lightweight_convolution.py (232:248) duplicated block id: 113 size: 13 cleaned lines of code in 2 files: - fairseq/tasks/translation.py (284:304) - fairseq/tasks/translation_lev.py (32:52) duplicated block id: 114 size: 13 cleaned lines of code in 2 files: - scripts/launch_multitask_xsum.py (102:117) - scripts/launch_sagemaker_unlikelihood_xsum.py (96:111) duplicated block id: 115 size: 13 cleaned lines of code in 2 files: - scripts/launch_multitask_newsroom.py (54:68) - scripts/launch_xsum.py (41:57) duplicated block id: 116 size: 13 cleaned lines of code in 2 files: - scripts/launch_multitask_cnndm.py (13:33) - scripts/launch_multitask_xsum.py (13:33) duplicated block id: 117 size: 13 cleaned lines of code in 2 files: - scripts/launch_cnndm.py (13:32) - scripts/launch_multitask_newsroom.py (13:33) duplicated block id: 118 size: 13 cleaned lines of code in 2 files: - scripts/launch_multitask_cnndm.py (13:33) - scripts/launch_multitask_newsroom.py (13:33) duplicated block id: 119 size: 13 cleaned lines of code in 2 files: - scripts/launch_cnndm.py (100:116) - scripts/launch_sagemaker_unlikelihood_xsum.py (96:111) duplicated block id: 120 size: 13 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (1018:1030) - preprocess/sm_inference_asum.py (1034:1046) duplicated block id: 121 size: 13 cleaned lines of code in 2 files: - scripts/launch_multitask_newsroom.py (13:33) - scripts/launch_multitask_xsum.py (13:33) duplicated block id: 122 size: 13 cleaned lines of code in 2 files: - scripts/launch_cnndm.py (13:32) - scripts/launch_multitask_cnndm.py (13:33) duplicated block id: 123 size: 13 cleaned lines of code in 2 files: - scripts/launch_multitask_cnndm.py (101:116) - scripts/launch_sagemaker_unlikelihood_xsum.py (96:111) duplicated block id: 124 size: 13 cleaned lines of code in 2 files: - preprocess/multi_gpu_generate.py (60:72) - preprocess/multi_gpu_generate.py (78:90) duplicated block id: 125 size: 13 cleaned lines of code in 2 files: - scripts/launch_cnndm.py (13:32) - scripts/launch_multitask_xsum.py (13:33) duplicated block id: 126 size: 13 cleaned lines of code in 2 files: - scripts/launch_multitask_cnndm.py (53:67) - scripts/launch_xsum.py (41:57) duplicated block id: 127 size: 13 cleaned lines of code in 2 files: - fairseq/models/roberta/model_camembert.py (22:36) - fairseq/models/roberta/model_xlmr.py (23:37) duplicated block id: 128 size: 13 cleaned lines of code in 2 files: - fairseq/models/nat/levenshtein_transformer.py (198:211) - fairseq/models/nat/nonautoregressive_ensembles.py (183:195) duplicated block id: 129 size: 12 cleaned lines of code in 2 files: - fairseq/modules/sparse_transformer_sentence_encoder_layer.py (15:26) - fairseq/modules/transformer_sentence_encoder_layer.py (23:34) duplicated block id: 130 size: 12 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (312:323) - preprocess/sm_inference_asum.py (374:385) duplicated block id: 131 size: 12 cleaned lines of code in 2 files: - fairseq/modules/dynamicconv_layer/dynamicconv_cuda.cuh (39:50) - fairseq/modules/dynamicconv_layer/dynamicconv_cuda_kernel.cu (73:84) duplicated block id: 132 size: 12 cleaned lines of code in 2 files: - fairseq/tasks/masked_lm.py (116:127) - fairseq/tasks/multilingual_masked_lm.py (168:179) duplicated block id: 133 size: 12 cleaned lines of code in 2 files: - fairseq/models/lightconv.py (186:197) - fairseq/models/transformer.py (210:221) duplicated block id: 134 size: 12 cleaned lines of code in 2 files: - fairseq/data/denoising_dataset.py (45:60) - fairseq/data/language_pair_dataset.py (69:84) duplicated block id: 135 size: 12 cleaned lines of code in 2 files: - fairseq/tasks/masked_lm.py (62:76) - fairseq/tasks/multilingual_masked_lm.py (68:82) duplicated block id: 136 size: 12 cleaned lines of code in 2 files: - scripts/launch_multitask_newsroom.py (102:116) - scripts/launch_sagemaker_unlikelihood_xsum.py (96:110) duplicated block id: 137 size: 12 cleaned lines of code in 2 files: - fairseq/models/nat/cmlm_transformer.py (133:147) - fairseq/models/nat/nonautoregressive_transformer.py (402:416) duplicated block id: 138 size: 12 cleaned lines of code in 2 files: - fairseq_cli/preprocess.py (211:223) - fairseq_cli/preprocess.py (265:277) duplicated block id: 139 size: 12 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (250:263) - preprocess/sm_inference_asum.py (821:834) duplicated block id: 140 size: 12 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (46:59) - preprocess/sm_inference_asum.py (401:414) duplicated block id: 141 size: 12 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (629:641) - preprocess/sm_inference_asum.py (756:768) duplicated block id: 142 size: 11 cleaned lines of code in 2 files: - fairseq/benchmark/dummy_lm.py (21:33) - fairseq/benchmark/dummy_masked_lm.py (21:33) duplicated block id: 143 size: 11 cleaned lines of code in 2 files: - fairseq/tasks/translation.py (17:27) - preprocess/sm_inference_asum.py (20:30) duplicated block id: 144 size: 11 cleaned lines of code in 2 files: - fairseq/models/lightconv_lm.py (73:83) - fairseq/models/transformer_lm.py (98:108) duplicated block id: 145 size: 11 cleaned lines of code in 2 files: - scripts/launch_multitask_newsroom.py (39:49) - scripts/launch_sagemaker_unlikelihood_cnndm.py (38:48) duplicated block id: 146 size: 11 cleaned lines of code in 2 files: - scripts/launch_multitask_xsum.py (39:49) - scripts/launch_sagemaker_unlikelihood_xsum.py (38:48) duplicated block id: 147 size: 11 cleaned lines of code in 2 files: - fairseq/optim/fp16_optimizer.py (228:240) - fairseq/optim/fp16_optimizer.py (415:427) duplicated block id: 148 size: 11 cleaned lines of code in 2 files: - scripts/launch_cnndm.py (38:48) - scripts/launch_sagemaker_unlikelihood_xsum.py (38:48) duplicated block id: 149 size: 11 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (848:858) - preprocess/sm_inference_asum.py (884:894) duplicated block id: 150 size: 11 cleaned lines of code in 2 files: - fairseq/models/lightconv.py (165:178) - fairseq/models/transformer.py (185:198) duplicated block id: 151 size: 11 cleaned lines of code in 2 files: - scripts/launch_multitask_cnndm.py (39:49) - scripts/launch_sagemaker_unlikelihood_xsum.py (38:48) duplicated block id: 152 size: 11 cleaned lines of code in 2 files: - fairseq/models/nat/levenshtein_transformer.py (258:269) - fairseq/models/nat/nonautoregressive_transformer.py (151:162) duplicated block id: 153 size: 11 cleaned lines of code in 2 files: - scripts/launch_newsroom.py (31:41) - scripts/launch_sagemaker_unlikelihood_xsum.py (38:48) duplicated block id: 154 size: 11 cleaned lines of code in 2 files: - fairseq/models/lstm.py (52:62) - fairseq/models/lstm_lm.py (31:41) duplicated block id: 155 size: 11 cleaned lines of code in 2 files: - fairseq/models/lightconv_lm.py (38:48) - fairseq/models/transformer_lm.py (61:71) duplicated block id: 156 size: 11 cleaned lines of code in 2 files: - fairseq/optim/adamax.py (87:106) - fairseq/optim/nag.py (48:67) duplicated block id: 157 size: 11 cleaned lines of code in 2 files: - fairseq/models/transformer.py (93:105) - fairseq/models/transformer_lm.py (48:60) duplicated block id: 158 size: 11 cleaned lines of code in 2 files: - scripts/launch_cnndm.py (38:48) - scripts/launch_sagemaker_unlikelihood_cnndm.py (38:48) duplicated block id: 159 size: 11 cleaned lines of code in 2 files: - fairseq/modules/sparse_transformer_sentence_encoder.py (17:27) - fairseq/modules/transformer_sentence_encoder.py (72:82) duplicated block id: 160 size: 11 cleaned lines of code in 2 files: - fairseq/models/nat/insertion_transformer.py (68:79) - fairseq/models/nat/levenshtein_utils.py (112:123) duplicated block id: 161 size: 11 cleaned lines of code in 2 files: - scripts/launch_multitask_newsroom.py (115:125) - scripts/launch_newsroom.py (91:101) duplicated block id: 162 size: 11 cleaned lines of code in 2 files: - fairseq/criterions/label_smoothed_cross_entropy.py (19:29) - fairseq/criterions/label_smoothed_cross_entropy_with_unlikelihood.py (21:31) duplicated block id: 163 size: 11 cleaned lines of code in 2 files: - scripts/launch_multitask_cnndm.py (39:49) - scripts/launch_sagemaker_unlikelihood_cnndm.py (38:48) duplicated block id: 164 size: 11 cleaned lines of code in 2 files: - scripts/launch_cnndm.py (50:62) - scripts/launch_multitask_cnndm.py (53:65) duplicated block id: 165 size: 11 cleaned lines of code in 2 files: - scripts/launch_sagemaker_unlikelihood_cnndm.py (38:48) - scripts/launch_xsum.py (29:39) duplicated block id: 166 size: 11 cleaned lines of code in 2 files: - scripts/launch_sagemaker_unlikelihood_xsum.py (38:48) - scripts/launch_xsum.py (29:39) duplicated block id: 167 size: 11 cleaned lines of code in 2 files: - scripts/launch_multitask_xsum.py (39:49) - scripts/launch_sagemaker_unlikelihood_cnndm.py (38:48) duplicated block id: 168 size: 11 cleaned lines of code in 2 files: - fairseq_cli/eval_lm.py (120:131) - fairseq_cli/generate.py (100:112) duplicated block id: 169 size: 11 cleaned lines of code in 2 files: - scripts/launch_multitask_newsroom.py (102:115) - scripts/launch_sagemaker_unlikelihood_cnndm.py (96:109) duplicated block id: 170 size: 11 cleaned lines of code in 2 files: - scripts/launch_newsroom.py (31:41) - scripts/launch_sagemaker_unlikelihood_cnndm.py (38:48) duplicated block id: 171 size: 11 cleaned lines of code in 2 files: - fairseq/optim/adam.py (123:142) - fairseq/optim/nag.py (48:67) duplicated block id: 172 size: 11 cleaned lines of code in 2 files: - scripts/launch_multitask_cnndm.py (101:114) - scripts/launch_sagemaker_unlikelihood_cnndm.py (96:109) duplicated block id: 173 size: 11 cleaned lines of code in 2 files: - scripts/launch_multitask_xsum.py (102:115) - scripts/launch_sagemaker_unlikelihood_cnndm.py (96:109) duplicated block id: 174 size: 11 cleaned lines of code in 2 files: - scripts/launch_sagemaker_unlikelihood_cnndm.py (13:29) - scripts/launch_sagemaker_unlikelihood_xsum.py (13:29) duplicated block id: 175 size: 11 cleaned lines of code in 2 files: - scripts/launch_multitask_newsroom.py (39:49) - scripts/launch_sagemaker_unlikelihood_xsum.py (38:48) duplicated block id: 176 size: 11 cleaned lines of code in 2 files: - scripts/launch_cnndm.py (100:114) - scripts/launch_sagemaker_unlikelihood_cnndm.py (96:109) duplicated block id: 177 size: 11 cleaned lines of code in 2 files: - scripts/launch_cnndm.py (50:62) - scripts/launch_multitask_newsroom.py (54:66) duplicated block id: 178 size: 10 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (1021:1030) - preprocess/sm_inference_asum.py (1127:1136) duplicated block id: 179 size: 10 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (1021:1030) - preprocess/sm_inference_asum.py (1091:1100) duplicated block id: 180 size: 10 cleaned lines of code in 2 files: - fairseq/tasks/translation.py (206:215) - fairseq/tasks/translation_with_unlikelihood.py (73:82) duplicated block id: 181 size: 10 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (401:410) - preprocess/sm_inference_asum.py (687:696) duplicated block id: 182 size: 10 cleaned lines of code in 2 files: - fairseq/modules/dynamic_convolution.py (219:231) - fairseq/modules/dynamicconv_layer/dynamicconv_layer.py (114:126) duplicated block id: 183 size: 10 cleaned lines of code in 2 files: - scripts/launch_multitask_xsum.py (53:62) - scripts/launch_newsroom.py (43:52) duplicated block id: 184 size: 10 cleaned lines of code in 2 files: - scripts/launch_cnndm.py (50:59) - scripts/launch_multitask_xsum.py (53:62) duplicated block id: 185 size: 10 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (249:258) - preprocess/sm_inference_asum.py (401:410) duplicated block id: 186 size: 10 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (1091:1100) - preprocess/sm_inference_asum.py (1127:1136) duplicated block id: 187 size: 10 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (401:410) - preprocess/sm_inference_asum.py (557:566) duplicated block id: 188 size: 10 cleaned lines of code in 2 files: - fairseq/modules/dynamic_convolution.py (126:136) - fairseq/modules/dynamic_convolution.py (180:189) duplicated block id: 189 size: 10 cleaned lines of code in 2 files: - fairseq/models/nat/insertion_transformer.py (69:79) - fairseq/models/nat/levenshtein_utils.py (51:61) duplicated block id: 190 size: 10 cleaned lines of code in 2 files: - scripts/launch_multitask_cnndm.py (53:62) - scripts/launch_multitask_xsum.py (53:62) duplicated block id: 191 size: 10 cleaned lines of code in 2 files: - fairseq/tasks/translation.py (17:26) - fairseq/tasks/translation_with_unlikelihood.py (18:27) duplicated block id: 192 size: 10 cleaned lines of code in 2 files: - fairseq/modules/lightconv_layer/lightconv_cuda.cuh (45:54) - fairseq/modules/lightconv_layer/lightconv_cuda_kernel.cu (136:145) duplicated block id: 193 size: 10 cleaned lines of code in 2 files: - fairseq/data/append_token_dataset.py (14:24) - fairseq/data/prepend_token_dataset.py (14:24) duplicated block id: 194 size: 10 cleaned lines of code in 2 files: - fairseq/tasks/translation.py (298:307) - fairseq/tasks/translation_with_unlikelihood.py (174:183) duplicated block id: 195 size: 10 cleaned lines of code in 2 files: - fairseq/optim/adafactor.py (143:159) - fairseq/optim/adamax.py (95:111) duplicated block id: 196 size: 10 cleaned lines of code in 2 files: - scripts/launch_multitask_newsroom.py (54:63) - scripts/launch_multitask_xsum.py (53:62) duplicated block id: 197 size: 10 cleaned lines of code in 2 files: - scripts/launch_multitask_xsum.py (53:62) - scripts/launch_xsum.py (41:50) duplicated block id: 198 size: 10 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (46:55) - preprocess/sm_inference_asum.py (687:696) duplicated block id: 199 size: 10 cleaned lines of code in 2 files: - fairseq/models/roberta/model.py (142:151) - fairseq/models/roberta/model_xlmr.py (28:37) duplicated block id: 200 size: 10 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (249:258) - preprocess/sm_inference_asum.py (687:696) duplicated block id: 201 size: 10 cleaned lines of code in 2 files: - fairseq/models/roberta/model.py (142:151) - fairseq/models/roberta/model_camembert.py (27:36) duplicated block id: 202 size: 10 cleaned lines of code in 2 files: - fairseq/modules/sparse_transformer_sentence_encoder.py (59:68) - fairseq/modules/transformer_sentence_encoder.py (139:148) duplicated block id: 203 size: 10 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (46:55) - preprocess/sm_inference_asum.py (557:566) duplicated block id: 204 size: 10 cleaned lines of code in 2 files: - fairseq/data/base_wrapper_dataset.py (33:46) - fairseq/data/transform_eos_lang_pair_dataset.py (75:88) duplicated block id: 205 size: 10 cleaned lines of code in 2 files: - fairseq_cli/preprocess.py (197:208) - fairseq_cli/preprocess.py (251:262) duplicated block id: 206 size: 10 cleaned lines of code in 2 files: - fairseq/modules/lightconv_layer/lightconv_cuda_kernel.cu (21:33) - fairseq/modules/lightconv_layer/lightconv_cuda_kernel.cu (81:93) duplicated block id: 207 size: 10 cleaned lines of code in 2 files: - scripts/launch_cnndm.py (64:73) - scripts/launch_sagemaker_unlikelihood_cnndm.py (61:70) duplicated block id: 208 size: 10 cleaned lines of code in 2 files: - fairseq/models/nat/levenshtein_utils.py (51:61) - fairseq/models/nat/levenshtein_utils.py (113:123) duplicated block id: 209 size: 10 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (249:258) - preprocess/sm_inference_asum.py (557:566) duplicated block id: 210 size: 10 cleaned lines of code in 2 files: - fairseq/modules/transformer_layer.py (108:117) - fairseq/modules/transformer_layer.py (304:313) duplicated block id: 211 size: 10 cleaned lines of code in 2 files: - fairseq/optim/adafactor.py (143:159) - fairseq/optim/adam.py (131:147) duplicated block id: 212 size: 10 cleaned lines of code in 2 files: - fairseq/models/nat/nat_crf_transformer.py (36:50) - fairseq/models/nat/nonautoregressive_transformer.py (77:91) duplicated block id: 213 size: 10 cleaned lines of code in 2 files: - fairseq/optim/nag.py (23:40) - fairseq/optim/sgd.py (22:39) duplicated block id: 214 size: 10 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (983:992) - preprocess/sm_inference_asum.py (1037:1046) duplicated block id: 215 size: 10 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (983:992) - preprocess/sm_inference_asum.py (1021:1030) duplicated block id: 216 size: 10 cleaned lines of code in 2 files: - fairseq/modules/dynamicconv_layer/dynamicconv_layer.py (114:126) - fairseq/modules/lightweight_convolution.py (234:246) duplicated block id: 217 size: 10 cleaned lines of code in 2 files: - fairseq/models/nat/nat_crf_transformer.py (87:97) - fairseq/models/nat/nonautoregressive_transformer.py (120:130) duplicated block id: 218 size: 10 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (983:992) - preprocess/sm_inference_asum.py (1127:1136) duplicated block id: 219 size: 10 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (46:55) - preprocess/sm_inference_asum.py (249:258) duplicated block id: 220 size: 10 cleaned lines of code in 2 files: - fairseq/models/bart/model.py (142:151) - fairseq/models/roberta/model.py (117:126) duplicated block id: 221 size: 10 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (983:992) - preprocess/sm_inference_asum.py (1091:1100) duplicated block id: 222 size: 10 cleaned lines of code in 2 files: - fairseq/tasks/translation_with_unlikelihood.py (18:27) - preprocess/sm_inference_asum.py (20:29) duplicated block id: 223 size: 10 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (1037:1046) - preprocess/sm_inference_asum.py (1127:1136) duplicated block id: 224 size: 10 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (1037:1046) - preprocess/sm_inference_asum.py (1091:1100) duplicated block id: 225 size: 9 cleaned lines of code in 2 files: - scripts/launch_multitask_xsum.py (53:61) - scripts/launch_sagemaker_unlikelihood_xsum.py (50:58) duplicated block id: 226 size: 9 cleaned lines of code in 2 files: - fairseq/tasks/multilingual_denoising.py (120:129) - fairseq/tasks/multilingual_masked_lm.py (143:152) duplicated block id: 227 size: 9 cleaned lines of code in 2 files: - fairseq/modules/lightconv_layer/lightconv_layer.py (114:124) - fairseq/modules/lightweight_convolution.py (236:246) duplicated block id: 228 size: 9 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (209:218) - preprocess/sm_inference_asum.py (808:817) duplicated block id: 229 size: 9 cleaned lines of code in 2 files: - scripts/launch_multitask_cnndm.py (53:61) - scripts/launch_sagemaker_unlikelihood_xsum.py (50:58) duplicated block id: 230 size: 9 cleaned lines of code in 2 files: - scripts/launch_multitask_newsroom.py (54:62) - scripts/launch_sagemaker_unlikelihood_cnndm.py (50:58) duplicated block id: 231 size: 9 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (402:410) - preprocess/sm_inference_asum.py (821:829) duplicated block id: 232 size: 9 cleaned lines of code in 2 files: - scripts/launch_newsroom.py (43:51) - scripts/launch_sagemaker_unlikelihood_cnndm.py (50:58) duplicated block id: 233 size: 9 cleaned lines of code in 2 files: - preprocess/data_prepro_clean.py (210:218) - preprocess/data_prepro_clean.py (347:355) duplicated block id: 234 size: 9 cleaned lines of code in 2 files: - fairseq/models/bart/model.py (128:136) - fairseq/models/roberta/model_camembert.py (27:35) duplicated block id: 235 size: 9 cleaned lines of code in 2 files: - fairseq/tasks/translation.py (291:302) - fairseq/tasks/translation_from_pretrained_bart.py (63:74) duplicated block id: 236 size: 9 cleaned lines of code in 2 files: - fairseq/tasks/translation_from_pretrained_bart.py (63:74) - fairseq/tasks/translation_lev.py (39:50) duplicated block id: 237 size: 9 cleaned lines of code in 2 files: - fairseq/modules/dynamic_convolution.py (112:121) - fairseq/modules/dynamicconv_layer/dynamicconv_layer.py (88:98) duplicated block id: 238 size: 9 cleaned lines of code in 2 files: - fairseq/models/nat/cmlm_transformer.py (133:142) - fairseq/models/nat/insertion_transformer.py (268:277) duplicated block id: 239 size: 9 cleaned lines of code in 2 files: - scripts/launch_cnndm.py (50:58) - scripts/launch_sagemaker_unlikelihood_cnndm.py (50:58) duplicated block id: 240 size: 9 cleaned lines of code in 2 files: - fairseq/modules/dynamic_convolution.py (221:231) - fairseq/modules/lightconv_layer/lightconv_layer.py (114:124) duplicated block id: 241 size: 9 cleaned lines of code in 2 files: - scripts/launch_multitask_newsroom.py (21:33) - scripts/launch_xsum.py (11:23) duplicated block id: 242 size: 9 cleaned lines of code in 2 files: - scripts/launch_multitask_cnndm.py (21:33) - scripts/launch_xsum.py (11:23) duplicated block id: 243 size: 9 cleaned lines of code in 2 files: - scripts/launch_cnndm.py (50:58) - scripts/launch_sagemaker_unlikelihood_xsum.py (50:58) duplicated block id: 244 size: 9 cleaned lines of code in 2 files: - scripts/launch_sagemaker_unlikelihood_xsum.py (62:70) - scripts/launch_xsum.py (57:65) duplicated block id: 245 size: 9 cleaned lines of code in 2 files: - scripts/launch_sagemaker_unlikelihood_cnndm.py (50:58) - scripts/launch_xsum.py (41:49) duplicated block id: 246 size: 9 cleaned lines of code in 2 files: - fairseq/modules/dynamic_convolution.py (58:67) - fairseq/modules/lightweight_convolution.py (115:124) duplicated block id: 247 size: 9 cleaned lines of code in 2 files: - scripts/launch_sagemaker_unlikelihood_xsum.py (50:58) - scripts/launch_xsum.py (41:49) duplicated block id: 248 size: 9 cleaned lines of code in 2 files: - scripts/launch_multitask_xsum.py (21:33) - scripts/launch_xsum.py (11:23) duplicated block id: 249 size: 9 cleaned lines of code in 2 files: - scripts/launch_cnndm.py (20:32) - scripts/launch_xsum.py (11:23) duplicated block id: 250 size: 9 cleaned lines of code in 2 files: - fairseq/tasks/cross_lingual_lm.py (84:94) - fairseq/tasks/legacy_masked_lm.py (61:71) duplicated block id: 251 size: 9 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (47:55) - preprocess/sm_inference_asum.py (821:829) duplicated block id: 252 size: 9 cleaned lines of code in 2 files: - fairseq/models/fconv.py (45:55) - fairseq/models/lightconv.py (50:61) duplicated block id: 253 size: 9 cleaned lines of code in 2 files: - fairseq/criterions/adaptive_loss.py (87:101) - fairseq/criterions/cross_entropy.py (59:73) duplicated block id: 254 size: 9 cleaned lines of code in 2 files: - fairseq/models/bart/model.py (128:136) - fairseq/models/roberta/model_xlmr.py (28:36) duplicated block id: 255 size: 9 cleaned lines of code in 2 files: - scripts/launch_multitask_xsum.py (53:61) - scripts/launch_sagemaker_unlikelihood_cnndm.py (50:58) duplicated block id: 256 size: 9 cleaned lines of code in 2 files: - scripts/launch_multitask_cnndm.py (53:61) - scripts/launch_sagemaker_unlikelihood_cnndm.py (50:58) duplicated block id: 257 size: 9 cleaned lines of code in 2 files: - fairseq/models/bart/hub_interface.py (90:98) - fairseq/models/roberta/hub_interface.py (67:75) duplicated block id: 258 size: 9 cleaned lines of code in 2 files: - fairseq/models/lightconv.py (392:404) - fairseq/models/transformer.py (753:765) duplicated block id: 259 size: 9 cleaned lines of code in 2 files: - scripts/launch_multitask_newsroom.py (54:62) - scripts/launch_sagemaker_unlikelihood_xsum.py (50:58) duplicated block id: 260 size: 9 cleaned lines of code in 2 files: - fairseq/modules/dynamicconv_layer/dynamicconv_cuda.cuh (1:11) - fairseq/modules/lightconv_layer/lightconv_cuda.cuh (1:11) duplicated block id: 261 size: 9 cleaned lines of code in 2 files: - fairseq/modules/dynamicconv_layer/dynamicconv_cuda.cuh (13:23) - fairseq/modules/lightconv_layer/lightconv_cuda.cuh (12:22) duplicated block id: 262 size: 9 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (558:566) - preprocess/sm_inference_asum.py (821:829) duplicated block id: 263 size: 9 cleaned lines of code in 2 files: - fairseq/modules/dynamic_convolution.py (141:149) - fairseq/modules/lightweight_convolution.py (178:186) duplicated block id: 264 size: 9 cleaned lines of code in 2 files: - fairseq/models/lightconv.py (398:411) - fairseq/models/nat/levenshtein_transformer.py (330:341) duplicated block id: 265 size: 9 cleaned lines of code in 2 files: - fairseq/modules/dynamicconv_layer/dynamicconv_layer.py (141:149) - fairseq/modules/lightweight_convolution.py (178:186) duplicated block id: 266 size: 9 cleaned lines of code in 2 files: - fairseq/modules/dynamicconv_layer/dynamicconv_layer.py (116:126) - fairseq/modules/lightconv_layer/lightconv_layer.py (114:124) duplicated block id: 267 size: 9 cleaned lines of code in 2 files: - fairseq/modules/lightconv_layer/lightconv_cuda.cuh (65:73) - fairseq/modules/lightconv_layer/lightconv_cuda_kernel.cu (264:272) duplicated block id: 268 size: 9 cleaned lines of code in 2 files: - fairseq/models/nat/cmlm_transformer.py (55:65) - fairseq/models/nat/nonautoregressive_transformer.py (97:106) duplicated block id: 269 size: 9 cleaned lines of code in 2 files: - fairseq_cli/generate.py (75:83) - fairseq_cli/interactive.py (104:112) duplicated block id: 270 size: 9 cleaned lines of code in 2 files: - fairseq/models/lightconv.py (677:687) - fairseq/models/transformer.py (912:922) duplicated block id: 271 size: 9 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (688:696) - preprocess/sm_inference_asum.py (821:829) duplicated block id: 272 size: 9 cleaned lines of code in 2 files: - fairseq/models/bart/model.py (128:136) - fairseq/models/roberta/model.py (142:150) duplicated block id: 273 size: 9 cleaned lines of code in 2 files: - preprocess/evaluate_hypo.py (172:181) - preprocess/evaluate_hypo.py (188:197) duplicated block id: 274 size: 9 cleaned lines of code in 2 files: - scripts/launch_newsroom.py (43:51) - scripts/launch_sagemaker_unlikelihood_xsum.py (50:58) duplicated block id: 275 size: 8 cleaned lines of code in 2 files: - fairseq_cli/generate.py (166:173) - fairseq_cli/interactive.py (177:184) duplicated block id: 276 size: 8 cleaned lines of code in 2 files: - fairseq/models/nat/levenshtein_transformer.py (458:465) - fairseq/models/transformer.py (993:1000) duplicated block id: 277 size: 8 cleaned lines of code in 2 files: - fairseq/models/nat/cmlm_transformer.py (94:102) - fairseq/models/nat/nat_crf_transformer.py (89:97) duplicated block id: 278 size: 8 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (462:469) - preprocess/sm_inference_asum.py (858:865) duplicated block id: 279 size: 8 cleaned lines of code in 2 files: - fairseq/models/lightconv.py (638:645) - fairseq/modules/transformer_layer.py (287:295) duplicated block id: 280 size: 8 cleaned lines of code in 2 files: - fairseq/models/nat/levenshtein_transformer.py (175:182) - fairseq/models/nat/nonautoregressive_ensembles.py (160:167) duplicated block id: 281 size: 8 cleaned lines of code in 2 files: - fairseq/tasks/fairseq_task.py (338:346) - fairseq/tasks/translation_lev.py (150:158) duplicated block id: 282 size: 8 cleaned lines of code in 2 files: - fairseq/models/lstm.py (169:176) - fairseq/models/lstm_lm.py (101:108) duplicated block id: 283 size: 8 cleaned lines of code in 2 files: - fairseq/models/lightconv_lm.py (135:142) - fairseq/models/transformer_lm.py (176:183) duplicated block id: 284 size: 8 cleaned lines of code in 2 files: - scripts/launch_multitask_xsum.py (118:125) - scripts/launch_newsroom.py (94:101) duplicated block id: 285 size: 8 cleaned lines of code in 2 files: - fairseq/models/lightconv.py (146:154) - fairseq/models/lightconv_lm.py (90:98) duplicated block id: 286 size: 8 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (169:176) - preprocess/sm_inference_asum.py (293:300) duplicated block id: 287 size: 8 cleaned lines of code in 2 files: - fairseq/modules/multihead_attention.py (209:216) - fairseq/modules/multihead_attention.py (295:302) duplicated block id: 288 size: 8 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (169:176) - preprocess/sm_inference_asum.py (354:361) duplicated block id: 289 size: 8 cleaned lines of code in 2 files: - fairseq/optim/lr_scheduler/fixed_schedule.py (13:23) - fairseq/optim/lr_scheduler/polynomial_decay_schedule.py (13:23) duplicated block id: 290 size: 8 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (129:136) - preprocess/sm_inference_asum.py (885:892) duplicated block id: 291 size: 8 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (129:136) - preprocess/sm_inference_asum.py (849:856) duplicated block id: 292 size: 8 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (169:176) - preprocess/sm_inference_asum.py (858:865) duplicated block id: 293 size: 8 cleaned lines of code in 2 files: - fairseq/tasks/masked_lm.py (89:96) - fairseq/tasks/multilingual_denoising.py (122:129) duplicated block id: 294 size: 8 cleaned lines of code in 2 files: - fairseq/models/nat/cmlm_transformer.py (133:141) - fairseq/models/nat/levenshtein_transformer.py (428:436) duplicated block id: 295 size: 8 cleaned lines of code in 2 files: - fairseq/modules/lightconv_layer/lightconv_cuda_kernel.cu (251:260) - fairseq/modules/lightconv_layer/lightconv_cuda_kernel.cu (366:375) duplicated block id: 296 size: 8 cleaned lines of code in 2 files: - scripts/launch_multitask_xsum.py (65:72) - scripts/launch_xsum.py (54:62) duplicated block id: 297 size: 8 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (226:233) - preprocess/sm_inference_asum.py (237:244) duplicated block id: 298 size: 8 cleaned lines of code in 2 files: - fairseq/models/nat/levenshtein_transformer.py (273:280) - fairseq/models/nat/nonautoregressive_transformer.py (192:199) duplicated block id: 299 size: 8 cleaned lines of code in 2 files: - fairseq/models/fconv_lm.py (17:26) - fairseq/models/lstm_lm.py (18:28) duplicated block id: 300 size: 8 cleaned lines of code in 2 files: - fairseq/optim/fused_adam.py (278:285) - fairseq/optim/fused_adam.py (292:299) duplicated block id: 301 size: 8 cleaned lines of code in 2 files: - fairseq/tasks/denoising.py (105:119) - fairseq/tasks/masked_lm.py (76:90) duplicated block id: 302 size: 8 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (139:146) - preprocess/sm_inference_asum.py (858:865) duplicated block id: 303 size: 8 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (262:269) - preprocess/sm_inference_asum.py (418:425) duplicated block id: 304 size: 8 cleaned lines of code in 2 files: - fairseq/models/nat/insertion_transformer.py (198:206) - fairseq/models/nat/nonautoregressive_transformer.py (122:130) duplicated block id: 305 size: 8 cleaned lines of code in 2 files: - fairseq/criterions/label_smoothed_cross_entropy.py (57:64) - fairseq/criterions/label_smoothed_cross_entropy_with_multitask.py (55:62) duplicated block id: 306 size: 8 cleaned lines of code in 2 files: - fairseq/models/nat/nonautoregressive_transformer.py (144:152) - fairseq/models/nat/nonautoregressive_transformer.py (174:182) duplicated block id: 307 size: 8 cleaned lines of code in 2 files: - fairseq/models/fconv.py (65:74) - fairseq/models/lstm.py (28:37) duplicated block id: 308 size: 8 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (293:300) - preprocess/sm_inference_asum.py (858:865) duplicated block id: 309 size: 8 cleaned lines of code in 2 files: - fairseq/models/nat/cmlm_transformer.py (94:102) - fairseq/models/nat/insertion_transformer.py (198:206) duplicated block id: 310 size: 8 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (389:397) - preprocess/sm_inference_asum.py (902:909) duplicated block id: 311 size: 8 cleaned lines of code in 2 files: - scripts/launch_multitask_cnndm.py (117:124) - scripts/launch_newsroom.py (94:101) duplicated block id: 312 size: 8 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (139:146) - preprocess/sm_inference_asum.py (293:300) duplicated block id: 313 size: 8 cleaned lines of code in 2 files: - fairseq/data/transform_eos_dataset.py (109:121) - fairseq/data/transform_eos_lang_pair_dataset.py (79:89) duplicated block id: 314 size: 8 cleaned lines of code in 2 files: - fairseq/criterions/cross_entropy.py (34:42) - fairseq/criterions/label_smoothed_cross_entropy.py (62:70) duplicated block id: 315 size: 8 cleaned lines of code in 2 files: - fairseq/models/nat/nonautoregressive_ensembles.py (45:54) - fairseq/sequence_generator.py (530:539) duplicated block id: 316 size: 8 cleaned lines of code in 2 files: - fairseq/modules/dynamicconv_layer/dynamicconv_layer.py (39:46) - fairseq/modules/lightconv_layer/lightconv_layer.py (38:45) duplicated block id: 317 size: 8 cleaned lines of code in 2 files: - fairseq/modules/dynamic_convolution.py (176:183) - fairseq/modules/lightweight_convolution.py (206:213) duplicated block id: 318 size: 8 cleaned lines of code in 2 files: - fairseq/models/nat/levenshtein_transformer.py (241:249) - fairseq/models/nat/nonautoregressive_ensembles.py (127:134) duplicated block id: 319 size: 8 cleaned lines of code in 2 files: - fairseq/models/fconv.py (45:53) - fairseq/models/transformer.py (55:64) duplicated block id: 320 size: 8 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (159:166) - preprocess/sm_inference_asum.py (849:856) duplicated block id: 321 size: 8 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (159:166) - preprocess/sm_inference_asum.py (885:892) duplicated block id: 322 size: 8 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (354:361) - preprocess/sm_inference_asum.py (858:865) duplicated block id: 323 size: 8 cleaned lines of code in 2 files: - fairseq/models/nat/insertion_transformer.py (133:144) - fairseq/models/nat/levenshtein_transformer.py (73:84) duplicated block id: 324 size: 8 cleaned lines of code in 2 files: - fairseq/models/nat/cmlm_transformer.py (94:102) - fairseq/models/nat/nonautoregressive_transformer.py (122:130) duplicated block id: 325 size: 8 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (139:146) - preprocess/sm_inference_asum.py (462:469) duplicated block id: 326 size: 8 cleaned lines of code in 2 files: - fairseq/modules/dynamicconv_layer/dynamicconv_layer.py (176:183) - fairseq/modules/lightweight_convolution.py (206:213) duplicated block id: 327 size: 8 cleaned lines of code in 2 files: - fairseq/models/lightconv.py (103:110) - fairseq/models/transformer.py (116:123) duplicated block id: 328 size: 8 cleaned lines of code in 2 files: - fairseq_cli/preprocess.py (184:191) - fairseq_cli/preprocess.py (239:246) duplicated block id: 329 size: 8 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (169:176) - preprocess/sm_inference_asum.py (462:469) duplicated block id: 330 size: 8 cleaned lines of code in 2 files: - scripts/launch_newsroom.py (94:101) - scripts/launch_xsum.py (95:102) duplicated block id: 331 size: 8 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (503:510) - preprocess/sm_inference_asum.py (858:865) duplicated block id: 332 size: 8 cleaned lines of code in 2 files: - fairseq/models/lightconv.py (167:175) - fairseq/models/multilingual_transformer.py (78:86) duplicated block id: 333 size: 8 cleaned lines of code in 2 files: - scripts/launch_cnndm.py (117:124) - scripts/launch_newsroom.py (94:101) duplicated block id: 334 size: 8 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (139:146) - preprocess/sm_inference_asum.py (503:510) duplicated block id: 335 size: 8 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (139:146) - preprocess/sm_inference_asum.py (354:361) duplicated block id: 336 size: 8 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (1004:1013) - preprocess/sm_inference_asum.py (1068:1077) duplicated block id: 337 size: 8 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (169:176) - preprocess/sm_inference_asum.py (503:510) duplicated block id: 338 size: 8 cleaned lines of code in 2 files: - fairseq/models/multilingual_transformer.py (78:86) - fairseq/models/transformer.py (187:195) duplicated block id: 339 size: 8 cleaned lines of code in 2 files: - fairseq/models/nat/cmlm_transformer.py (54:61) - fairseq/models/nat/iterative_nonautoregressive_transformer.py (137:144) duplicated block id: 340 size: 8 cleaned lines of code in 2 files: - fairseq/models/nat/insertion_transformer.py (198:206) - fairseq/models/nat/nat_crf_transformer.py (89:97) duplicated block id: 341 size: 8 cleaned lines of code in 2 files: - fairseq/models/lightconv.py (50:59) - fairseq/models/transformer.py (55:64) duplicated block id: 342 size: 7 cleaned lines of code in 2 files: - scripts/launch_multitask_cnndm.py (91:97) - scripts/launch_sagemaker_unlikelihood_xsum.py (86:92) duplicated block id: 343 size: 7 cleaned lines of code in 2 files: - fairseq/data/denoising_dataset.py (23:30) - fairseq/data/language_pair_dataset.py (22:29) duplicated block id: 344 size: 7 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (545:554) - preprocess/sm_inference_asum.py (674:683) duplicated block id: 345 size: 7 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (130:136) - preprocess/sm_inference_asum.py (281:287) duplicated block id: 346 size: 7 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (130:136) - preprocess/sm_inference_asum.py (342:348) duplicated block id: 347 size: 7 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (665:674) - preprocess/sm_inference_asum.py (799:808) duplicated block id: 348 size: 7 cleaned lines of code in 2 files: - fairseq/tasks/translation_lev.py (46:52) - fairseq/tasks/translation_with_unlikelihood.py (174:180) duplicated block id: 349 size: 7 cleaned lines of code in 2 files: - fairseq_cli/preprocess.py (153:159) - fairseq_cli/preprocess.py (265:271) duplicated block id: 350 size: 7 cleaned lines of code in 2 files: - scripts/launch_cnndm.py (89:96) - scripts/launch_sagemaker_unlikelihood_xsum.py (86:92) duplicated block id: 351 size: 7 cleaned lines of code in 2 files: - fairseq/models/transformer.py (66:73) - fairseq/models/transformer_lm.py (30:37) duplicated block id: 352 size: 7 cleaned lines of code in 2 files: - fairseq/models/lstm.py (128:134) - fairseq/models/lstm_lm.py (72:78) duplicated block id: 353 size: 7 cleaned lines of code in 2 files: - scripts/launch_multitask_newsroom.py (92:98) - scripts/launch_sagemaker_unlikelihood_xsum.py (86:92) duplicated block id: 354 size: 7 cleaned lines of code in 2 files: - fairseq/modules/lightconv_layer/lightconv_layer.py (93:102) - fairseq/modules/lightweight_convolution.py (195:203) duplicated block id: 355 size: 7 cleaned lines of code in 2 files: - preprocess/evaluate_hypo.py (500:506) - preprocess/evaluate_hypo.py (526:533) duplicated block id: 356 size: 7 cleaned lines of code in 2 files: - scripts/launch_multitask_newsroom.py (92:98) - scripts/launch_sagemaker_unlikelihood_cnndm.py (86:92) duplicated block id: 357 size: 7 cleaned lines of code in 2 files: - scripts/launch_sagemaker_unlikelihood_cnndm.py (86:92) - scripts/launch_xsum.py (81:87) duplicated block id: 358 size: 7 cleaned lines of code in 2 files: - scripts/launch_cnndm.py (13:24) - scripts/launch_sagemaker_unlikelihood_cnndm.py (13:24) duplicated block id: 359 size: 7 cleaned lines of code in 2 files: - fairseq/data/monolingual_dataset.py (185:193) - fairseq/data/subsample_dataset.py (60:68) duplicated block id: 360 size: 7 cleaned lines of code in 2 files: - fairseq_cli/preprocess.py (153:159) - fairseq_cli/preprocess.py (211:217) duplicated block id: 361 size: 7 cleaned lines of code in 2 files: - fairseq/options.py (17:23) - preprocess/sm_inference_asum.py (36:42) duplicated block id: 362 size: 7 cleaned lines of code in 2 files: - fairseq/models/transformer.py (405:411) - fairseq/models/transformer.py (666:672) duplicated block id: 363 size: 7 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (491:497) - preprocess/sm_inference_asum.py (850:856) duplicated block id: 364 size: 7 cleaned lines of code in 2 files: - fairseq/modules/dynamic_convolution.py (123:129) - fairseq/modules/dynamicconv_layer/dynamicconv_layer.py (128:134) duplicated block id: 365 size: 7 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (491:497) - preprocess/sm_inference_asum.py (886:892) duplicated block id: 366 size: 7 cleaned lines of code in 2 files: - scripts/launch_multitask_xsum.py (13:25) - scripts/launch_sagemaker_unlikelihood_cnndm.py (13:24) duplicated block id: 367 size: 7 cleaned lines of code in 2 files: - preprocess/multi_gpu_generate.py (90:96) - preprocess/sm_inference_asum.py (903:909) duplicated block id: 368 size: 7 cleaned lines of code in 2 files: - fairseq/tasks/sentence_prediction.py (74:86) - fairseq/tasks/sentence_ranking.py (61:73) duplicated block id: 369 size: 7 cleaned lines of code in 2 files: - scripts/launch_multitask_newsroom.py (13:25) - scripts/launch_sagemaker_unlikelihood_cnndm.py (13:24) duplicated block id: 370 size: 7 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (342:348) - preprocess/sm_inference_asum.py (850:856) duplicated block id: 371 size: 7 cleaned lines of code in 2 files: - preprocess/multi_gpu_generate.py (36:44) - preprocess/sm_inference_asum.py (825:833) duplicated block id: 372 size: 7 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (342:348) - preprocess/sm_inference_asum.py (886:892) duplicated block id: 373 size: 7 cleaned lines of code in 2 files: - fairseq/modules/dynamicconv_layer/dynamicconv_layer.py (142:148) - fairseq/modules/lightconv_layer/lightconv_layer.py (81:87) duplicated block id: 374 size: 7 cleaned lines of code in 2 files: - fairseq/models/lightconv.py (485:492) - fairseq/models/lightconv.py (573:580) duplicated block id: 375 size: 7 cleaned lines of code in 2 files: - fairseq/data/audio/raw_audio_dataset.py (108:118) - fairseq/data/subsample_dataset.py (60:68) duplicated block id: 376 size: 7 cleaned lines of code in 2 files: - fairseq/data/audio/raw_audio_dataset.py (108:118) - fairseq/data/monolingual_dataset.py (185:193) duplicated block id: 377 size: 7 cleaned lines of code in 2 files: - fairseq/modules/dynamic_convolution.py (208:216) - fairseq/modules/lightweight_convolution.py (222:229) duplicated block id: 378 size: 7 cleaned lines of code in 2 files: - scripts/launch_newsroom.py (80:86) - scripts/launch_sagemaker_unlikelihood_cnndm.py (86:92) duplicated block id: 379 size: 7 cleaned lines of code in 2 files: - fairseq/models/lightconv.py (355:361) - fairseq/models/transformer.py (651:657) duplicated block id: 380 size: 7 cleaned lines of code in 2 files: - scripts/launch_multitask_xsum.py (91:98) - scripts/launch_sagemaker_unlikelihood_xsum.py (86:92) duplicated block id: 381 size: 7 cleaned lines of code in 2 files: - fairseq/models/nat/cmlm_transformer.py (39:48) - fairseq/models/nat/nat_crf_transformer.py (40:50) duplicated block id: 382 size: 7 cleaned lines of code in 2 files: - fairseq/models/bart/model.py (125:132) - fairseq/models/fairseq_model.py (188:217) duplicated block id: 383 size: 7 cleaned lines of code in 2 files: - fairseq/modules/dynamic_convolution.py (142:148) - fairseq/modules/lightconv_layer/lightconv_layer.py (81:87) duplicated block id: 384 size: 7 cleaned lines of code in 2 files: - fairseq/data/base_wrapper_dataset.py (37:46) - fairseq/data/transform_eos_dataset.py (109:120) duplicated block id: 385 size: 7 cleaned lines of code in 2 files: - fairseq/tasks/sentence_prediction.py (12:18) - fairseq/tasks/sentence_ranking.py (12:18) duplicated block id: 386 size: 7 cleaned lines of code in 2 files: - scripts/launch_multitask_newsroom.py (70:76) - scripts/launch_multitask_xsum.py (69:75) duplicated block id: 387 size: 7 cleaned lines of code in 2 files: - scripts/launch_cnndm.py (89:96) - scripts/launch_sagemaker_unlikelihood_cnndm.py (86:92) duplicated block id: 388 size: 7 cleaned lines of code in 2 files: - scripts/launch_multitask_cnndm.py (13:25) - scripts/launch_sagemaker_unlikelihood_xsum.py (13:24) duplicated block id: 389 size: 7 cleaned lines of code in 2 files: - fairseq/models/lstm.py (161:167) - fairseq/models/lstm_lm.py (93:99) duplicated block id: 390 size: 7 cleaned lines of code in 2 files: - fairseq/models/nat/nat_crf_transformer.py (65:72) - fairseq/models/nat/nonautoregressive_transformer.py (98:105) duplicated block id: 391 size: 7 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (160:166) - preprocess/sm_inference_asum.py (281:287) duplicated block id: 392 size: 7 cleaned lines of code in 2 files: - fairseq/modules/dynamicconv_layer/dynamicconv_layer.py (203:211) - fairseq/modules/lightweight_convolution.py (222:229) duplicated block id: 393 size: 7 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (160:166) - preprocess/sm_inference_asum.py (342:348) duplicated block id: 394 size: 7 cleaned lines of code in 2 files: - fairseq/data/encoders/utils.py (18:24) - fairseq/tasks/multilingual_masked_lm.py (95:102) duplicated block id: 395 size: 7 cleaned lines of code in 2 files: - scripts/launch_multitask_cnndm.py (13:25) - scripts/launch_sagemaker_unlikelihood_cnndm.py (13:24) duplicated block id: 396 size: 7 cleaned lines of code in 2 files: - fairseq/models/nat/iterative_nonautoregressive_transformer.py (138:144) - fairseq/models/nat/nonautoregressive_transformer.py (97:103) duplicated block id: 397 size: 7 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (281:287) - preprocess/sm_inference_asum.py (886:892) duplicated block id: 398 size: 7 cleaned lines of code in 2 files: - fairseq/models/fairseq_model.py (46:53) - fairseq/models/transformer.py (280:287) duplicated block id: 399 size: 7 cleaned lines of code in 2 files: - fairseq/modules/transformer_layer.py (41:48) - fairseq/modules/transformer_layer.py (152:159) duplicated block id: 400 size: 7 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (120:128) - preprocess/sm_inference_asum.py (437:445) duplicated block id: 401 size: 7 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (281:287) - preprocess/sm_inference_asum.py (850:856) duplicated block id: 402 size: 7 cleaned lines of code in 2 files: - fairseq/models/nat/insertion_transformer.py (168:176) - fairseq/models/nat/levenshtein_transformer.py (137:145) duplicated block id: 403 size: 7 cleaned lines of code in 2 files: - fairseq/criterions/label_smoothed_cross_entropy.py (12:18) - fairseq/criterions/label_smoothed_cross_entropy_with_unlikelihood.py (13:19) duplicated block id: 404 size: 7 cleaned lines of code in 2 files: - fairseq/criterions/label_smoothed_cross_entropy_with_multitask.py (109:115) - fairseq/criterions/label_smoothed_cross_entropy_with_multitask.py (141:147) duplicated block id: 405 size: 7 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (160:166) - preprocess/sm_inference_asum.py (450:456) duplicated block id: 406 size: 7 cleaned lines of code in 2 files: - scripts/launch_multitask_newsroom.py (13:25) - scripts/launch_sagemaker_unlikelihood_xsum.py (13:24) duplicated block id: 407 size: 7 cleaned lines of code in 2 files: - fairseq/tasks/multilingual_denoising.py (159:166) - fairseq/tasks/multilingual_masked_lm.py (201:209) duplicated block id: 408 size: 7 cleaned lines of code in 2 files: - scripts/launch_multitask_xsum.py (91:98) - scripts/launch_sagemaker_unlikelihood_cnndm.py (86:92) duplicated block id: 409 size: 7 cleaned lines of code in 2 files: - fairseq/models/lightconv_lm.py (52:58) - fairseq/models/transformer_lm.py (77:83) duplicated block id: 410 size: 7 cleaned lines of code in 2 files: - preprocess/multi_gpu_generate.py (36:44) - preprocess/sm_inference_asum.py (254:262) duplicated block id: 411 size: 7 cleaned lines of code in 2 files: - scripts/launch_newsroom.py (80:86) - scripts/launch_sagemaker_unlikelihood_xsum.py (86:92) duplicated block id: 412 size: 7 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (130:136) - preprocess/sm_inference_asum.py (491:497) duplicated block id: 413 size: 7 cleaned lines of code in 2 files: - fairseq/modules/dynamic_crf_layer.py (111:117) - fairseq/modules/dynamic_crf_layer.py (138:144) duplicated block id: 414 size: 7 cleaned lines of code in 2 files: - scripts/launch_sagemaker_unlikelihood_xsum.py (86:92) - scripts/launch_xsum.py (81:87) duplicated block id: 415 size: 7 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (160:166) - preprocess/sm_inference_asum.py (491:497) duplicated block id: 416 size: 7 cleaned lines of code in 2 files: - fairseq/models/nat/cmlm_transformer.py (56:63) - fairseq/models/nat/nat_crf_transformer.py (65:72) duplicated block id: 417 size: 7 cleaned lines of code in 2 files: - scripts/launch_multitask_cnndm.py (91:97) - scripts/launch_sagemaker_unlikelihood_cnndm.py (86:92) duplicated block id: 418 size: 7 cleaned lines of code in 2 files: - fairseq/modules/lightconv_layer/lightconv_layer.py (81:87) - fairseq/modules/lightweight_convolution.py (179:185) duplicated block id: 419 size: 7 cleaned lines of code in 2 files: - fairseq/modules/dynamicconv_layer/dynamicconv_cuda.cpp (40:47) - fairseq/modules/lightconv_layer/lightconv_cuda.cpp (39:46) duplicated block id: 420 size: 7 cleaned lines of code in 2 files: - fairseq/models/nat/cmlm_transformer.py (39:48) - fairseq/models/nat/nonautoregressive_transformer.py (81:91) duplicated block id: 421 size: 7 cleaned lines of code in 2 files: - fairseq/data/encoders/gpt2_bpe.py (42:49) - fairseq/data/encoders/hf_byte_bpe.py (39:46) duplicated block id: 422 size: 7 cleaned lines of code in 2 files: - scripts/launch_cnndm.py (13:24) - scripts/launch_sagemaker_unlikelihood_xsum.py (13:24) duplicated block id: 423 size: 7 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (130:136) - preprocess/sm_inference_asum.py (450:456) duplicated block id: 424 size: 7 cleaned lines of code in 2 files: - preprocess/multi_gpu_generate.py (90:96) - preprocess/sm_inference_asum.py (390:397) duplicated block id: 425 size: 7 cleaned lines of code in 2 files: - fairseq/models/lstm.py (93:99) - fairseq/models/lstm_lm.py (64:70) duplicated block id: 426 size: 7 cleaned lines of code in 2 files: - fairseq/models/model_utils.py (84:90) - fairseq/models/nat/levenshtein_utils.py (278:284) duplicated block id: 427 size: 7 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (450:456) - preprocess/sm_inference_asum.py (886:892) duplicated block id: 428 size: 7 cleaned lines of code in 2 files: - fairseq/models/fconv.py (152:159) - fairseq/models/fconv_self_att.py (175:182) duplicated block id: 429 size: 7 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (450:456) - preprocess/sm_inference_asum.py (850:856) duplicated block id: 430 size: 7 cleaned lines of code in 2 files: - fairseq/modules/lightconv_layer/lightconv_cuda_kernel.cu (221:228) - fairseq/modules/lightconv_layer/lightconv_cuda_kernel.cu (337:344) duplicated block id: 431 size: 7 cleaned lines of code in 2 files: - fairseq/tasks/sentence_prediction.py (110:119) - fairseq/tasks/sentence_ranking.py (86:96) duplicated block id: 432 size: 7 cleaned lines of code in 2 files: - fairseq/tasks/masked_lm.py (14:20) - fairseq/tasks/multilingual_masked_lm.py (17:23) duplicated block id: 433 size: 7 cleaned lines of code in 2 files: - fairseq/optim/lr_scheduler/fixed_schedule.py (45:54) - fairseq/optim/lr_scheduler/polynomial_decay_schedule.py (48:57) duplicated block id: 434 size: 7 cleaned lines of code in 2 files: - scripts/launch_multitask_xsum.py (13:25) - scripts/launch_sagemaker_unlikelihood_xsum.py (13:24) duplicated block id: 435 size: 7 cleaned lines of code in 2 files: - fairseq_cli/preprocess.py (127:133) - fairseq_cli/preprocess.py (185:191) duplicated block id: 436 size: 7 cleaned lines of code in 2 files: - fairseq_cli/preprocess.py (127:133) - fairseq_cli/preprocess.py (240:246) duplicated block id: 437 size: 6 cleaned lines of code in 2 files: - fairseq/tasks/masked_lm.py (188:194) - fairseq/tasks/sentence_ranking.py (190:196) duplicated block id: 438 size: 6 cleaned lines of code in 2 files: - scripts/launch_multitask_newsroom.py (120:125) - scripts/launch_sagemaker_unlikelihood_xsum.py (114:119) duplicated block id: 439 size: 6 cleaned lines of code in 2 files: - fairseq/modules/lightconv_layer/lightconv_cuda_kernel.cu (230:237) - fairseq/modules/lightconv_layer/lightconv_cuda_kernel.cu (346:352) duplicated block id: 440 size: 6 cleaned lines of code in 2 files: - fairseq/tasks/masked_lm.py (188:194) - fairseq/tasks/multilingual_masked_lm.py (311:317) duplicated block id: 441 size: 6 cleaned lines of code in 2 files: - fairseq/tasks/denoising.py (163:171) - fairseq/tasks/sentence_ranking.py (190:196) duplicated block id: 442 size: 6 cleaned lines of code in 2 files: - scripts/launch_multitask_cnndm.py (119:124) - scripts/launch_sagemaker_unlikelihood_cnndm.py (114:119) duplicated block id: 443 size: 6 cleaned lines of code in 2 files: - fairseq/modules/lightconv_layer/lightconv_cuda.cuh (26:31) - fairseq/modules/lightconv_layer/lightconv_cuda_kernel.cu (13:18) duplicated block id: 444 size: 6 cleaned lines of code in 2 files: - fairseq/data/audio/raw_audio_dataset.py (20:25) - fairseq/data/audio/raw_audio_dataset.py (125:130) duplicated block id: 445 size: 6 cleaned lines of code in 2 files: - fairseq/modules/lightconv_layer/cuda_function_gen.py (98:103) - fairseq/modules/lightconv_layer/cuda_function_gen.py (107:112) duplicated block id: 446 size: 6 cleaned lines of code in 2 files: - preprocess/create_entity_classification_labels.py (53:59) - preprocess/create_entity_classification_labels.py (92:98) duplicated block id: 447 size: 6 cleaned lines of code in 2 files: - fairseq/tasks/multilingual_denoising.py (80:90) - fairseq/tasks/multilingual_masked_lm.py (109:119) duplicated block id: 448 size: 6 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (213:218) - preprocess/sm_inference_asum.py (392:397) duplicated block id: 449 size: 6 cleaned lines of code in 2 files: - fairseq/benchmark/dummy_masked_lm.py (81:87) - fairseq/tasks/multilingual_masked_lm.py (311:317) duplicated block id: 450 size: 6 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (678:683) - preprocess/sm_inference_asum.py (812:817) duplicated block id: 451 size: 6 cleaned lines of code in 2 files: - fairseq/tasks/sentence_prediction.py (90:95) - fairseq/tasks/sentence_ranking.py (78:83) duplicated block id: 452 size: 6 cleaned lines of code in 2 files: - fairseq/modules/dynamicconv_layer/dynamicconv_cuda.cuh (1:6) - fairseq/modules/lightconv_layer/lightconv_cuda_kernel.cu (1:6) duplicated block id: 453 size: 6 cleaned lines of code in 2 files: - scripts/launch_cnndm.py (119:124) - scripts/launch_sagemaker_unlikelihood_xsum.py (114:119) duplicated block id: 454 size: 6 cleaned lines of code in 2 files: - fairseq/optim/adam.py (123:129) - fairseq/optim/fused_adam.py (213:219) duplicated block id: 455 size: 6 cleaned lines of code in 2 files: - fairseq/models/lightconv.py (96:101) - fairseq/models/transformer.py (109:114) duplicated block id: 456 size: 6 cleaned lines of code in 2 files: - fairseq/optim/fp16_optimizer.py (221:226) - fairseq/optim/fp16_optimizer.py (408:413) duplicated block id: 457 size: 6 cleaned lines of code in 2 files: - fairseq_cli/eval_lm.py (122:127) - fairseq_cli/train.py (263:268) duplicated block id: 458 size: 6 cleaned lines of code in 2 files: - fairseq/optim/adam.py (123:129) - fairseq/optim/fused_adam.py (89:95) duplicated block id: 459 size: 6 cleaned lines of code in 2 files: - fairseq/modules/transformer_sentence_encoder.py (77:82) - fairseq/modules/transformer_sentence_encoder_layer.py (25:30) duplicated block id: 460 size: 6 cleaned lines of code in 2 files: - fairseq/models/nat/iterative_nonautoregressive_transformer.py (139:144) - fairseq/models/nat/nat_crf_transformer.py (65:70) duplicated block id: 461 size: 6 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (392:397) - preprocess/sm_inference_asum.py (549:554) duplicated block id: 462 size: 6 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (549:554) - preprocess/sm_inference_asum.py (904:909) duplicated block id: 463 size: 6 cleaned lines of code in 2 files: - preprocess/multi_gpu_generate.py (91:96) - preprocess/sm_inference_asum.py (213:218) duplicated block id: 464 size: 6 cleaned lines of code in 2 files: - fairseq/models/lightconv.py (112:117) - fairseq/models/transformer.py (123:128) duplicated block id: 465 size: 6 cleaned lines of code in 2 files: - fairseq/models/bart/hub_interface.py (30:36) - fairseq/models/roberta/hub_interface.py (21:27) duplicated block id: 466 size: 6 cleaned lines of code in 2 files: - fairseq/optim/adagrad.py (21:34) - fairseq/optim/sgd.py (23:36) duplicated block id: 467 size: 6 cleaned lines of code in 2 files: - preprocess/evaluate_hypo.py (433:438) - preprocess/evaluate_hypo.py (441:446) duplicated block id: 468 size: 6 cleaned lines of code in 2 files: - fairseq/tasks/denoising.py (163:171) - fairseq/tasks/sentence_prediction.py (223:229) duplicated block id: 469 size: 6 cleaned lines of code in 2 files: - fairseq/models/nat/insertion_transformer.py (213:219) - fairseq/models/nat/nonautoregressive_transformer.py (194:199) duplicated block id: 470 size: 6 cleaned lines of code in 2 files: - preprocess/evaluate_hypo.py (634:640) - preprocess/evaluate_hypo.py (643:648) duplicated block id: 471 size: 6 cleaned lines of code in 2 files: - fairseq_cli/preprocess.py (141:146) - fairseq_cli/preprocess.py (252:257) duplicated block id: 472 size: 6 cleaned lines of code in 2 files: - scripts/launch_multitask_cnndm.py (119:124) - scripts/launch_sagemaker_unlikelihood_xsum.py (114:119) duplicated block id: 473 size: 6 cleaned lines of code in 2 files: - fairseq/modules/dynamicconv_layer/dynamicconv_cuda_kernel.cu (1:6) - fairseq/modules/lightconv_layer/lightconv_cuda_kernel.cu (1:6) duplicated block id: 474 size: 6 cleaned lines of code in 2 files: - fairseq/benchmark/dummy_masked_lm.py (81:87) - fairseq/tasks/sentence_ranking.py (190:196) duplicated block id: 475 size: 6 cleaned lines of code in 2 files: - fairseq_cli/preprocess.py (141:146) - fairseq_cli/preprocess.py (198:203) duplicated block id: 476 size: 6 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (213:218) - preprocess/sm_inference_asum.py (549:554) duplicated block id: 477 size: 6 cleaned lines of code in 2 files: - fairseq/models/fconv.py (366:372) - fairseq/models/fconv_self_att.py (321:327) duplicated block id: 478 size: 6 cleaned lines of code in 2 files: - fairseq/models/masked_lm.py (61:66) - fairseq/models/transformer.py (110:115) duplicated block id: 479 size: 6 cleaned lines of code in 2 files: - fairseq/benchmark/dummy_masked_lm.py (81:87) - fairseq/tasks/masked_lm.py (188:194) duplicated block id: 480 size: 6 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (549:554) - preprocess/sm_inference_asum.py (812:817) duplicated block id: 481 size: 6 cleaned lines of code in 2 files: - fairseq/data/legacy/masked_lm_dictionary.py (14:19) - fairseq/data/legacy/masked_lm_dictionary.py (36:41) duplicated block id: 482 size: 6 cleaned lines of code in 2 files: - scripts/launch_multitask_newsroom.py (68:73) - scripts/launch_sagemaker_unlikelihood_cnndm.py (62:67) duplicated block id: 483 size: 6 cleaned lines of code in 2 files: - fairseq/modules/cuda_utils.cu (1:6) - fairseq/modules/dynamicconv_layer/dynamicconv_cuda.cuh (1:6) duplicated block id: 484 size: 6 cleaned lines of code in 2 files: - fairseq/modules/lightweight_convolution.py (62:68) - fairseq/modules/lightweight_convolution.py (132:138) duplicated block id: 485 size: 6 cleaned lines of code in 2 files: - fairseq_cli/preprocess.py (24:29) - fairseq_cli/train.py (31:36) duplicated block id: 486 size: 6 cleaned lines of code in 2 files: - scripts/launch_newsroom.py (96:101) - scripts/launch_sagemaker_unlikelihood_cnndm.py (114:119) duplicated block id: 487 size: 6 cleaned lines of code in 2 files: - fairseq/models/nat/insertion_transformer.py (213:219) - fairseq/models/nat/levenshtein_transformer.py (275:280) duplicated block id: 488 size: 6 cleaned lines of code in 2 files: - fairseq/tasks/multilingual_translation.py (213:219) - fairseq/tasks/translation_with_unlikelihood.py (187:193) duplicated block id: 489 size: 6 cleaned lines of code in 2 files: - fairseq_cli/generate.py (102:107) - fairseq_cli/validate.py (81:86) duplicated block id: 490 size: 6 cleaned lines of code in 2 files: - fairseq/optim/fused_lamb.py (29:42) - fairseq/optim/nag.py (24:37) duplicated block id: 491 size: 6 cleaned lines of code in 2 files: - fairseq_cli/interactive.py (23:28) - fairseq_cli/train.py (31:36) duplicated block id: 492 size: 6 cleaned lines of code in 2 files: - fairseq_cli/train.py (263:268) - fairseq_cli/validate.py (81:86) duplicated block id: 493 size: 6 cleaned lines of code in 2 files: - fairseq_cli/preprocess.py (24:29) - fairseq_cli/validate.py (16:21) duplicated block id: 494 size: 6 cleaned lines of code in 2 files: - fairseq/models/lightconv_lm.py (122:127) - fairseq/models/transformer_lm.py (148:153) duplicated block id: 495 size: 6 cleaned lines of code in 2 files: - scripts/launch_cnndm.py (65:70) - scripts/launch_multitask_newsroom.py (68:73) duplicated block id: 496 size: 6 cleaned lines of code in 2 files: - fairseq/tasks/multilingual_masked_lm.py (311:317) - fairseq/tasks/sentence_prediction.py (223:229) duplicated block id: 497 size: 6 cleaned lines of code in 2 files: - fairseq/modules/sparse_transformer_sentence_encoder.py (22:27) - fairseq/modules/sparse_transformer_sentence_encoder_layer.py (17:22) duplicated block id: 498 size: 6 cleaned lines of code in 2 files: - fairseq/models/transformer.py (866:871) - fairseq/modules/transformer_layer.py (331:336) duplicated block id: 499 size: 6 cleaned lines of code in 2 files: - fairseq/tasks/cross_lingual_lm.py (41:47) - fairseq/tasks/legacy_masked_lm.py (38:44) duplicated block id: 500 size: 6 cleaned lines of code in 2 files: - fairseq/models/lightconv.py (14:19) - fairseq/models/lstm.py (12:17) duplicated block id: 501 size: 6 cleaned lines of code in 2 files: - scripts/launch_multitask_xsum.py (120:125) - scripts/launch_sagemaker_unlikelihood_cnndm.py (114:119) duplicated block id: 502 size: 6 cleaned lines of code in 2 files: - fairseq/tasks/multilingual_masked_lm.py (145:150) - fairseq/tasks/sentence_ranking.py (95:100) duplicated block id: 503 size: 6 cleaned lines of code in 2 files: - fairseq/optim/fused_lamb.py (29:42) - fairseq/optim/sgd.py (23:36) duplicated block id: 504 size: 6 cleaned lines of code in 2 files: - preprocess/multi_gpu_generate.py (49:54) - preprocess/sm_inference_asum.py (834:839) duplicated block id: 505 size: 6 cleaned lines of code in 2 files: - fairseq/data/denoising_dataset.py (32:38) - fairseq/data/language_pair_dataset.py (53:59) duplicated block id: 506 size: 6 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (392:397) - preprocess/sm_inference_asum.py (812:817) duplicated block id: 507 size: 6 cleaned lines of code in 2 files: - fairseq/benchmark/dummy_masked_lm.py (79:86) - fairseq/tasks/language_modeling.py (255:264) duplicated block id: 508 size: 6 cleaned lines of code in 2 files: - fairseq/models/transformer.py (919:924) - fairseq/modules/transformer_layer.py (344:349) duplicated block id: 509 size: 6 cleaned lines of code in 2 files: - fairseq/modules/cuda_utils.cu (1:6) - fairseq/modules/lightconv_layer/lightconv_cuda.cuh (1:6) duplicated block id: 510 size: 6 cleaned lines of code in 2 files: - fairseq/criterions/adaptive_loss.py (74:80) - fairseq/criterions/legacy_masked_lm.py (123:129) duplicated block id: 511 size: 6 cleaned lines of code in 2 files: - fairseq/models/fconv.py (178:183) - fairseq/models/fconv.py (390:395) duplicated block id: 512 size: 6 cleaned lines of code in 2 files: - fairseq/tasks/denoising.py (107:118) - fairseq/tasks/language_modeling.py (151:163) duplicated block id: 513 size: 6 cleaned lines of code in 2 files: - scripts/launch_cnndm.py (65:70) - scripts/launch_multitask_cnndm.py (67:72) duplicated block id: 514 size: 6 cleaned lines of code in 2 files: - fairseq_cli/generate.py (102:107) - fairseq_cli/train.py (263:268) duplicated block id: 515 size: 6 cleaned lines of code in 2 files: - fairseq/models/nat/insertion_transformer.py (133:139) - fairseq/models/nat/nonautoregressive_transformer.py (73:79) duplicated block id: 516 size: 6 cleaned lines of code in 2 files: - fairseq_cli/eval_lm.py (122:127) - fairseq_cli/validate.py (81:86) duplicated block id: 517 size: 6 cleaned lines of code in 2 files: - fairseq/criterions/sentence_prediction.py (67:72) - fairseq/criterions/sentence_ranking.py (85:90) duplicated block id: 518 size: 6 cleaned lines of code in 2 files: - fairseq/tasks/language_modeling.py (70:75) - fairseq/tasks/multilingual_masked_lm.py (46:51) duplicated block id: 519 size: 6 cleaned lines of code in 2 files: - fairseq/tasks/masked_lm.py (89:94) - fairseq/tasks/sentence_ranking.py (95:100) duplicated block id: 520 size: 6 cleaned lines of code in 2 files: - fairseq/tasks/multilingual_translation.py (81:86) - fairseq/tasks/translation_with_unlikelihood.py (89:94) duplicated block id: 521 size: 6 cleaned lines of code in 2 files: - fairseq/benchmark/dummy_masked_lm.py (81:87) - fairseq/tasks/denoising.py (163:171) duplicated block id: 522 size: 6 cleaned lines of code in 2 files: - fairseq/benchmark/dummy_lm.py (70:76) - fairseq/tasks/sentence_ranking.py (190:196) duplicated block id: 523 size: 6 cleaned lines of code in 2 files: - fairseq/file_io.py (39:44) - fairseq/file_io.py (47:52) duplicated block id: 524 size: 6 cleaned lines of code in 2 files: - fairseq/modules/lightconv_layer/lightconv_cuda_kernel.cu (131:137) - fairseq/modules/lightconv_layer/lightconv_cuda_kernel.cu (259:265) duplicated block id: 525 size: 6 cleaned lines of code in 2 files: - fairseq/models/fconv.py (65:72) - fairseq/models/fconv_self_att.py (69:76) duplicated block id: 526 size: 6 cleaned lines of code in 2 files: - fairseq/benchmark/dummy_lm.py (70:76) - fairseq/tasks/masked_lm.py (188:194) duplicated block id: 527 size: 6 cleaned lines of code in 2 files: - fairseq/modules/dynamicconv_layer/dynamicconv_cuda.cuh (1:6) - fairseq/modules/dynamicconv_layer/dynamicconv_cuda_kernel.cu (1:6) duplicated block id: 528 size: 6 cleaned lines of code in 2 files: - fairseq/models/transformer.py (124:129) - fairseq/models/transformer_lm.py (67:72) duplicated block id: 529 size: 6 cleaned lines of code in 2 files: - fairseq/modules/dynamicconv_layer/dynamicconv_cuda_kernel.cu (1:6) - fairseq/modules/lightconv_layer/lightconv_cuda.cuh (1:6) duplicated block id: 530 size: 6 cleaned lines of code in 2 files: - fairseq/modules/dynamic_convolution.py (124:129) - fairseq/modules/lightweight_convolution.py (170:175) duplicated block id: 531 size: 6 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (812:817) - preprocess/sm_inference_asum.py (904:909) duplicated block id: 532 size: 6 cleaned lines of code in 2 files: - fairseq/criterions/sentence_prediction.py (53:58) - fairseq/criterions/sentence_ranking.py (67:72) duplicated block id: 533 size: 6 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (392:397) - preprocess/sm_inference_asum.py (678:683) duplicated block id: 534 size: 6 cleaned lines of code in 2 files: - fairseq/models/lightconv.py (113:118) - fairseq/models/lightconv_lm.py (44:49) duplicated block id: 535 size: 6 cleaned lines of code in 2 files: - fairseq/models/fconv.py (14:19) - fairseq/models/lightconv.py (14:19) duplicated block id: 536 size: 6 cleaned lines of code in 2 files: - scripts/launch_multitask_cnndm.py (67:72) - scripts/launch_sagemaker_unlikelihood_cnndm.py (62:67) duplicated block id: 537 size: 6 cleaned lines of code in 2 files: - scripts/launch_cnndm.py (65:70) - scripts/launch_newsroom.py (57:62) duplicated block id: 538 size: 6 cleaned lines of code in 2 files: - preprocess/ner.py (69:74) - preprocess/ner.py (133:138) duplicated block id: 539 size: 6 cleaned lines of code in 2 files: - fairseq/modules/sparse_transformer_sentence_encoder_layer.py (17:22) - fairseq/modules/transformer_sentence_encoder.py (77:82) duplicated block id: 540 size: 6 cleaned lines of code in 2 files: - fairseq_cli/interactive.py (23:28) - fairseq_cli/preprocess.py (24:29) duplicated block id: 541 size: 6 cleaned lines of code in 2 files: - fairseq/clib/libnat_cuda/edit_dist.cu (86:91) - fairseq/clib/libnat_cuda/edit_dist.cu (165:170) duplicated block id: 542 size: 6 cleaned lines of code in 2 files: - fairseq/models/lightconv.py (77:84) - fairseq/models/lstm.py (25:33) duplicated block id: 543 size: 6 cleaned lines of code in 2 files: - fairseq/models/lightconv.py (684:689) - fairseq/modules/dynamic_convolution.py (34:39) duplicated block id: 544 size: 6 cleaned lines of code in 2 files: - fairseq/models/fconv.py (647:652) - fairseq/models/fconv.py (663:668) duplicated block id: 545 size: 6 cleaned lines of code in 2 files: - fairseq/tasks/sentence_prediction.py (223:229) - fairseq/tasks/sentence_ranking.py (190:196) duplicated block id: 546 size: 6 cleaned lines of code in 2 files: - fairseq/modules/dynamicconv_layer/dynamicconv_layer.py (30:37) - fairseq/modules/lightconv_layer/lightconv_layer.py (29:36) duplicated block id: 547 size: 6 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (213:218) - preprocess/sm_inference_asum.py (904:909) duplicated block id: 548 size: 6 cleaned lines of code in 2 files: - fairseq/tasks/language_modeling.py (151:163) - fairseq/tasks/masked_lm.py (78:89) duplicated block id: 549 size: 6 cleaned lines of code in 2 files: - fairseq/tasks/multilingual_denoising.py (122:127) - fairseq/tasks/sentence_ranking.py (95:100) duplicated block id: 550 size: 6 cleaned lines of code in 2 files: - preprocess/multi_gpu_generate.py (91:96) - preprocess/sm_inference_asum.py (678:683) duplicated block id: 551 size: 6 cleaned lines of code in 2 files: - fairseq/optim/adam.py (151:161) - fairseq/optim/fused_adam.py (157:167) duplicated block id: 552 size: 6 cleaned lines of code in 2 files: - fairseq/models/bart/model.py (227:233) - fairseq/models/roberta/model.py (234:240) duplicated block id: 553 size: 6 cleaned lines of code in 2 files: - scripts/launch_sagemaker_unlikelihood_cnndm.py (114:119) - scripts/launch_xsum.py (97:102) duplicated block id: 554 size: 6 cleaned lines of code in 2 files: - fairseq/modules/adaptive_input.py (24:30) - fairseq/modules/adaptive_softmax.py (58:64) duplicated block id: 555 size: 6 cleaned lines of code in 2 files: - fairseq/optim/fused_adam.py (89:95) - fairseq/optim/nag.py (48:54) duplicated block id: 556 size: 6 cleaned lines of code in 2 files: - scripts/launch_multitask_newsroom.py (120:125) - scripts/launch_sagemaker_unlikelihood_cnndm.py (114:119) duplicated block id: 557 size: 6 cleaned lines of code in 2 files: - fairseq/models/fconv.py (14:19) - fairseq/models/lstm.py (12:17) duplicated block id: 558 size: 6 cleaned lines of code in 2 files: - fairseq/benchmark/dummy_masked_lm.py (81:87) - fairseq/tasks/sentence_prediction.py (223:229) duplicated block id: 559 size: 6 cleaned lines of code in 2 files: - fairseq/modules/lightconv_layer/lightconv_cuda.cuh (48:53) - fairseq/modules/lightconv_layer/lightconv_cuda_kernel.cu (267:272) duplicated block id: 560 size: 6 cleaned lines of code in 2 files: - fairseq/modules/gumbel_vector_quantizer.py (129:134) - fairseq/modules/kmeans_vector_quantizer.py (107:112) duplicated block id: 561 size: 6 cleaned lines of code in 2 files: - fairseq/data/legacy/masked_lm_dataset.py (312:318) - fairseq/data/monolingual_dataset.py (190:196) duplicated block id: 562 size: 6 cleaned lines of code in 2 files: - fairseq/modules/lightconv_layer/lightconv_cuda_kernel.cu (139:144) - fairseq/modules/lightconv_layer/lightconv_cuda_kernel.cu (267:272) duplicated block id: 563 size: 6 cleaned lines of code in 2 files: - fairseq/modules/lightconv_layer/lightconv_cuda.cuh (68:73) - fairseq/modules/lightconv_layer/lightconv_cuda_kernel.cu (139:144) duplicated block id: 564 size: 6 cleaned lines of code in 2 files: - fairseq/benchmark/dummy_lm.py (70:76) - fairseq/tasks/multilingual_masked_lm.py (311:317) duplicated block id: 565 size: 6 cleaned lines of code in 2 files: - fairseq/modules/dynamicconv_layer/dynamicconv_layer.py (129:134) - fairseq/modules/lightweight_convolution.py (170:175) duplicated block id: 566 size: 6 cleaned lines of code in 2 files: - fairseq/optim/adagrad.py (21:34) - fairseq/optim/nag.py (24:37) duplicated block id: 567 size: 6 cleaned lines of code in 2 files: - fairseq/optim/fused_adam.py (213:219) - fairseq/optim/nag.py (48:54) duplicated block id: 568 size: 6 cleaned lines of code in 2 files: - fairseq/tasks/masked_lm.py (188:194) - fairseq/tasks/sentence_prediction.py (223:229) duplicated block id: 569 size: 6 cleaned lines of code in 2 files: - scripts/launch_cnndm.py (119:124) - scripts/launch_sagemaker_unlikelihood_cnndm.py (114:119) duplicated block id: 570 size: 6 cleaned lines of code in 2 files: - fairseq/benchmark/dummy_lm.py (68:75) - fairseq/tasks/language_modeling.py (255:264) duplicated block id: 571 size: 6 cleaned lines of code in 2 files: - preprocess/multi_gpu_generate.py (91:96) - preprocess/sm_inference_asum.py (812:817) duplicated block id: 572 size: 6 cleaned lines of code in 2 files: - fairseq/optim/adamax.py (87:93) - fairseq/optim/fused_adam.py (89:95) duplicated block id: 573 size: 6 cleaned lines of code in 2 files: - fairseq/optim/adagrad.py (21:34) - fairseq/optim/fused_lamb.py (29:42) duplicated block id: 574 size: 6 cleaned lines of code in 2 files: - fairseq/models/lightconv.py (441:448) - fairseq/models/transformer.py (845:852) duplicated block id: 575 size: 6 cleaned lines of code in 2 files: - fairseq/modules/lightconv_layer/lightconv_cuda.cuh (1:6) - fairseq/modules/lightconv_layer/lightconv_cuda_kernel.cu (1:6) duplicated block id: 576 size: 6 cleaned lines of code in 2 files: - fairseq/data/denoising_dataset.py (357:370) - fairseq/data/monolingual_dataset.py (175:188) duplicated block id: 577 size: 6 cleaned lines of code in 2 files: - fairseq/models/bart/model.py (153:158) - fairseq/models/roberta/model.py (128:133) duplicated block id: 578 size: 6 cleaned lines of code in 2 files: - scripts/launch_sagemaker_unlikelihood_xsum.py (114:119) - scripts/launch_xsum.py (97:102) duplicated block id: 579 size: 6 cleaned lines of code in 2 files: - fairseq/clib/libnat_cuda/edit_dist.cu (309:314) - fairseq/clib/libnat_cuda/edit_dist.cu (321:326) duplicated block id: 580 size: 6 cleaned lines of code in 2 files: - scripts/launch_newsroom.py (96:101) - scripts/launch_sagemaker_unlikelihood_xsum.py (114:119) duplicated block id: 581 size: 6 cleaned lines of code in 2 files: - fairseq/benchmark/dummy_lm.py (70:76) - fairseq/tasks/denoising.py (163:171) duplicated block id: 582 size: 6 cleaned lines of code in 2 files: - fairseq/optim/adamax.py (87:93) - fairseq/optim/fused_adam.py (213:219) duplicated block id: 583 size: 6 cleaned lines of code in 2 files: - fairseq/logging/meters.py (92:97) - fairseq/logging/meters.py (145:150) duplicated block id: 584 size: 6 cleaned lines of code in 2 files: - fairseq/sequence_generator.py (576:581) - fairseq/sequence_generator.py (691:696) duplicated block id: 585 size: 6 cleaned lines of code in 2 files: - fairseq/models/masked_lm.py (143:148) - fairseq/models/roberta/model.py (272:277) duplicated block id: 586 size: 6 cleaned lines of code in 2 files: - preprocess/evaluate_hypo.py (710:715) - preprocess/evaluate_hypo.py (826:831) duplicated block id: 587 size: 6 cleaned lines of code in 2 files: - fairseq/models/fconv.py (81:86) - fairseq/models/fconv_self_att.py (81:86) duplicated block id: 588 size: 6 cleaned lines of code in 2 files: - fairseq/tasks/multilingual_masked_lm.py (311:317) - fairseq/tasks/sentence_ranking.py (190:196) duplicated block id: 589 size: 6 cleaned lines of code in 2 files: - scripts/launch_multitask_xsum.py (120:125) - scripts/launch_sagemaker_unlikelihood_xsum.py (114:119) duplicated block id: 590 size: 6 cleaned lines of code in 2 files: - fairseq/modules/lightconv_layer/lightconv_cuda.cuh (48:53) - fairseq/modules/lightconv_layer/lightconv_cuda.cuh (68:73) duplicated block id: 591 size: 6 cleaned lines of code in 2 files: - fairseq_cli/train.py (31:36) - fairseq_cli/validate.py (16:21) duplicated block id: 592 size: 6 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (651:656) - preprocess/sm_inference_asum.py (785:790) duplicated block id: 593 size: 6 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (213:218) - preprocess/sm_inference_asum.py (678:683) duplicated block id: 594 size: 6 cleaned lines of code in 2 files: - fairseq/models/bart/model.py (235:240) - fairseq/models/roberta/model.py (242:247) duplicated block id: 595 size: 6 cleaned lines of code in 2 files: - fairseq/tasks/cross_lingual_lm.py (130:135) - fairseq/tasks/legacy_masked_lm.py (127:132) duplicated block id: 596 size: 6 cleaned lines of code in 2 files: - scripts/launch_newsroom.py (57:62) - scripts/launch_sagemaker_unlikelihood_cnndm.py (62:67) duplicated block id: 597 size: 6 cleaned lines of code in 2 files: - fairseq/data/indexed_dataset.py (165:172) - fairseq/data/indexed_dataset.py (274:281) duplicated block id: 598 size: 6 cleaned lines of code in 2 files: - fairseq/trainer.py (286:293) - fairseq/trainer.py (430:435) duplicated block id: 599 size: 6 cleaned lines of code in 2 files: - fairseq/models/fconv_self_att.py (202:207) - fairseq/models/fconv_self_att.py (340:345) duplicated block id: 600 size: 6 cleaned lines of code in 2 files: - fairseq/tasks/denoising.py (163:171) - fairseq/tasks/masked_lm.py (188:194) duplicated block id: 601 size: 6 cleaned lines of code in 2 files: - fairseq/modules/sparse_transformer_sentence_encoder.py (41:47) - fairseq/modules/sparse_transformer_sentence_encoder_layer.py (26:32) duplicated block id: 602 size: 6 cleaned lines of code in 2 files: - fairseq/tasks/denoising.py (163:171) - fairseq/tasks/multilingual_masked_lm.py (311:317) duplicated block id: 603 size: 6 cleaned lines of code in 2 files: - fairseq/optim/fused_adam.py (89:95) - fairseq/optim/fused_adam.py (213:219) duplicated block id: 604 size: 6 cleaned lines of code in 2 files: - fairseq/modules/cuda_utils.cu (1:6) - fairseq/modules/lightconv_layer/lightconv_cuda_kernel.cu (1:6) duplicated block id: 605 size: 6 cleaned lines of code in 2 files: - fairseq/models/fconv.py (259:269) - fairseq/models/fconv_self_att.py (252:262) duplicated block id: 606 size: 6 cleaned lines of code in 2 files: - fairseq_cli/generate.py (67:72) - fairseq_cli/interactive.py (92:97) duplicated block id: 607 size: 6 cleaned lines of code in 2 files: - fairseq/benchmark/dummy_lm.py (70:76) - fairseq/tasks/sentence_prediction.py (223:229) duplicated block id: 608 size: 6 cleaned lines of code in 2 files: - scripts/launch_multitask_xsum.py (67:72) - scripts/launch_sagemaker_unlikelihood_xsum.py (62:67) duplicated block id: 609 size: 6 cleaned lines of code in 2 files: - fairseq/modules/sparse_transformer_sentence_encoder.py (22:27) - fairseq/modules/transformer_sentence_encoder_layer.py (25:30) duplicated block id: 610 size: 6 cleaned lines of code in 2 files: - fairseq/models/nat/levenshtein_transformer.py (73:79) - fairseq/models/nat/nonautoregressive_transformer.py (73:79) duplicated block id: 611 size: 6 cleaned lines of code in 2 files: - fairseq/modules/cuda_utils.cu (1:6) - fairseq/modules/dynamicconv_layer/dynamicconv_cuda_kernel.cu (1:6) duplicated block id: 612 size: 6 cleaned lines of code in 2 files: - fairseq/tasks/multilingual_translation.py (81:86) - fairseq/tasks/translation.py (220:225) duplicated block id: 613 size: 6 cleaned lines of code in 2 files: - fairseq/models/fconv_self_att.py (69:76) - fairseq/models/lstm.py (28:35) duplicated block id: 614 size: 6 cleaned lines of code in 2 files: - fairseq/models/fconv_lm.py (25:30) - fairseq/models/fconv_self_att.py (79:84) duplicated block id: 615 size: 6 cleaned lines of code in 2 files: - fairseq/tasks/language_modeling.py (70:75) - fairseq/tasks/masked_lm.py (42:47) duplicated block id: 616 size: 6 cleaned lines of code in 2 files: - preprocess/multi_gpu_generate.py (91:96) - preprocess/sm_inference_asum.py (549:554) duplicated block id: 617 size: 6 cleaned lines of code in 2 files: - fairseq/models/transformer_from_pretrained_xlm.py (124:129) - fairseq/models/transformer_from_pretrained_xlm.py (142:148) duplicated block id: 618 size: 6 cleaned lines of code in 2 files: - fairseq_cli/interactive.py (23:28) - fairseq_cli/validate.py (16:21) duplicated block id: 619 size: 6 cleaned lines of code in 2 files: - preprocess/sm_inference_asum.py (678:683) - preprocess/sm_inference_asum.py (904:909)