duplicated block id: 1 size: 17 cleaned lines of code in 2 files: - src/hyperpod_nemo_adapter/collections/model/nlp/custom_models/modeling_deepseek.py (694:713) - src/hyperpod_nemo_adapter/collections/model/nlp/custom_models/modeling_deepseek.py (812:834) duplicated block id: 2 size: 16 cleaned lines of code in 2 files: - src/hyperpod_nemo_adapter/collections/model/sagemaker_base_model.py (414:429) - src/hyperpod_nemo_adapter/collections/model/sagemaker_base_model.py (506:521) duplicated block id: 3 size: 14 cleaned lines of code in 2 files: - src/hyperpod_nemo_adapter/collections/data/dummy_data_module.py (29:52) - src/hyperpod_nemo_adapter/collections/data/dummy_dpo_data_module.py (29:52) duplicated block id: 4 size: 12 cleaned lines of code in 2 files: - src/hyperpod_nemo_adapter/collections/parts/fsdp_strategy.py (186:198) - src/hyperpod_nemo_adapter/collections/parts/fsdp_strategy.py (228:239) duplicated block id: 5 size: 12 cleaned lines of code in 2 files: - src/hyperpod_nemo_adapter/collections/model/nlp/custom_models/modeling_deepseek.py (163:175) - src/hyperpod_nemo_adapter/collections/model/nlp/custom_models/modeling_deepseek.py (190:202) duplicated block id: 6 size: 12 cleaned lines of code in 2 files: - src/hyperpod_nemo_adapter/collections/model/nlp/custom_models/modeling_deepseek.py (1389:1400) - src/hyperpod_nemo_adapter/collections/model/nlp/custom_models/modeling_deepseek.py (1570:1581) duplicated block id: 7 size: 11 cleaned lines of code in 2 files: - src/hyperpod_nemo_adapter/collections/data/dummy_dpo_data_module.py (52:63) - src/hyperpod_nemo_adapter/collections/data/hf_dpo_data_module.py (49:60) duplicated block id: 8 size: 10 cleaned lines of code in 2 files: - src/hyperpod_nemo_adapter/utils/callbacks/local_ckpt_io.py (118:127) - src/hyperpod_nemo_adapter/utils/callbacks/sharded_ckpt_io.py (36:45) duplicated block id: 9 size: 10 cleaned lines of code in 2 files: - src/hyperpod_nemo_adapter/collections/model/nlp/custom_models/modeling_deepseek.py (684:693) - src/hyperpod_nemo_adapter/collections/model/nlp/custom_models/modeling_deepseek.py (795:805) duplicated block id: 10 size: 10 cleaned lines of code in 2 files: - src/hyperpod_nemo_adapter/collections/model/nlp/sagemaker_mistral_model.py (41:50) - src/hyperpod_nemo_adapter/collections/model/nlp/sagemaker_qwen_model.py (41:50) duplicated block id: 11 size: 9 cleaned lines of code in 2 files: - src/hyperpod_nemo_adapter/collections/model/nlp/custom_models/modeling_deepseek.py (721:732) - src/hyperpod_nemo_adapter/collections/model/nlp/custom_models/modeling_deepseek.py (838:849) duplicated block id: 12 size: 9 cleaned lines of code in 2 files: - src/hyperpod_nemo_adapter/collections/model/nlp/custom_models/modeling_deepseek.py (1328:1342) - src/hyperpod_nemo_adapter/patches/patch_llama_flash_attn_cp.py (399:413) duplicated block id: 13 size: 9 cleaned lines of code in 2 files: - src/hyperpod_nemo_adapter/utils/callbacks/peft_ckpt_io.py (181:189) - src/hyperpod_nemo_adapter/utils/callbacks/sharded_ckpt_io.py (61:69) duplicated block id: 14 size: 9 cleaned lines of code in 2 files: - src/hyperpod_nemo_adapter/collections/model/nlp/custom_models/modeling_deepseek.py (1430:1438) - src/hyperpod_nemo_adapter/collections/model/nlp/custom_models/modeling_deepseek.py (1593:1601) duplicated block id: 15 size: 8 cleaned lines of code in 2 files: - src/hyperpod_nemo_adapter/collections/model/nlp/custom_models/modeling_deepseek.py (1246:1253) - src/hyperpod_nemo_adapter/collections/model/nlp/custom_models/modeling_deepseek.py (1569:1576) duplicated block id: 16 size: 8 cleaned lines of code in 2 files: - src/hyperpod_nemo_adapter/collections/model/nlp/sagemaker_mistral_model.py (41:48) - src/hyperpod_nemo_adapter/collections/model/nlp/sagemaker_mixtral_model.py (41:48) duplicated block id: 17 size: 8 cleaned lines of code in 2 files: - src/hyperpod_nemo_adapter/utils/callbacks/ckpt_io.py (55:62) - src/hyperpod_nemo_adapter/utils/callbacks/peft_ckpt_io.py (47:54) duplicated block id: 18 size: 8 cleaned lines of code in 2 files: - src/hyperpod_nemo_adapter/collections/model/nlp/custom_models/modeling_deepseek.py (1347:1354) - src/hyperpod_nemo_adapter/patches/patch_llama_flash_attn_cp.py (419:426) duplicated block id: 19 size: 8 cleaned lines of code in 2 files: - src/hyperpod_nemo_adapter/collections/model/nlp/custom_models/modeling_deepseek.py (882:890) - src/hyperpod_nemo_adapter/patches/patch_llama_flash_attn_cp.py (221:229) duplicated block id: 20 size: 8 cleaned lines of code in 2 files: - src/hyperpod_nemo_adapter/collections/model/nlp/sagemaker_mixtral_model.py (41:48) - src/hyperpod_nemo_adapter/collections/model/nlp/sagemaker_qwen_model.py (41:48) duplicated block id: 21 size: 8 cleaned lines of code in 2 files: - src/hyperpod_nemo_adapter/utils/callbacks/local_ckpt_io.py (118:125) - src/hyperpod_nemo_adapter/utils/callbacks/peft_ckpt_io.py (163:170) duplicated block id: 22 size: 8 cleaned lines of code in 2 files: - src/hyperpod_nemo_adapter/utils/callbacks/peft_ckpt_io.py (163:170) - src/hyperpod_nemo_adapter/utils/callbacks/sharded_ckpt_io.py (36:43) duplicated block id: 23 size: 7 cleaned lines of code in 2 files: - src/hyperpod_nemo_adapter/collections/model/sagemaker_base_model.py (462:468) - src/hyperpod_nemo_adapter/collections/model/sagemaker_base_model.py (478:484) duplicated block id: 24 size: 7 cleaned lines of code in 2 files: - src/hyperpod_nemo_adapter/collections/data/datasets/dummy_dataset.py (23:30) - src/hyperpod_nemo_adapter/collections/data/datasets/dummy_dpo_dataset.py (26:33) duplicated block id: 25 size: 7 cleaned lines of code in 2 files: - src/hyperpod_nemo_adapter/collections/model/nlp/custom_models/modeling_deepseek.py (190:196) - src/hyperpod_nemo_adapter/collections/model/nlp/custom_models/modeling_deepseek.py (249:255) duplicated block id: 26 size: 7 cleaned lines of code in 2 files: - src/hyperpod_nemo_adapter/collections/model/nlp/custom_models/modeling_deepseek.py (1258:1265) - src/hyperpod_nemo_adapter/patches/patch_llama_flash_attn_cp.py (312:318) duplicated block id: 27 size: 7 cleaned lines of code in 2 files: - src/hyperpod_nemo_adapter/collections/model/nlp/custom_models/modeling_deepseek.py (792:798) - src/hyperpod_nemo_adapter/patches/patch_llama_flash_attn_cp.py (145:151) duplicated block id: 28 size: 7 cleaned lines of code in 2 files: - src/hyperpod_nemo_adapter/collections/model/nlp/custom_models/modeling_deepseek.py (1247:1253) - src/hyperpod_nemo_adapter/collections/model/nlp/custom_models/modeling_deepseek.py (1389:1395) duplicated block id: 29 size: 7 cleaned lines of code in 2 files: - src/hyperpod_nemo_adapter/utils/callbacks/peft_ckpt_io.py (44:51) - src/hyperpod_nemo_adapter/utils/callbacks/peft_ckpt_io.py (160:167) duplicated block id: 30 size: 7 cleaned lines of code in 2 files: - src/hyperpod_nemo_adapter/utils/callbacks/peft_ckpt_io.py (191:197) - src/hyperpod_nemo_adapter/utils/callbacks/sharded_ckpt_io.py (70:76) duplicated block id: 31 size: 7 cleaned lines of code in 2 files: - src/hyperpod_nemo_adapter/collections/model/nlp/sagemaker_llama_model.py (103:109) - src/hyperpod_nemo_adapter/collections/model/nlp/sagemaker_llama_model.py (111:117) duplicated block id: 32 size: 7 cleaned lines of code in 2 files: - src/hyperpod_nemo_adapter/collections/model/nlp/custom_models/modeling_deepseek.py (163:169) - src/hyperpod_nemo_adapter/collections/model/nlp/custom_models/modeling_deepseek.py (249:255) duplicated block id: 33 size: 7 cleaned lines of code in 2 files: - src/hyperpod_nemo_adapter/utils/callbacks/ckpt_io.py (82:89) - src/hyperpod_nemo_adapter/utils/callbacks/sharded_ckpt_io.py (59:66) duplicated block id: 34 size: 6 cleaned lines of code in 2 files: - src/hyperpod_nemo_adapter/utils/train_utils.py (60:65) - src/hyperpod_nemo_adapter/utils/train_utils.py (101:106) duplicated block id: 35 size: 6 cleaned lines of code in 2 files: - src/hyperpod_nemo_adapter/utils/callbacks/peft_ckpt_io.py (73:78) - src/hyperpod_nemo_adapter/utils/callbacks/sharded_ckpt_io.py (61:66) duplicated block id: 36 size: 6 cleaned lines of code in 2 files: - src/hyperpod_nemo_adapter/utils/callbacks/ckpt_io.py (84:89) - src/hyperpod_nemo_adapter/utils/callbacks/peft_ckpt_io.py (181:186) duplicated block id: 37 size: 6 cleaned lines of code in 2 files: - src/hyperpod_nemo_adapter/collections/model/nlp/sagemaker_llama_model.py (112:117) - src/hyperpod_nemo_adapter/collections/model/sagemaker_base_model.py (384:389) duplicated block id: 38 size: 6 cleaned lines of code in 2 files: - src/hyperpod_nemo_adapter/collections/data/hf_data_module.py (52:58) - src/hyperpod_nemo_adapter/collections/data/hf_data_module.py (81:87) duplicated block id: 39 size: 6 cleaned lines of code in 2 files: - src/hyperpod_nemo_adapter/collections/model/nlp/sagemaker_mistral_model.py (25:34) - src/hyperpod_nemo_adapter/collections/model/nlp/sagemaker_mixtral_model.py (25:34) duplicated block id: 40 size: 6 cleaned lines of code in 2 files: - src/hyperpod_nemo_adapter/collections/data/hf_data_module.py (81:87) - src/hyperpod_nemo_adapter/collections/data/hf_dpo_data_module.py (38:44) duplicated block id: 41 size: 6 cleaned lines of code in 2 files: - src/hyperpod_nemo_adapter/utils/callbacks/checkpoint.py (67:72) - src/hyperpod_nemo_adapter/utils/callbacks/checkpoint.py (274:279) duplicated block id: 42 size: 6 cleaned lines of code in 2 files: - src/hyperpod_nemo_adapter/collections/model/nlp/sagemaker_deepseek_model.py (71:76) - src/hyperpod_nemo_adapter/collections/model/nlp/sagemaker_deepseek_model.py (80:85) duplicated block id: 43 size: 6 cleaned lines of code in 2 files: - src/hyperpod_nemo_adapter/utils/callbacks/ckpt_io.py (84:89) - src/hyperpod_nemo_adapter/utils/callbacks/peft_ckpt_io.py (73:78) duplicated block id: 44 size: 6 cleaned lines of code in 2 files: - src/hyperpod_nemo_adapter/collections/model/nlp/sagemaker_llama_model.py (104:109) - src/hyperpod_nemo_adapter/collections/model/sagemaker_base_model.py (384:389) duplicated block id: 45 size: 6 cleaned lines of code in 2 files: - src/hyperpod_nemo_adapter/collections/model/nlp/sagemaker_mixtral_model.py (25:34) - src/hyperpod_nemo_adapter/collections/model/nlp/sagemaker_qwen_model.py (25:34) duplicated block id: 46 size: 6 cleaned lines of code in 2 files: - src/hyperpod_nemo_adapter/collections/model/nlp/sagemaker_mistral_model.py (25:34) - src/hyperpod_nemo_adapter/collections/model/nlp/sagemaker_qwen_model.py (25:34) duplicated block id: 47 size: 6 cleaned lines of code in 2 files: - src/hyperpod_nemo_adapter/collections/data/hf_data_module.py (52:58) - src/hyperpod_nemo_adapter/collections/data/hf_dpo_data_module.py (38:44) duplicated block id: 48 size: 6 cleaned lines of code in 2 files: - src/hyperpod_nemo_adapter/utils/callbacks/peft_ckpt_io.py (73:78) - src/hyperpod_nemo_adapter/utils/callbacks/peft_ckpt_io.py (181:186) duplicated block id: 49 size: 6 cleaned lines of code in 2 files: - src/hyperpod_nemo_adapter/collections/parts/fsdp_strategy.py (494:500) - src/hyperpod_nemo_adapter/utils/callbacks/ckpt_io.py (79:85)