aws / sagemaker-hyperpod-training-adapter-for-nemo

Duplication

Places in code with 6 or more lines that are exactly the same.

Longest Duplicates

The list of 49 longest duplicates.

See data for all 49 duplicates...

Size	#	Folders	Files	Lines	Code
17	x 2	src/hyperpod_nemo_adapte...model/nlp/custom_models src/hyperpod_nemo_adapte...model/nlp/custom_models	modeling_deepseek.py modeling_deepseek.py	694:713 (1%) 812:834 (1%)	view
16	x 2	src/hyperpod_nemo_adapter/collections/model src/hyperpod_nemo_adapter/collections/model	sagemaker_base_model.py sagemaker_base_model.py	414:429 (3%) 506:521 (3%)	view
14	x 2	src/hyperpod_nemo_adapter/collections/data src/hyperpod_nemo_adapter/collections/data	dummy_data_module.py dummy_dpo_data_module.py	29:52 (63%) 29:52 (48%)	view
12	x 2	src/hyperpod_nemo_adapter/collections/parts src/hyperpod_nemo_adapter/collections/parts	fsdp_strategy.py fsdp_strategy.py	186:198 (3%) 228:239 (3%)	view
12	x 2	src/hyperpod_nemo_adapte...model/nlp/custom_models src/hyperpod_nemo_adapte...model/nlp/custom_models	modeling_deepseek.py modeling_deepseek.py	163:175 (1%) 190:202 (1%)	view
12	x 2	src/hyperpod_nemo_adapte...model/nlp/custom_models src/hyperpod_nemo_adapte...model/nlp/custom_models	modeling_deepseek.py modeling_deepseek.py	1389:1400 (1%) 1570:1581 (1%)	view
11	x 2	src/hyperpod_nemo_adapter/collections/data src/hyperpod_nemo_adapter/collections/data	dummy_dpo_data_module.py hf_dpo_data_module.py	52:63 (37%) 49:60 (37%)	view
10	x 2	src/hyperpod_nemo_adapter/utils/callbacks src/hyperpod_nemo_adapter/utils/callbacks	local_ckpt_io.py sharded_ckpt_io.py	118:127 (5%) 36:45 (17%)	view
10	x 2	src/hyperpod_nemo_adapte...model/nlp/custom_models src/hyperpod_nemo_adapte...model/nlp/custom_models	modeling_deepseek.py modeling_deepseek.py	684:693 (<1%) 795:805 (<1%)	view
10	x 2	src/hyperpod_nemo_adapter/collections/model/nlp src/hyperpod_nemo_adapter/collections/model/nlp	sagemaker_mistral_model.py sagemaker_qwen_model.py	41:50 (45%) 41:50 (45%)	view
9	x 2	src/hyperpod_nemo_adapte...model/nlp/custom_models src/hyperpod_nemo_adapte...model/nlp/custom_models	modeling_deepseek.py modeling_deepseek.py	721:732 (<1%) 838:849 (<1%)	view
9	x 2	src/hyperpod_nemo_adapte...model/nlp/custom_models src/hyperpod_nemo_adapter/patches	modeling_deepseek.py patch_llama_flash_attn_cp.py	1328:1342 (<1%) 399:413 (3%)	view
9	x 2	src/hyperpod_nemo_adapter/utils/callbacks src/hyperpod_nemo_adapter/utils/callbacks	peft_ckpt_io.py sharded_ckpt_io.py	181:189 (6%) 61:69 (16%)	view
9	x 2	src/hyperpod_nemo_adapte...model/nlp/custom_models src/hyperpod_nemo_adapte...model/nlp/custom_models	modeling_deepseek.py modeling_deepseek.py	1430:1438 (<1%) 1593:1601 (<1%)	view
8	x 2	src/hyperpod_nemo_adapte...model/nlp/custom_models src/hyperpod_nemo_adapte...model/nlp/custom_models	modeling_deepseek.py modeling_deepseek.py	1246:1253 (<1%) 1569:1576 (<1%)	view
8	x 2	src/hyperpod_nemo_adapter/collections/model/nlp src/hyperpod_nemo_adapter/collections/model/nlp	sagemaker_mistral_model.py sagemaker_mixtral_model.py	41:48 (36%) 41:48 (33%)	view
8	x 2	src/hyperpod_nemo_adapter/utils/callbacks src/hyperpod_nemo_adapter/utils/callbacks	ckpt_io.py peft_ckpt_io.py	55:62 (9%) 47:54 (5%)	view
8	x 2	src/hyperpod_nemo_adapte...model/nlp/custom_models src/hyperpod_nemo_adapter/patches	modeling_deepseek.py patch_llama_flash_attn_cp.py	1347:1354 (<1%) 419:426 (2%)	view
8	x 2	src/hyperpod_nemo_adapte...model/nlp/custom_models src/hyperpod_nemo_adapter/patches	modeling_deepseek.py patch_llama_flash_attn_cp.py	882:890 (<1%) 221:229 (2%)	view
8	x 2	src/hyperpod_nemo_adapter/collections/model/nlp src/hyperpod_nemo_adapter/collections/model/nlp	sagemaker_mixtral_model.py sagemaker_qwen_model.py	41:48 (33%) 41:48 (36%)	view
8	x 2	src/hyperpod_nemo_adapter/utils/callbacks src/hyperpod_nemo_adapter/utils/callbacks	local_ckpt_io.py peft_ckpt_io.py	118:125 (4%) 163:170 (5%)	view
8	x 2	src/hyperpod_nemo_adapter/utils/callbacks src/hyperpod_nemo_adapter/utils/callbacks	peft_ckpt_io.py sharded_ckpt_io.py	163:170 (5%) 36:43 (14%)	view
7	x 2	src/hyperpod_nemo_adapter/collections/model src/hyperpod_nemo_adapter/collections/model	sagemaker_base_model.py sagemaker_base_model.py	462:468 (1%) 478:484 (1%)	view
7	x 2	src/hyperpod_nemo_adapte...llections/data/datasets src/hyperpod_nemo_adapte...llections/data/datasets	dummy_dataset.py dummy_dpo_dataset.py	23:30 (50%) 26:33 (26%)	view
7	x 2	src/hyperpod_nemo_adapte...model/nlp/custom_models src/hyperpod_nemo_adapte...model/nlp/custom_models	modeling_deepseek.py modeling_deepseek.py	190:196 (<1%) 249:255 (<1%)	view
7	x 2	src/hyperpod_nemo_adapte...model/nlp/custom_models src/hyperpod_nemo_adapter/patches	modeling_deepseek.py patch_llama_flash_attn_cp.py	1258:1265 (<1%) 312:318 (2%)	view
7	x 2	src/hyperpod_nemo_adapte...model/nlp/custom_models src/hyperpod_nemo_adapter/patches	modeling_deepseek.py patch_llama_flash_attn_cp.py	792:798 (<1%) 145:151 (2%)	view
7	x 2	src/hyperpod_nemo_adapte...model/nlp/custom_models src/hyperpod_nemo_adapte...model/nlp/custom_models	modeling_deepseek.py modeling_deepseek.py	1247:1253 (<1%) 1389:1395 (<1%)	view
7	x 2	src/hyperpod_nemo_adapter/utils/callbacks src/hyperpod_nemo_adapter/utils/callbacks	peft_ckpt_io.py peft_ckpt_io.py	44:51 (5%) 160:167 (5%)	view
7	x 2	src/hyperpod_nemo_adapter/utils/callbacks src/hyperpod_nemo_adapter/utils/callbacks	peft_ckpt_io.py sharded_ckpt_io.py	191:197 (5%) 70:76 (12%)	view
7	x 2	src/hyperpod_nemo_adapter/collections/model/nlp src/hyperpod_nemo_adapter/collections/model/nlp	sagemaker_llama_model.py sagemaker_llama_model.py	103:109 (10%) 111:117 (10%)	view
7	x 2	src/hyperpod_nemo_adapte...model/nlp/custom_models src/hyperpod_nemo_adapte...model/nlp/custom_models	modeling_deepseek.py modeling_deepseek.py	163:169 (<1%) 249:255 (<1%)	view
7	x 2	src/hyperpod_nemo_adapter/utils/callbacks src/hyperpod_nemo_adapter/utils/callbacks	ckpt_io.py sharded_ckpt_io.py	82:89 (8%) 59:66 (12%)	view
6	x 2	src/hyperpod_nemo_adapter/utils src/hyperpod_nemo_adapter/utils	train_utils.py train_utils.py	60:65 (4%) 101:106 (4%)	view
6	x 2	src/hyperpod_nemo_adapter/utils/callbacks src/hyperpod_nemo_adapter/utils/callbacks	peft_ckpt_io.py sharded_ckpt_io.py	73:78 (4%) 61:66 (10%)	view
6	x 2	src/hyperpod_nemo_adapter/utils/callbacks src/hyperpod_nemo_adapter/utils/callbacks	ckpt_io.py peft_ckpt_io.py	84:89 (7%) 181:186 (4%)	view
6	x 2	src/hyperpod_nemo_adapter/collections/model/nlp src/hyperpod_nemo_adapter/collections/model	sagemaker_llama_model.py sagemaker_base_model.py	112:117 (9%) 384:389 (1%)	view
6	x 2	src/hyperpod_nemo_adapter/collections/data src/hyperpod_nemo_adapter/collections/data	hf_data_module.py hf_data_module.py	52:58 (11%) 81:87 (11%)	view
6	x 2	src/hyperpod_nemo_adapter/collections/model/nlp src/hyperpod_nemo_adapter/collections/model/nlp	sagemaker_mistral_model.py sagemaker_mixtral_model.py	25:34 (27%) 25:34 (25%)	view
6	x 2	src/hyperpod_nemo_adapter/collections/data src/hyperpod_nemo_adapter/collections/data	hf_data_module.py hf_dpo_data_module.py	81:87 (11%) 38:44 (20%)	view
6	x 2	src/hyperpod_nemo_adapter/utils/callbacks src/hyperpod_nemo_adapter/utils/callbacks	checkpoint.py checkpoint.py	67:72 (1%) 274:279 (1%)	view
6	x 2	src/hyperpod_nemo_adapter/collections/model/nlp src/hyperpod_nemo_adapter/collections/model/nlp	sagemaker_deepseek_model.py sagemaker_deepseek_model.py	71:76 (8%) 80:85 (8%)	view
6	x 2	src/hyperpod_nemo_adapter/utils/callbacks src/hyperpod_nemo_adapter/utils/callbacks	ckpt_io.py peft_ckpt_io.py	84:89 (7%) 73:78 (4%)	view
6	x 2	src/hyperpod_nemo_adapter/collections/model/nlp src/hyperpod_nemo_adapter/collections/model	sagemaker_llama_model.py sagemaker_base_model.py	104:109 (9%) 384:389 (1%)	view
6	x 2	src/hyperpod_nemo_adapter/collections/model/nlp src/hyperpod_nemo_adapter/collections/model/nlp	sagemaker_mixtral_model.py sagemaker_qwen_model.py	25:34 (25%) 25:34 (27%)	view
6	x 2	src/hyperpod_nemo_adapter/collections/model/nlp src/hyperpod_nemo_adapter/collections/model/nlp	sagemaker_mistral_model.py sagemaker_qwen_model.py	25:34 (27%) 25:34 (27%)	view
6	x 2	src/hyperpod_nemo_adapter/collections/data src/hyperpod_nemo_adapter/collections/data	hf_data_module.py hf_dpo_data_module.py	52:58 (11%) 38:44 (20%)	view
6	x 2	src/hyperpod_nemo_adapter/utils/callbacks src/hyperpod_nemo_adapter/utils/callbacks	peft_ckpt_io.py peft_ckpt_io.py	73:78 (4%) 181:186 (4%)	view
6	x 2	src/hyperpod_nemo_adapter/collections/parts src/hyperpod_nemo_adapter/utils/callbacks	fsdp_strategy.py ckpt_io.py	494:500 (1%) 79:85 (7%)	view

Duplicated Units

The list of top 1 duplicated units.

See data for all 1 unit duplicate

Size	#	Folders	Files	Lines	Code
12	x 2	src/hyperpod_nemo_adapter/collections/data src/hyperpod_nemo_adapter/collections/data	dummy_data_module.py dummy_dpo_data_module.py	0:0  0:0	view