alibaba / Pai-Megatron-Patch

Duplication

Places in code with 6 or more lines that are exactly the same.

Longest Duplicates

The list of 50 longest duplicates.

See data for all 10,214 duplicates...

Size	#	Folders	Files	Lines	Code
1267	x 2	megatron_patch/model/llava megatron_patch/model/qwen_vl	transformer.py transformer.py	39:1870 (100%) 39:1870 (100%)	view
795	x 2	megatron_patch/model/llama2 megatron_patch/model/llava	transformer.py transformer.py	732:1868 (62%) 731:1867 (62%)	view
795	x 2	megatron_patch/model/llama2 megatron_patch/model/qwen_vl	transformer.py transformer.py	732:1868 (62%) 731:1867 (62%)	view
726	x 2	megatron_patch/model/llava megatron_patch/model/mistral	transformer.py transformer.py	838:1865 (57%) 836:1863 (57%)	view
726	x 2	megatron_patch/model/mistral megatron_patch/model/qwen_vl	transformer.py transformer.py	836:1863 (57%) 838:1865 (57%)	view
726	x 2	megatron_patch/model/llama2 megatron_patch/model/mistral	transformer.py transformer.py	839:1866 (57%) 836:1863 (57%)	view
651	x 2	megatron_patch/model/deepseek_v2/moe megatron_patch/model/mixtral/moe	experts.py experts.py	16:832 (100%) 16:833 (100%)	view
481	x 2	megatron_patch/model/falcon megatron_patch/model/falcon40b	language_model.py language_model.py	29:696 (100%) 29:696 (100%)	view
463	x 2	megatron_patch/model/baichuan2 megatron_patch/model/qwen	transformer.py transformer.py	1225:1863 (36%) 1167:1805 (37%)	view
452	x 2	megatron_patch/model/qwen2_vl megatron_patch/model/qwen2_vl	attention.py attention_vision.py	114:744 (87%) 113:743 (88%)	view
442	x 2	megatron_patch/model/llama2 megatron_patch/model/qwen1_5_megablocks	language_model.py language_model.py	30:651 (100%) 29:650 (100%)	view
416	x 2	megatron_patch/model/baichuan2 megatron_patch/model/qwen	language_model.py language_model.py	56:644 (95%) 46:633 (97%)	view
387	x 2	megatron_patch/model/llama3/transformer megatron_patch/model/qwen1_5/transformer	attention.py attention.py	26:595 (100%) 26:595 (100%)	view
386	x 2	megatron_patch/model/mistral megatron_patch/model/qwen_vl	transformer.py transformer.py	39:616 (30%) 39:616 (30%)	view
386	x 2	megatron_patch/model/llava megatron_patch/model/mistral	transformer.py transformer.py	39:616 (30%) 39:616 (30%)	view
377	x 2	megatron_patch/model/falcon40b megatron_patch/model/llama	language_model.py language_model.py	29:558 (78%) 29:558 (76%)	view
377	x 2	megatron_patch/model/falcon megatron_patch/model/llama	language_model.py language_model.py	29:558 (78%) 29:558 (76%)	view
319	x 2	megatron_patch/model/baichuan2 megatron_patch/model/llava	transformer.py transformer.py	1225:1652 (25%) 1224:1651 (25%)	view
319	x 2	megatron_patch/model/qwen megatron_patch/model/qwen_vl	transformer.py transformer.py	1167:1594 (26%) 1224:1651 (25%)	view
319	x 2	megatron_patch/model/llama2 megatron_patch/model/qwen	transformer.py transformer.py	1225:1652 (25%) 1167:1594 (26%)	view
319	x 2	megatron_patch/model/baichuan2 megatron_patch/model/mistral	transformer.py transformer.py	1225:1652 (25%) 1222:1649 (25%)	view
319	x 2	megatron_patch/model/mistral megatron_patch/model/qwen	transformer.py transformer.py	1222:1649 (25%) 1167:1594 (26%)	view
319	x 2	megatron_patch/model/baichuan2 megatron_patch/model/llama2	transformer.py transformer.py	1225:1652 (25%) 1225:1652 (25%)	view
319	x 2	megatron_patch/model/baichuan2 megatron_patch/model/qwen_vl	transformer.py transformer.py	1225:1652 (25%) 1224:1651 (25%)	view
319	x 2	megatron_patch/model/llava megatron_patch/model/qwen	transformer.py transformer.py	1224:1651 (25%) 1167:1594 (26%)	view
284	x 2	megatron_patch/model/llama2 megatron_patch/model/qwen	transformer.py transformer.py	147:572 (22%) 141:566 (23%)	view
269	x 2	megatron_patch/model/llama2 megatron_patch/model/qwen1_5_megablocks	transformer.py transformer.py	574:962 (21%) 465:852 (23%)	view
257	x 2	megatron_patch/model/llama3 megatron_patch/model/qwen	transformer_legacy.py transformer.py	1172:1514 (20%) 1156:1498 (21%)	view
250	x 2	megatron_patch/model/llama2 megatron_patch/model/llama3	transformer.py transformer_legacy.py	1225:1556 (19%) 1183:1514 (20%)	view
250	x 2	megatron_patch/model/llama3 megatron_patch/model/qwen_vl	transformer_legacy.py transformer.py	1183:1514 (20%) 1224:1555 (19%)	view
250	x 2	megatron_patch/model/llama3 megatron_patch/model/llava	transformer_legacy.py transformer.py	1183:1514 (20%) 1224:1555 (19%)	view
250	x 2	megatron_patch/model/baichuan2 megatron_patch/model/llama3	transformer.py transformer_legacy.py	1225:1556 (19%) 1183:1514 (20%)	view
250	x 2	megatron_patch/model/llama3 megatron_patch/model/mistral	transformer_legacy.py transformer.py	1183:1514 (20%) 1222:1553 (19%)	view
244	x 2	megatron_patch/model/llava megatron_patch/model/qwen	transformer.py transformer.py	845:1205 (19%) 791:1151 (20%)	view
244	x 2	megatron_patch/model/qwen megatron_patch/model/qwen_vl	transformer.py transformer.py	791:1151 (20%) 845:1205 (19%)	view
244	x 2	megatron_patch/model/llama2 megatron_patch/model/qwen	transformer.py transformer.py	846:1206 (19%) 791:1151 (20%)	view
244	x 2	megatron_patch/model/mistral megatron_patch/model/qwen	transformer.py transformer.py	843:1203 (19%) 791:1151 (20%)	view
242	x 2	megatron_patch/model/llama2 megatron_patch/model/qwen_vl	transformer.py transformer.py	39:391 (19%) 39:391 (19%)	view
242	x 2	megatron_patch/model/llama2 megatron_patch/model/mistral	transformer.py transformer.py	39:391 (19%) 39:391 (19%)	view
242	x 2	megatron_patch/model/llama2 megatron_patch/model/llava	transformer.py transformer.py	39:391 (19%) 39:391 (19%)	view
237	x 2	megatron_patch/model/qwen1_5/transformer megatron_patch/model/qwen2/transformer	attention.py attention.py	26:382 (61%) 22:378 (60%)	view
237	x 2	megatron_patch/model/llama3/transformer megatron_patch/model/qwen2/transformer	attention.py attention.py	26:382 (61%) 22:378 (60%)	view
223	x 2	megatron_patch/model/llama2 megatron_patch/model/qwen	language_model.py language_model.py	30:369 (50%) 30:369 (52%)	view
223	x 2	megatron_patch/model/qwen megatron_patch/model/qwen1_5_megablocks	language_model.py language_model.py	30:369 (52%) 29:368 (50%)	view
222	x 2	megatron_patch/model/mistral megatron_patch/model/qwen1_5_megablocks	transformer.py transformer.py	981:1288 (17%) 868:1175 (19%)	view
222	x 2	megatron_patch/model/qwen1_5_megablocks megatron_patch/model/qwen_vl	transformer.py transformer.py	868:1175 (19%) 983:1290 (17%)	view
222	x 2	megatron_patch/model/llava megatron_patch/model/qwen1_5_megablocks	transformer.py transformer.py	983:1290 (17%) 868:1175 (19%)	view
222	x 2	megatron_patch/model/llama2 megatron_patch/model/qwen1_5_megablocks	transformer.py transformer.py	984:1291 (17%) 868:1175 (19%)	view
215	x 2	megatron_patch/model/mistral megatron_patch/model/qwen	language_model.py language_model.py	31:360 (47%) 30:359 (50%)	view
215	x 2	megatron_patch/model/mistral megatron_patch/model/qwen1_5_megablocks	language_model.py language_model.py	31:360 (47%) 29:358 (48%)	view

Duplicated Units

The list of top 24 duplicated units.

See data for all 24 unit duplicates...

Size	#	Folders	Files	Lines	Code
349	x 2	megatron_patch/model/mixtral/moe megatron_patch/model/deepseek_v2/moe	experts.py experts.py	0:0  0:0	view
192	x 6	megatron_patch/model/llava megatron_patch/model/mistral megatron_patch/model/baichuan2 megatron_patch/model/qwen_vl megatron_patch/model/llama2 megatron_patch/model/qwen	transformer.py transformer.py transformer.py transformer.py transformer.py transformer.py	0:0  0:0  0:0  0:0  0:0  0:0	view
134	x 2	toolkits/model_checkpoints_convertor/qwen toolkits/model_checkpoints_convertor/qwen	hf2mcore_qwen2.5_vl.py hf2mcore_qwen2_vl.py	0:0  0:0	view
100	x 2	toolkits/model_checkpoints_convertor/qwen toolkits/model_checkpoints_convertor/qwen	hf2mcore_qwen1.5_moe.py hf2mcore_qwen1.5_dense_mha_to_moe.py	0:0  0:0	view
102	x 3	megatron_patch/model/mixtral/transformer megatron_patch/model/deepseek_v2 megatron_patch/model/qwen2/transformer	mlp.py mlp.py mlp.py	0:0  0:0  0:0	view
91	x 2	megatron_patch/model/qwen1_5_megablocks megatron_patch/model/llama2	language_model.py language_model.py	0:0  0:0	view
89	x 2	megatron_patch/model/baichuan2 megatron_patch/model/qwen	language_model.py language_model.py	0:0  0:0	view
77	x 3	megatron_patch/model/falcon40b megatron_patch/model/galactica megatron_patch/model/falcon	language_model.py language_model.py language_model.py	0:0  0:0  0:0	view
60	x 3	megatron_patch/model/qwen2_vl megatron_patch/model/qwen2_vl megatron_patch/model/mixtral/transformer	attention.py attention_vision.py attention.py	0:0  0:0  0:0	view
59	x 2	megatron_patch/model/qwen1_5/transformer megatron_patch/model/llama3/transformer	attention.py attention.py	0:0  0:0	view
58	x 2	megatron_patch/model/mixtral megatron_patch/model/mixtral_bak	model.py model.py	0:0  0:0	view
36	x 2	megatron_patch/model/mixtral megatron_patch/model/mixtral_bak	layer_specs.py layer_specs.py	0:0  0:0	view
23	x 4	megatron_patch/model/falcon40b megatron_patch/model/galactica megatron_patch/model/llama megatron_patch/model/falcon	gpt_model.py gpt_model.py gpt_model.py gpt_model.py	0:0  0:0  0:0  0:0	view
23	x 2	megatron_patch/model/mixtral/moe megatron_patch/model/deepseek_v2/moe	moe_layer.py moe_layer.py	0:0  0:0	view
25	x 2	megatron_patch/model/qwen1_5/transformer megatron_patch/model/llama3/transformer	mlp.py mlp.py	0:0  0:0	view
25	x 2	megatron_patch/model/qwen1_5_megablocks megatron_patch/model/llama2	rotary_pos_embedding.py rotary_pos_embedding.py	0:0  0:0	view
21	x 2	megatron_patch/model/qwen2_vl megatron_patch/model/qwen2_5_vl	model.py model.py	0:0  0:0	view
20	x 8	megatron_patch/model/llava megatron_patch/model/qwen1_5_megablocks megatron_patch/model/mistral megatron_patch/model/baichuan megatron_patch/model/qwen_vl megatron_patch/model/llama2 megatron_patch/model/llama3 megatron_patch/model/qwen	gpt_model.py gpt_model.py gpt_model.py gpt_model.py gpt_model.py gpt_model.py gpt_model.py gpt_model.py	0:0  0:0  0:0  0:0  0:0  0:0  0:0  0:0	view
23	x 2	megatron_patch/model/llava megatron_patch/model/llama	rotary_pos_embedding.py positional_embeddings.py	0:0  0:0	view
19	x 2	megatron_patch/model/glm130b megatron_patch/model/bloom	gpt_model.py gpt_model.py	0:0  0:0	view
24	x 2	megatron_patch/model/mixtral_bak/moe megatron_patch/model/qwen1_5/moe	router.py router.py	0:0  0:0	view
17	x 2	toolkits/model_checkpoints_convertor/baichuan toolkits/model_checkpoints_convertor/baichuan2	configuration_baichuan.py configuration_baichuan.py	0:0  0:0	view
16	x 2	megatron_patch/tokenizer megatron_patch/tokenizer	tokenization_baichuan.py tokenization_yi.py	0:0  0:0	view
13	x 2	megatron_patch/model/qwen1_5 megatron_patch/model/llama3	model.py model.py	0:0  0:0	view