huggingface / transformers-research-projects

Duplication

Places in code with 6 or more lines that are exactly the same.

Longest Duplicates

The list of 50 longest duplicates.

See data for all 1,488 duplicates...

Size	#	Folders	Files	Lines	Code
1324	x 2	lxmert visual_bert	modeling_frcnn.py modeling_frcnn.py	37:1920 (100%) 37:1920 (100%)	view
440	x 2	lxmert visual_bert	visualizing_image.py visualizing_image.py	33:500 (100%) 33:500 (100%)	view
399	x 2	lxmert visual_bert	utils.py utils.py	47:554 (100%) 47:554 (100%)	view
261	x 2	rag-end2end-retriever rag	eval_rag.py eval_rag.py	17:320 (100%) 17:320 (100%)	view
212	x 2	tapex tapex	run_wikisql_with_tapex.py run_wikitablequestions_with_tapex.py	122:386 (44%) 120:384 (45%)	view
171	x 2	rag-end2end-retriever rag	utils_rag.py utils_rag.py	21:244 (100%) 21:244 (100%)	view
127	x 2	rag seq2seq-distillation	lightning_base.py lightning_base.py	11:168 (39%) 11:168 (39%)	view
109	x 2	lxmert visual_bert	extracting_data.py extracting_data.py	24:147 (100%) 24:147 (100%)	view
103	x 2	tapex tapex	run_wikisql_with_tapex.py run_wikitablequestions_with_tapex.py	426:550 (21%) 402:526 (22%)	view
100	x 2	rag-end2end-retriever seq2seq-distillation	lightning_base.py lightning_base.py	11:134 (30%) 11:134 (31%)	view
100	x 2	rag-end2end-retriever rag	lightning_base.py lightning_base.py	11:134 (30%) 11:134 (30%)	view
95	x 2	lxmert visual_bert	processing_image.py processing_image.py	31:151 (100%) 31:151 (100%)	view
88	x 2	rag seq2seq-distillation	lightning_base.py lightning_base.py	172:266 (27%) 172:266 (27%)	view
79	x 2	robust-speech-event robust-speech-event	run_speech_recognition_ctc_bnb.py run_speech_recognition_ctc_streaming.py	59:155 (14%) 58:154 (16%)	view
77	x 2	movement-pruning movement-pruning	masked_run_glue.py masked_run_squad.py	652:735 (9%) 770:853 (8%)	view
75	x 2	tapex tapex	run_wikisql_with_tapex.py run_wikitablequestions_with_tapex.py	554:649 (15%) 530:625 (16%)	view
75	x 2	distillation movement-pruning	run_squad_w_distillation.py masked_run_squad.py	539:617 (11%) 682:760 (8%)	view
75	x 2	rag-end2end-retriever rag	lightning_base.py lightning_base.py	138:227 (22%) 136:225 (23%)	view
73	x 2	rag-end2end-retriever rag	finetune_rag.py finetune_rag.py	502:574 (11%) 378:450 (13%)	view
73	x 2	movement-pruning movement-pruning	masked_run_glue.py masked_run_squad.py	106:196 (9%) 113:203 (7%)	view
67	x 2	seq2seq-distillation seq2seq-distillation	_test_seq2seq_examples.py _test_seq2seq_examples_multi_gpu.py	29:97 (18%) 16:84 (49%)	view
66	x 2	rag seq2seq-distillation	lightning_base.py lightning_base.py	278:354 (20%) 269:345 (20%)	view
61	x 2	rag-end2end-retriever rag	finetune_rag.py finetune_rag.py	414:487 (9%) 301:374 (11%)	view
61	x 2	bertology bertology	run_bertology.py run_prune_gpt.py	304:376 (19%) 279:351 (21%)	view
61	x 2	tapex tapex	run_wikisql_with_tapex.py run_wikitablequestions_with_tapex.py	40:120 (12%) 38:118 (13%)	view
59	x 2	rag-end2end-retriever rag	callbacks_rag.py callbacks_rag.py	43:117 (70%) 41:114 (71%)	view
58	x 2	wav2vec2 wav2vec2	run_asr.py run_common_voice.py	223:343 (18%) 142:262 (17%)	view
57	x 2	jax-projects/wav2vec2 wav2vec2	run_wav2vec2_pretrain_flax.py run_pretrain.py	283:363 (14%) 287:367 (22%)	view
57	x 2	mlm_wwm performer	run_mlm_wwm.py run_mlm_performer.py	114:177 (18%) 124:187 (13%)	view
56	x 2	jax-projects/dataset-streaming performer	run_mlm_flax_stream.py run_mlm_performer.py	109:170 (14%) 124:185 (13%)	view
56	x 2	jax-projects/dataset-streaming mlm_wwm	run_mlm_flax_stream.py run_mlm_wwm.py	109:170 (14%) 114:175 (18%)	view
53	x 2	robust-speech-event xtreme-s	run_speech_recognition_ctc_bnb.py run_xtreme_s.py	208:260 (9%) 264:316 (7%)	view
52	x 2	rag-end2end-retriever rag	finetune_rag.py finetune_rag.py	642:698 (7%) 487:544 (9%)	view
52	x 2	bert-loses-patience deebert	run_glue_with_pabee.py run_glue_deebert.py	587:652 (8%) 540:605 (8%)	view
50	x 2	jax-projects/dataset-streaming jax-projects/model_parallel	run_mlm_flax_stream.py run_clm_mp.py	67:128 (12%) 60:121 (11%)	view
50	x 2	robust-speech-event xtreme-s	run_speech_recognition_ctc_streaming.py run_xtreme_s.py	81:130 (10%) 94:143 (7%)	view
50	x 2	robust-speech-event xtreme-s	run_speech_recognition_ctc_bnb.py run_xtreme_s.py	82:131 (9%) 94:143 (7%)	view
50	x 2	deebert/src deebert/src	modeling_highway_bert.py modeling_highway_roberta.py	337:397 (20%) 95:154 (52%)	view
49	x 2	rag-end2end-retriever seq2seq-distillation	callbacks_rag.py callbacks.py	58:117 (58%) 22:81 (60%)	view
49	x 2	rag seq2seq-distillation	callbacks_rag.py callbacks.py	55:114 (59%) 22:81 (60%)	view
49	x 2	rag-end2end-retriever rag	use_own_knowledge_dataset.py use_own_knowledge_dataset.py	16:102 (43%) 23:110 (37%)	view
49	x 2	rag-end2end-retriever rag	finetune_rag.py finetune_rag.py	198:254 (7%) 197:254 (9%)	view
48	x 2	bert-loses-patience movement-pruning	run_glue_with_pabee.py masked_run_glue.py	458:508 (8%) 599:648 (6%)	view
47	x 2	rag-end2end-retriever seq2seq-distillation	lightning_base.py lightning_base.py	174:227 (14%) 172:225 (14%)	view
47	x 2	distillation movement-pruning	run_squad_w_distillation.py masked_run_squad.py	459:516 (6%) 628:685 (5%)	view
46	x 2	rag-end2end-retriever rag	lightning_base.py lightning_base.py	229:278 (13%) 227:275 (14%)	view
45	x 2	bert-loses-patience deebert	run_glue_with_pabee.py run_glue_deebert.py	398:447 (7%) 385:434 (7%)	view
45	x 2	distillation mm-imdb	run_squad_w_distillation.py run_mmimdb.py	707:766 (6%) 435:494 (10%)	view
44	x 2	self-training-text-classification self-training-text-classification	finetuning.py finetuning.py	280:331 (7%) 342:393 (7%)	view
42	x 2	deebert mm-imdb	run_glue_deebert.py run_mmimdb.py	517:567 (7%) 410:460 (9%)	view

Duplicated Units

The list of top 10 duplicated units.

See data for all 10 unit duplicates...

Size	#	Folders	Files	Lines	Code
144	x 2	visual_bert lxmert	modeling_frcnn.py modeling_frcnn.py	0:0  0:0	view
109	x 2	rag rag-end2end-retriever	eval_rag.py eval_rag.py	0:0  0:0	view
58	x 2	seq2seq-distillation rag	lightning_base.py lightning_base.py	0:0  0:0	view
32	x 2	visual_bert lxmert	utils.py utils.py	0:0  0:0	view
29	x 2	visual_bert lxmert	extracting_data.py extracting_data.py	0:0  0:0	view
29	x 2	visual_bert lxmert	visualizing_image.py visualizing_image.py	0:0  0:0	view
28	x 2	visual_bert lxmert	processing_image.py processing_image.py	0:0  0:0	view
23	x 2	rag rag-end2end-retriever	utils_rag.py utils_rag.py	0:0  0:0	view
9	x 2	deebert/src deebert/src	modeling_highway_roberta.py modeling_highway_bert.py	0:0  0:0	view
7	x 2	rag rag-end2end-retriever	use_own_knowledge_dataset.py use_own_knowledge_dataset.py	0:0  0:0	view