huggingface / trl-tuto Duplication

Intro

For duplication, we look at places in code where there are 6 or more lines of code that are exactly the same.
Before duplication is calculated, the code is cleaned to remove empty lines, comments, and frequently duplicated constructs such as imports.
You should aim at having as little as possible (<5%) of duplicated code as high-level of duplication can lead to maintenance difficulties, poor factoring, and logical contradictions.

Learn more...

Duplication Overall

47% duplication:

2,057 cleaned lines of cleaned code (without empty lines, comments, and frequently duplicated constructs such as imports)
972 duplicated lines

3,146 duplicates

dependency graphs: 2D graph | 3D graph | 3D graph (with duplicates)...

Duplication per Extension

Duplication per Component (primary)

Longest Duplicates

The list of 50 longest duplicates.

See data for all 3,146 duplicates...

Size	#	Folders	Files	Lines	Code
39	x 2	train_rick train_rick	01_sft.ipynb 02_grpo.ipynb	8:46 (8%) 8:46 (7%)	view
35	x 2	train_rick train_rick	00_llm_pretraining_and_data_preparati... 01_sft.ipynb	20:54 (5%) 19:53 (7%)	view
28	x 2	train_rick train_rick	00_llm_pretraining_and_data_preparati... 02_grpo.ipynb	20:47 (4%) 19:46 (5%)	view
25	x 2	train_rick train_rick	01_sft.ipynb 02_grpo.ipynb	441:465 (5%) 464:488 (5%)	view
25	x 2	train_rick train_rick	00_llm_pretraining_and_data_preparati... 01_sft.ipynb	630:654 (3%) 441:465 (5%)	view
25	x 2	train_rick train_rick	00_llm_pretraining_and_data_preparati... 02_grpo.ipynb	630:654 (3%) 464:488 (5%)	view
11	x 2	train_rick train_rick	00_llm_pretraining_and_data_preparati... 00_llm_pretraining_and_data_preparati...	92:102 (1%) 427:437 (1%)	view
11	x 2	train_rick train_rick	02_grpo.ipynb 02_grpo.ipynb	79:89 (2%) 102:112 (2%)	view
11	x 2	train_rick train_rick	00_llm_pretraining_and_data_preparati... 02_grpo.ipynb	8:18 (1%) 8:18 (2%)	view
11	x 2	train_rick train_rick	00_llm_pretraining_and_data_preparati... 01_sft.ipynb	8:18 (1%) 8:18 (2%)	view
10	x 2	train_rick train_rick	00_llm_pretraining_and_data_preparati... 01_sft.ipynb	484:493 (1%) 140:149 (2%)	view
10	x 2	train_rick train_rick	01_sft.ipynb 01_sft.ipynb	114:123 (2%) 389:398 (2%)	view
10	x 2	train_rick train_rick	00_llm_pretraining_and_data_preparati... 02_grpo.ipynb	484:493 (1%) 334:343 (2%)	view
10	x 2	train_rick train_rick	00_llm_pretraining_and_data_preparati... 01_sft.ipynb	59:68 (1%) 389:398 (2%)	view
10	x 2	train_rick train_rick	01_sft.ipynb 02_grpo.ipynb	140:149 (2%) 334:343 (2%)	view
10	x 2	train_rick train_rick	00_llm_pretraining_and_data_preparati... 01_sft.ipynb	59:68 (1%) 114:123 (2%)	view
9	x 2	train_rick train_rick	01_sft.ipynb 02_grpo.ipynb	47:55 (1%) 48:56 (1%)	view
8	x 2	train_rick train_rick	02_grpo.ipynb 02_grpo.ipynb	167:174 (1%) 334:341 (1%)	view
8	x 2	train_rick train_rick	00_llm_pretraining_and_data_preparati... 01_sft.ipynb	205:212 (1%) 159:166 (1%)	view
8	x 2	train_rick train_rick	00_llm_pretraining_and_data_preparati... 00_llm_pretraining_and_data_preparati...	39:46 (1%) 127:134 (1%)	view
8	x 2	train_rick train_rick	01_sft.ipynb 02_grpo.ipynb	234:241 (1%) 93:100 (1%)	view
8	x 2	train_rick train_rick	01_sft.ipynb 01_sft.ipynb	159:166 (1%) 412:419 (1%)	view
8	x 2	train_rick train_rick	00_llm_pretraining_and_data_preparati... 00_llm_pretraining_and_data_preparati...	502:509 (1%) 608:615 (1%)	view
8	x 2	train_rick train_rick	00_llm_pretraining_and_data_preparati... 01_sft.ipynb	205:212 (1%) 140:147 (1%)	view
8	x 2	train_rick train_rick	01_sft.ipynb 02_grpo.ipynb	234:241 (1%) 191:198 (1%)	view
8	x 2	train_rick train_rick	00_llm_pretraining_and_data_preparati... 01_sft.ipynb	22:29 (1%) 140:147 (1%)	view
8	x 2	train_rick train_rick	00_llm_pretraining_and_data_preparati... 01_sft.ipynb	22:29 (1%) 159:166 (1%)	view
8	x 2	train_rick train_rick	02_grpo.ipynb 02_grpo.ipynb	65:72 (1%) 191:198 (1%)	view
8	x 2	train_rick train_rick	00_llm_pretraining_and_data_preparati... 02_grpo.ipynb	205:212 (1%) 217:224 (1%)	view
8	x 2	train_rick train_rick	02_grpo.ipynb 02_grpo.ipynb	167:174 (1%) 353:360 (1%)	view
8	x 2	train_rick train_rick	02_grpo.ipynb 02_grpo.ipynb	93:100 (1%) 129:136 (1%)	view
8	x 2	train_rick train_rick	00_llm_pretraining_and_data_preparati... 01_sft.ipynb	39:46 (1%) 159:166 (1%)	view
8	x 2	train_rick train_rick	00_llm_pretraining_and_data_preparati... 00_llm_pretraining_and_data_preparati...	484:491 (1%) 518:525 (1%)	view
8	x 2	train_rick train_rick	00_llm_pretraining_and_data_preparati... 01_sft.ipynb	39:46 (1%) 140:147 (1%)	view
8	x 2	train_rick train_rick	00_llm_pretraining_and_data_preparati... 00_llm_pretraining_and_data_preparati...	39:46 (1%) 154:161 (1%)	view
8	x 2	train_rick train_rick	00_llm_pretraining_and_data_preparati... 00_llm_pretraining_and_data_preparati...	247:254 (1%) 388:395 (1%)	view
8	x 2	train_rick train_rick	00_llm_pretraining_and_data_preparati... 01_sft.ipynb	608:615 (1%) 140:147 (1%)	view
8	x 2	train_rick train_rick	02_grpo.ipynb 02_grpo.ipynb	167:174 (1%) 374:381 (1%)	view
8	x 2	train_rick train_rick	01_sft.ipynb 01_sft.ipynb	140:147 (1%) 159:166 (1%)	view
8	x 2	train_rick train_rick	00_llm_pretraining_and_data_preparati... 01_sft.ipynb	205:212 (1%) 114:121 (1%)	view
8	x 2	train_rick train_rick	00_llm_pretraining_and_data_preparati... 02_grpo.ipynb	452:459 (1%) 420:427 (1%)	view
8	x 2	train_rick train_rick	00_llm_pretraining_and_data_preparati... 01_sft.ipynb	409:416 (1%) 296:303 (1%)	view
8	x 2	train_rick train_rick	01_sft.ipynb 02_grpo.ipynb	234:241 (1%) 167:174 (1%)	view
8	x 2	train_rick train_rick	00_llm_pretraining_and_data_preparati... 02_grpo.ipynb	502:509 (1%) 217:224 (1%)	view
8	x 2	train_rick train_rick	00_llm_pretraining_and_data_preparati... 01_sft.ipynb	22:29 (1%) 114:121 (1%)	view
8	x 2	train_rick train_rick	00_llm_pretraining_and_data_preparati... 01_sft.ipynb	484:491 (1%) 159:166 (1%)	view
8	x 2	train_rick train_rick	00_llm_pretraining_and_data_preparati... 02_grpo.ipynb	127:134 (1%) 420:427 (1%)	view
8	x 2	train_rick train_rick	00_llm_pretraining_and_data_preparati... 00_llm_pretraining_and_data_preparati...	484:491 (1%) 567:574 (1%)	view
8	x 2	train_rick train_rick	00_llm_pretraining_and_data_preparati... 01_sft.ipynb	608:615 (1%) 159:166 (1%)	view
8	x 2	train_rick train_rick	00_llm_pretraining_and_data_preparati... 02_grpo.ipynb	205:212 (1%) 240:247 (1%)	view