huggingface / accelerate Duplication

Intro

For duplication, we look at places in code where there are 6 or more lines of code that are exactly the same.
Before duplication is calculated, the code is cleaned to remove empty lines, comments, and frequently duplicated constructs such as imports.
You should aim at having as little as possible (<5%) of duplicated code as high-level of duplication can lead to maintenance difficulties, poor factoring, and logical contradictions.

Learn more...

Duplication Overall

13% duplication:

20,138 cleaned lines of cleaned code (without empty lines, comments, and frequently duplicated constructs such as imports)
2,729 duplicated lines

475 duplicates

dependency graphs: 2D graph | 3D graph | 3D graph (with duplicates)...

Duplication per Extension

Duplication per Component (primary)

Longest Duplicates

The list of 50 longest duplicates.

See data for all 475 duplicates...

Size	#	Folders	Files	Lines	Code
54	x 2	src/accelerate/utils src/accelerate/utils	launch.py launch.py	219:277 (9%) 412:470 (9%)	view
52	x 2	manim_animations/dataloaders manim_animations/dataloaders	stage_5.py stage_6.py	33:88 (30%) 34:89 (32%)	view
51	x 2	benchmarks/fp8/ms_amp benchmarks/fp8/transformer_engine	fp8_utils.py fp8_utils.py	17:109 (86%) 17:109 (87%)	view
39	x 2	benchmarks/fp8/transformer_engine benchmarks/fp8/transformer_engine	ddp.py fsdp.py	97:144 (46%) 114:161 (41%)	view
38	x 2	benchmarks/fp8/torchao benchmarks/fp8/torchao	ddp.py fsdp.py	112:158 (38%) 127:173 (35%)	view
32	x 2	manim_animations/dataloaders manim_animations/dataloaders	stage_5.py stage_6.py	171:203 (18%) 161:193 (19%)	view
32	x 2	benchmarks/fp8/torchao benchmarks/fp8/torchao	ddp.py non_distributed.py	34:73 (32%) 33:73 (34%)	view
31	x 2	benchmarks/fp8/torchao benchmarks/fp8/transformer_engine	fsdp.py fsdp.py	136:173 (28%) 124:161 (32%)	view
31	x 2	benchmarks/fp8/torchao benchmarks/fp8/transformer_engine	ddp.py ddp.py	121:158 (31%) 107:144 (36%)	view
31	x 2	benchmarks/fp8/torchao benchmarks/fp8/transformer_engine	fsdp.py ddp.py	136:173 (28%) 107:144 (36%)	view
31	x 2	benchmarks/fp8/torchao benchmarks/fp8/transformer_engine	ddp.py fsdp.py	121:158 (31%) 124:161 (32%)	view
30	x 2	benchmarks/fp8/torchao benchmarks/fp8/transformer_engine	fp8_utils.py fp8_utils.py	17:62 (51%) 17:62 (51%)	view
30	x 2	benchmarks/fp8/ms_amp benchmarks/fp8/torchao	fp8_utils.py fp8_utils.py	17:62 (50%) 17:62 (51%)	view
28	x 2	benchmarks/fp8/transformer_engine benchmarks/fp8/transformer_engine	ddp.py fsdp.py	61:95 (33%) 73:107 (29%)	view
27	x 2	benchmarks/fp8/torchao benchmarks/fp8/transformer_engine	fp8_utils.py fp8_utils.py	80:116 (46%) 80:116 (46%)	view
26	x 2	manim_animations/dataloaders manim_animations/dataloaders	stage_5.py stage_6.py	103:130 (15%) 92:119 (16%)	view
26	x 2	src/accelerate/commands/config src/accelerate/commands/config	cluster.py sagemaker.py	169:195 (3%) 174:200 (11%)	view
25	x 2	benchmarks/fp8/ms_amp benchmarks/fp8/transformer_engine	non_distributed.py non_distributed.py	71:102 (35%) 86:117 (32%)	view
25	x 2	manim_animations/big_model_inference manim_animations/big_model_inference	stage_4.py stage_5.py	18:46 (22%) 18:47 (15%)	view
23	x 2	manim_animations/big_model_inference manim_animations/big_model_inference	stage_3.py stage_5.py	23:49 (20%) 24:50 (14%)	view
22	x 2	benchmarks/fp8/transformer_engine benchmarks/fp8/transformer_engine	fsdp.py non_distributed.py	134:159 (23%) 107:132 (28%)	view
22	x 2	manim_animations/big_model_inference manim_animations/big_model_inference	stage_2.py stage_3.py	20:45 (25%) 21:46 (19%)	view
22	x 2	benchmarks/fp8/torchao benchmarks/fp8/transformer_engine	fsdp.py non_distributed.py	146:171 (20%) 107:132 (28%)	view
22	x 2	benchmarks/fp8/transformer_engine benchmarks/fp8/transformer_engine	ddp.py non_distributed.py	117:142 (26%) 107:132 (28%)	view
22	x 2	benchmarks/fp8/torchao benchmarks/fp8/transformer_engine	ddp.py non_distributed.py	131:156 (22%) 107:132 (28%)	view
21	x 2	manim_animations/big_model_inference manim_animations/big_model_inference	stage_3.py stage_4.py	23:46 (18%) 23:46 (18%)	view
21	x 2	manim_animations/big_model_inference manim_animations/big_model_inference	stage_2.py stage_5.py	22:45 (24%) 24:47 (13%)	view
21	x 2	manim_animations/dataloaders manim_animations/dataloaders	stage_5.py stage_6.py	131:152 (12%) 121:142 (12%)	view
21	x 2	manim_animations/big_model_inference manim_animations/big_model_inference	stage_2.py stage_4.py	22:45 (24%) 23:46 (18%)	view
21	x 2	benchmarks/fp8/torchao benchmarks/fp8/torchao	ddp.py fsdp.py	84:110 (21%) 94:120 (19%)	view
20	x 2	benchmarks/fp8/ms_amp benchmarks/fp8/torchao	fp8_utils.py fp8_utils.py	80:109 (33%) 80:109 (34%)	view
19	x 2	src/accelerate/commands/config src/accelerate/commands/config	cluster.py sagemaker.py	197:215 (2%) 202:220 (8%)	view
19	x 2	benchmarks/fp8/torchao benchmarks/fp8/transformer_engine	distrib_deepspeed.py distrib_deepspeed.py	191:211 (12%) 169:189 (14%)	view
18	x 2	src/accelerate/utils src/accelerate/utils	launch.py launch.py	175:193 (3%) 284:302 (3%)	view
18	x 2	benchmarks/fp8/torchao benchmarks/fp8/transformer_engine	distrib_deepspeed.py distrib_deepspeed.py	84:102 (11%) 70:88 (14%)	view
17	x 2	benchmarks/fp8/torchao benchmarks/fp8/transformer_engine	ddp.py fsdp.py	88:109 (17%) 83:104 (18%)	view
17	x 2	benchmarks/fp8/torchao benchmarks/fp8/transformer_engine	ddp.py ddp.py	88:109 (17%) 71:92 (20%)	view
17	x 2	benchmarks/fp8/torchao benchmarks/fp8/transformer_engine	fsdp.py ddp.py	98:119 (15%) 71:92 (20%)	view
17	x 2	benchmarks/fp8/torchao benchmarks/fp8/transformer_engine	fsdp.py fsdp.py	98:119 (15%) 83:104 (18%)	view
17	x 2	benchmarks/fp8/torchao benchmarks/fp8/transformer_engine	distrib_deepspeed.py distrib_deepspeed.py	171:188 (11%) 150:167 (13%)	view
17	x 2	src/accelerate src/accelerate/utils	accelerator.py __init__.py	54:70 (<1%) 43:59 (6%)	view
16	x 2	benchmarks/fp8/transformer_engine benchmarks/fp8/transformer_engine	ddp.py non_distributed.py	82:100 (19%) 73:91 (20%)	view
16	x 2	src/accelerate/commands src/accelerate/commands	launch.py launch.py	806:822 (1%) 860:875 (1%)	view
16	x 2	benchmarks/fp8/torchao benchmarks/fp8/transformer_engine	distrib_deepspeed.py distrib_deepspeed.py	118:134 (10%) 105:121 (12%)	view
15	x 2	benchmarks/fp8/ms_amp benchmarks/fp8/torchao	ddp.py fsdp.py	89:108 (20%) 137:156 (13%)	view
15	x 2	benchmarks/fp8/ms_amp benchmarks/fp8/transformer_engine	ddp.py fsdp.py	89:108 (20%) 125:144 (15%)	view
15	x 2	benchmarks/fp8/ms_amp benchmarks/fp8/torchao	ddp.py ddp.py	89:108 (20%) 122:141 (15%)	view
15	x 2	benchmarks/fp8/torchao benchmarks/fp8/torchao	ddp.py fsdp.py	63:78 (15%) 69:84 (13%)	view
15	x 2	benchmarks/fp8/ms_amp benchmarks/fp8/ms_amp	ddp.py non_distributed.py	65:82 (20%) 59:76 (21%)	view
15	x 2	benchmarks/fp8/ms_amp benchmarks/fp8/transformer_engine	ddp.py ddp.py	89:108 (20%) 108:127 (17%)	view

Duplicated Units

The list of top 2 duplicated units.

See data for all 2 unit duplicates...

Size	#	Folders	Files	Lines	Code
37	x 3	benchmarks/fp8/torchao benchmarks/fp8/transformer_engine benchmarks/fp8/ms_amp	fp8_utils.py fp8_utils.py fp8_utils.py	0:0 0:0 0:0	view
12	x 2	manim_animations/dataloaders manim_animations/dataloaders	stage_0.py stage_1.py	0:0 0:0	view