aliyun / aicb - Duplication

Intro

For duplication, we look at places in code where there are 6 or more lines of code that are exactly the same.
Before duplication is calculated, the code is cleaned to remove empty lines, comments, and frequently duplicated constructs such as imports.
You should aim at having as little as possible (<5%) of duplicated code as high-level of duplication can lead to maintenance difficulties, poor factoring, and logical contradictions.

Learn more...

Duplication Overall

23% duplication:

3,959 cleaned lines of cleaned code (without empty lines, comments, and frequently duplicated constructs such as imports)
929 duplicated lines

109 duplicates

dependency graphs: 2D graph | 3D graph | 3D graph (with duplicates)...

Duplication per Extension

Duplication per Component (primary)

Longest Duplicates

The list of 50 longest duplicates.

See data for all 109 duplicates...

Size	#	Folders	Files	Lines	Code
32	x 2	workload_generator workload_generator	AIOB_simAI_workload_generator.py AIOB_simAI_workload_generator.py	504:542 (3%) 744:776 (3%)	view
24	x 2	workload_generator workload_generator	generate_megatron_workload.py generate_megatron_workload.py	157:184 (6%) 210:235 (6%)	view
21	x 2	workload_generator workload_generator	generate_megatron_workload.py generate_megatron_workload.py	157:179 (5%) 333:353 (5%)	view
21	x 2	workload_generator workload_generator	generate_megatron_workload.py generate_megatron_workload.py	210:231 (5%) 333:353 (5%)	view
17	x 2	workload_generator workload_generator	AIOB_simAI_workload_generator.py AIOB_simAI_workload_generator.py	212:229 (2%) 578:594 (2%)	view
15	x 2	workload_generator workload_generator	AIOB_simAI_workload_generator.py AIOB_simAI_workload_generator.py	249:263 (1%) 596:610 (1%)	view
15	x 2	workload_generator workload_generator	AIOB_simAI_workload_generator.py AIOB_simAI_workload_generator.py	615:629 (1%) 702:716 (1%)	view
14	x 2	workload_generator workload_generator	AIOB_simAI_workload_generator.py AIOB_simAI_workload_generator.py	348:361 (1%) 492:505 (1%)	view
13	x 2	workload_generator workload_generator	AIOB_simAI_workload_generator.py AIOB_simAI_workload_generator.py	409:421 (1%) 684:697 (1%)	view
13	x 2	workload_generator workload_generator	AIOB_simAI_workload_generator.py AIOB_simAI_workload_generator.py	389:401 (1%) 664:676 (1%)	view
12	x 2	workload_generator workload_generator	AIOB_simAI_workload_generator.py AIOB_simAI_workload_generator.py	149:160 (1%) 546:557 (1%)	view
11	x 2	workload_generator workload_generator	generate_megatron_workload.py generate_megatron_workload.py	263:273 (2%) 318:328 (2%)	view
11	x 2	workload_generator workload_generator	generate_megatron_workload.py generate_megatron_workload.py	146:156 (2%) 196:206 (2%)	view
11	x 2	workload_generator workload_generator	AIOB_simAI_workload_generator.py AIOB_simAI_workload_generator.py	475:485 (1%) 707:717 (1%)	view
11	x 2	workload_generator workload_generator	generate_deepspeed_stage3_workload.py generate_deepspeed_stage3_workload.py	126:136 (4%) 139:149 (4%)	view
11	x 2	workload_generator workload_generator	generate_megatron_workload.py generate_megatron_workload.py	276:286 (2%) 318:328 (2%)	view
11	x 2	workload_generator workload_generator	generate_megatron_workload.py generate_megatron_workload.py	183:193 (2%) 246:256 (2%)	view
11	x 2	workload_generator workload_generator	AIOB_simAI_workload_generator.py AIOB_simAI_workload_generator.py	386:396 (1%) 422:432 (1%)	view
11	x 2	workload_generator workload_generator	generate_megatron_workload.py generate_megatron_workload.py	263:273 (2%) 276:286 (2%)	view
10	x 2	workload_generator workload_generator	AIOB_simAI_workload_generator.py AIOB_simAI_workload_generator.py	475:484 (1%) 620:629 (1%)	view
10	x 2	workload_generator workload_generator	AIOB_simAI_workload_generator.py AIOB_simAI_workload_generator.py	295:305 (1%) 329:339 (1%)	view
9	x 2	workload_generator workload_generator	generate_megatron_workload.py generate_megatron_workload.py	49:57 (2%) 59:67 (2%)	view
9	x 2	workload_generator workload_generator	AIOB_simAI_workload_generator.py AIOB_simAI_workload_generator.py	703:711 (1%) 718:726 (1%)	view
9	x 2	workload_generator workload_generator	AIOB_simAI_workload_generator.py AIOB_simAI_workload_generator.py	285:293 (1%) 622:631 (1%)	view
9	x 2	workload_generator workload_generator	AIOB_simAI_workload_generator.py AIOB_simAI_workload_generator.py	616:624 (1%) 718:726 (1%)	view
9	x 2	log_analyzer workload_generator	ds_comm_log_analyzer.py analysis_pytorch_trace.py	39:47 (8%) 37:45 (22%)	view
9	x 2	workload_generator workload_generator	generate_deepspeed_stage1_2_workload.py generate_deepspeed_stage3_workload.py	56:64 (4%) 78:86 (3%)	view
9	x 2	workload_generator workload_generator	generate_deepspeed_stage1_2_workload.py generate_deepspeed_stage3_workload.py	124:132 (4%) 148:158 (3%)	view
9	x 2	workload_generator workload_generator	generate_deepspeed_stage3_workload.py generate_megatron_workload.py	315:324 (3%) 438:447 (2%)	view
8	x 2	workload_generator workload_generator	AIOB_simAI_workload_generator.py AIOB_simAI_workload_generator.py	753:760 (<1%) 769:776 (<1%)	view
8	x 2	workload_generator workload_generator	generate_megatron_workload.py generate_megatron_workload.py	232:240 (2%) 301:308 (2%)	view
8	x 2	workload_generator workload_generator	generate_megatron_workload.py generate_megatron_workload.py	196:203 (2%) 234:241 (2%)	view
8	x 2	workload_generator workload_generator	generate_deepspeed_stage1_2_workload.py generate_deepspeed_stage3_workload.py	83:90 (3%) 126:133 (3%)	view
8	x 2	workload_generator workload_generator	AIOB_simAI_workload_generator.py AIOB_simAI_workload_generator.py	425:432 (<1%) 664:671 (<1%)	view
8	x 2	workload_generator workload_generator	AIOB_simAI_workload_generator.py AIOB_simAI_workload_generator.py	313:320 (<1%) 348:355 (<1%)	view
8	x 2	workload_generator workload_generator	generate_megatron_workload.py generate_megatron_workload.py	146:153 (2%) 234:241 (2%)	view
8	x 2	workload_generator workload_generator	AIOB_simAI_workload_generator.py AIOB_simAI_workload_generator.py	519:526 (<1%) 535:542 (<1%)	view
8	x 2	workload_generator workload_generator	AIOB_simAI_workload_generator.py AIOB_simAI_workload_generator.py	285:292 (<1%) 477:484 (<1%)	view
8	x 2	workload_generator workload_generator	generate_megatron_workload.py generate_megatron_workload.py	75:83 (2%) 108:116 (2%)	view
8	x 2	workload_generator workload_generator	generate_deepspeed_stage1_2_workload.py generate_deepspeed_stage1_2_workload.py	83:90 (3%) 115:122 (3%)	view
8	x 2	workload_generator workload_generator	AIOB_simAI_workload_generator.py AIOB_simAI_workload_generator.py	313:320 (<1%) 492:499 (<1%)	view
8	x 2	workload_generator workload_generator	AIOB_simAI_workload_generator.py AIOB_simAI_workload_generator.py	535:542 (<1%) 753:760 (<1%)	view
8	x 2	ROOT ROOT	workload_applyer.py workload_applyer.py	298:305 (2%) 321:328 (2%)	view
8	x 2	workload_generator workload_generator	generate_deepspeed_stage1_2_workload.py generate_deepspeed_stage3_workload.py	115:122 (3%) 139:146 (3%)	view
8	x 2	workload_generator workload_generator	AIOB_simAI_workload_generator.py AIOB_simAI_workload_generator.py	404:411 (<1%) 440:447 (<1%)	view
8	x 2	workload_generator workload_generator	AIOB_simAI_workload_generator.py AIOB_simAI_workload_generator.py	285:292 (<1%) 709:716 (<1%)	view
8	x 2	workload_generator workload_generator	AIOB_simAI_workload_generator.py AIOB_simAI_workload_generator.py	450:457 (<1%) 689:697 (<1%)	view
8	x 2	workload_generator workload_generator	generate_deepspeed_stage1_2_workload.py generate_deepspeed_stage3_workload.py	83:90 (3%) 139:146 (3%)	view
8	x 2	workload_generator workload_generator	generate_megatron_workload.py generate_megatron_workload.py	287:294 (2%) 302:309 (2%)	view
8	x 2	workload_generator workload_generator	AIOB_simAI_workload_generator.py AIOB_simAI_workload_generator.py	519:526 (<1%) 769:776 (<1%)	view