alibaba / Tora - Duplication

Intro

For duplication, we look at places in code where there are 6 or more lines of code that are exactly the same.
Before duplication is calculated, the code is cleaned to remove empty lines, comments, and frequently duplicated constructs such as imports.
You should aim at having as little as possible (<5%) of duplicated code as high-level of duplication can lead to maintenance difficulties, poor factoring, and logical contradictions.

Learn more...

Duplication Overall

35% duplication:

30,982 cleaned lines of cleaned code (without empty lines, comments, and frequently duplicated constructs such as imports)
11,047 duplicated lines

964 duplicates

dependency graphs: 2D graph | 3D graph | 3D graph (with duplicates)...

Duplication per Extension

Duplication per Component (primary)

Duplication Between Components (50+ lines)

Download: SVG DOT (open online Graphviz editor)

From Component --> To Component	Duplicated Lines	File Pairs	Details
modules (4%) --> sat (3%)	1188	18 file pairs	details...
diffusers-version (22%) --> sat (2%)	925	7 file pairs	details...

Open 2D force graph... Open 3D force graph...

Show more details on duplication between components...

Longest Duplicates

The list of 50 longest duplicates.

See data for all 964 duplicates...

Size	#	Folders	Files	Lines	Code
426	x 2	sat/sgm/modules sat/vae_modules	attention.py attention.py	11:572 (100%) 11:572 (100%)	view
244	x 2	sat/sgm/models sat/vae_modules	autoencoder.py autoencoder.py	210:484 (55%) 225:499 (47%)	view
194	x 2	sat/sgm/modules/autoencoding/vqvae sat/sgm/modules/autoencoding/vqvae	movq_dec_3d.py movq_dec_3d_dev.py	239:495 (52%) 272:535 (51%)	view
176	x 2	sat/sgm/modules/autoencoding/vqvae sat/sgm/modules/autoencoding/vqvae	movq_dec_3d.py movq_dec_3d_dev.py	10:233 (47%) 15:257 (46%)	view
149	x 2	diffusers-version/tora sat/utils	traj_utils.py flow_utils.py	94:324 (84%) 7:237 (100%)	view
140	x 2	sat/sgm sat/vae_modules	util.py utils.py	32:251 (57%) 25:243 (54%)	view
119	x 2	diffusers-version/tora diffusers-version/tora	i2v_pipeline.py t2v_pipeline.py	158:343 (21%) 145:329 (24%)	view
108	x 2	diffusers-version/tora sat/sgm/modules	traj_module.py traj_module.py	25:171 (43%) 7:153 (47%)	view
99	x 2	diffusers-version/tora diffusers-version/tora	i2v_pipeline.py t2v_pipeline.py	485:605 (18%) 420:537 (20%)	view
66	x 2	sat/sgm/modules sat/vae_modules	cp_enc_dec.py cp_enc_dec.py	14:138 (68%) 26:150 (9%)	view
65	x 2	sat/sgm sat/vae_modules	util.py utils.py	264:359 (26%) 256:351 (25%)	view
60	x 2	sat/configs/tora/model sat/configs/tora/model	cogvideox_5b_tora.yaml cogvideox_5b_tora_i2v.yaml	55:123 (43%) 59:127 (41%)	view
57	x 2	sat/sgm/modules/autoencoding/vqvae sat/sgm/modules/autoencoding/vqvae	movq_dec_3d.py movq_dec_3d_dev.py	379:454 (15%) 260:335 (15%)	view
57	x 2	sat/sgm/modules/autoencoding/vqvae sat/sgm/modules/autoencoding/vqvae	movq_dec_3d_dev.py movq_dec_3d_dev.py	260:335 (15%) 419:494 (15%)	view
57	x 2	modules/SwissArmyTransformer/sat/model modules/SwissArmyTransformer/sat/model	base_model.py base_model.py	212:269 (16%) 337:394 (16%)	view
55	x 2	diffusers-version/tora sat/sgm/modules	traj_module.py traj_module.py	281:348 (22%) 232:297 (24%)	view
54	x 2	sat/sgm/models sat/vae_modules	autoencoder.py autoencoder.py	72:140 (12%) 88:156 (10%)	view
54	x 2	diffusers-version/tora diffusers-version/tora	i2v_pipeline.py t2v_pipeline.py	31:140 (9%) 29:141 (11%)	view
52	x 2	sat/sgm/modules/autoencoding/vqvae sat/sgm/modules/autoencoding/vqvae	movq_dec_3d.py movq_dec_3d.py	239:296 (13%) 391:454 (13%)	view
52	x 2	sat/sgm/modules/autoencoding/vqvae sat/sgm/modules/autoencoding/vqvae	movq_dec_3d.py movq_dec_3d_dev.py	239:296 (13%) 431:494 (13%)	view
51	x 2	sat/sgm/models sat/vae_modules	autoencoder.py autoencoder.py	146:206 (11%) 162:222 (9%)	view
51	x 2	modules/SwissArmyTransformer/sat/model/official modules/SwissArmyTransformer/sat/model/official	chatglm2_model.py chatglm4_model.py	17:82 (46%) 17:82 (46%)	view
50	x 2	modules/SwissArmyTransformer/sat/data_utils sat/sgm	webds.py webds.py	20:86 (19%) 17:90 (19%)	view
49	x 2	sat/sgm/modules sat/vae_modules	ema.py ema.py	5:82 (100%) 5:82 (100%)	view
48	x 2	modules/SwissArmyTransformer/sat/model/official modules/SwissArmyTransformer/sat/model/official	chatglm2_model.py chatglm3_model.py	17:79 (44%) 17:79 (44%)	view
48	x 2	modules/SwissArmyTransformer/sat/model/official modules/SwissArmyTransformer/sat/model/official	chatglm3_model.py chatglm4_model.py	17:79 (44%) 17:79 (44%)	view
45	x 2	modules/SwissArmyTransformer/sat/model/official modules/SwissArmyTransformer/sat/model/official	chatglm2_model.py chatglm4_model.py	92:141 (41%) 92:141 (41%)	view
45	x 2	diffusers-version/tora diffusers-version/tora	i2v_pipeline.py t2v_pipeline.py	842:900 (8%) 758:816 (9%)	view
44	x 2	modules/SwissArmyTransformer/sat/model/official modules/SwissArmyTransformer/sat/model/official	chatglm3_model.py chatglm4_model.py	92:140 (40%) 93:141 (40%)	view
44	x 2	modules/SwissArmyTransformer/sat/model/official modules/SwissArmyTransformer/sat/model/official	chatglm2_model.py chatglm3_model.py	93:141 (40%) 92:140 (40%)	view
39	x 2	sat/sgm/modules/diffusionmodules sat/sgm/modules/diffusionmodules	model.py model.py	306:346 (7%) 467:507 (7%)	view
38	x 2	sat/sgm/modules/autoencoding/losses sat/sgm/modules/autoencoding/losses	video_loss.py video_loss.py	349:396 (8%) 444:491 (8%)	view
38	x 2	modules/SwissArmyTransfo...enization/cogview/vqvae modules/SwissArmyTransfo...enization/cogview/vqvae	vqvae_diffusion.py vqvae_diffusion.py	219:259 (6%) 355:395 (6%)	view
37	x 2	modules/SwissArmyTransfo...enization/cogview/vqvae modules/SwissArmyTransfo...enization/cogview/vqvae	vqvae_diffusion.py vqvae_diffusion.py	252:292 (5%) 608:648 (5%)	view
36	x 2	sat/sgm/models sat/vae_modules	autoencoder.py autoencoder.py	501:544 (8%) 526:569 (6%)	view
36	x 2	sat/sgm/modules/diffusionmodules sat/vae_modules	util.py utils.py	127:177 (17%) 354:404 (14%)	view
34	x 2	sat/configs/tora sat/configs/tora	train_dense.yaml train_sparse.yaml	22:58 (53%) 22:58 (53%)	view
34	x 2	sat/sgm/modules/autoencoding/vqvae sat/sgm/modules/diffusionmodules	vqvae_blocks.py model.py	72:111 (11%) 95:134 (6%)	view
34	x 2	sat/sgm/modules/autoencoding/vqvae sat/sgm/modules/diffusionmodules	movq_modules.py model.py	78:118 (12%) 53:93 (6%)	view
33	x 2	sat sat	app.py sample_video.py	1069:1109 (3%) 334:373 (10%)	view
32	x 2	diffusers-version/tora diffusers-version/tora	i2v_pipeline.py t2v_pipeline.py	734:780 (5%) 661:707 (6%)	view
32	x 2	sat/sgm/modules/autoencoding/losses sat/sgm/modules/autoencoding	video_loss.py magvit2_pytorch.py	151:189 (6%) 550:588 (2%)	view
30	x 2	sat/configs/tora/model sat/configs/tora/model	cogvideox_5b_tora.yaml cogvideox_5b_tora_i2v.yaml	4:35 (21%) 7:38 (20%)	view
30	x 2	sat/sgm/modules/autoencoding/losses sat/sgm/modules/autoencoding/losses	video_loss.py video_loss.py	312:347 (6%) 407:442 (6%)	view
29	x 2	sat sat/utils	app.py flow_utils.py	634:666 (2%) 149:180 (19%)	view
29	x 2	diffusers-version/tora sat	traj_utils.py app.py	236:267 (16%) 634:666 (2%)	view
29	x 2	sat/sgm/modules/distributions sat/vae_modules	distributions.py regularizers.py	40:71 (45%) 26:57 (37%)	view
28	x 2	diffusers-version/tora sat/sgm/modules	traj_module.py traj_module.py	222:249 (11%) 173:200 (12%)	view
28	x 2	diffusers-version/tora sat/utils	traj_utils.py traj_utils.py	21:91 (15%) 4:74 (100%)	view
28	x 2	sat/sgm/modules/autoencoding sat/sgm/modules/autoencoding	temporal_ae.py temporal_ae.py	138:171 (10%) 204:237 (10%)	view

Duplicated Units

The list of top 9 duplicated units.

See data for all 9 unit duplicates...

Size	#	Folders	Files	Lines	Code
49	x 2	sat/sgm/models sat/vae_modules	autoencoder.py autoencoder.py	0:0 0:0	view
29	x 3	modules/SwissArmyTransformer/sat/model/official modules/SwissArmyTransformer/sat/model/official modules/SwissArmyTransformer/sat/model/official	chatglm2_model.py chatglm3_model.py chatglm4_model.py	0:0 0:0 0:0	view
29	x 2	sat/utils diffusers-version/tora	flow_utils.py traj_utils.py	0:0 0:0	view
24	x 2	sat/sgm/modules diffusers-version/tora	traj_module.py traj_module.py	0:0 0:0	view
22	x 2	sat/sgm sat/vae_modules	util.py utils.py	0:0 0:0	view
28	x 2	sat/sgm/modules/autoencoding/vqvae sat/sgm/modules/autoencoding/vqvae	movq_dec_3d.py movq_dec_3d_dev.py	0:0 0:0	view
20	x 2	sat/sgm/modules sat/vae_modules	attention.py attention.py	0:0 0:0	view
16	x 2	sat/sgm/modules/distributions sat/vae_modules	distributions.py regularizers.py	0:0 0:0	view
15	x 2	sat/sgm/modules sat/vae_modules	ema.py ema.py	0:0 0:0	view