openai / jukebox - Duplication

Intro

For duplication, we look at places in code where there are 6 or more lines of code that are exactly the same.
Before duplication is calculated, the code is cleaned to remove empty lines, comments, and frequently duplicated constructs such as imports.
You should aim at having as little as possible (<5%) of duplicated code as high-level of duplication can lead to maintenance difficulties, poor factoring, and logical contradictions.

Learn more...

Duplication Overall

11% duplication:

16,610 cleaned lines of cleaned code (without empty lines, comments, and frequently duplicated constructs such as imports)
1,839 duplicated lines

492 duplicates

dependency graphs: 2D graph | 3D graph | 3D graph (with duplicates)...

Duplication per Extension

Duplication per Component (primary)

Longest Duplicates

The list of 50 longest duplicates.

See data for all 492 duplicates...

Size	#	Folders	Files	Lines	Code
28	x 2	apex/csrc apex/csrc	type_shim.h type_shim.h	19:50 (34%) 45:76 (34%)	view
26	x 2	apex/csrc apex/csrc	layer_norm_cuda_kernel.cu layer_norm_cuda_kernel.cu	327:352 (3%) 371:396 (3%)	view
20	x 2	jukebox/utils jukebox/utils	fp16.py fp16.py	193:214 (8%) 279:300 (8%)	view
20	x 2	apex/apex/amp apex/apex/amp	_process_optimizer.py _process_optimizer.py	137:158 (6%) 214:235 (6%)	view
18	x 2	jukebox jukebox	Interacting_with_Jukebox.ipynb Interacting_with_Jukebox.ipynb	162:179 (1%) 431:448 (1%)	view
16	x 2	jukebox/utils jukebox/utils	fp16.py fp16.py	129:144 (6%) 230:245 (6%)	view
15	x 2	jukebox/prior jukebox/prior	autoregressive.py autoregressive.py	233:249 (4%) 343:359 (4%)	view
13	x 2	tensorboardX/tensorboardX/proto tensorboardX/tensorboardX/proto	proto layout.proto plugin_pr_curve.proto	1:18 (16%) 1:18 (72%)	view
13	x 2	tensorboardX/tensorboardX/proto tensorboardX/tensorboardX/proto	proto layout.proto plugin_text.proto	1:18 (16%) 1:18 (61%)	view
13	x 2	tensorboardX/tensorboardX/proto tensorboardX/tensorboardX/proto	proto plugin_pr_curve.proto plugin_text.proto	1:18 (72%) 1:18 (61%)	view
13	x 2	jukebox/prior jukebox/prior	autoregressive.py autoregressive.py	204:218 (3%) 266:280 (3%)	view
12	x 2	apex/csrc apex/csrc	welford.cu welford.cu	512:527 (1%) 729:744 (1%)	view
12	x 2	apex/csrc apex/csrc	layer_norm_cuda_kernel.cu layer_norm_cuda_kernel.cu	673:684 (1%) 772:783 (1%)	view
12	x 2	apex/csrc apex/csrc	welford.cu welford.cu	874:887 (1%) 1013:1027 (1%)	view
11	x 2	apex/csrc apex/csrc	multi_tensor_axpby_kernel.cu multi_tensor_lamb_stage_1.cu	1:14 (10%) 1:14 (8%)	view
11	x 2	apex/csrc apex/csrc	multi_tensor_axpby_kernel.cu multi_tensor_lamb_stage_2.cu	1:14 (10%) 1:14 (11%)	view
11	x 2	apex/csrc apex/csrc	multi_tensor_l2norm_kernel.cu multi_tensor_lamb_stage_2.cu	1:14 (7%) 1:14 (11%)	view
11	x 2	apex/csrc apex/csrc	welford.cu welford.cu	1177:1187 (<1%) 1325:1335 (<1%)	view
11	x 2	tensorboardX/tensorboardX/proto tensorboardX/tensorboardX/proto	proto api.proto plugin_hparams.proto	1:14 (3%) 1:14 (18%)	view
11	x 2	jukebox/utils jukebox/utils	fp16.py fp16.py	165:185 (4%) 249:269 (4%)	view
11	x 2	apex/csrc apex/csrc	multi_tensor_l2norm_kernel.cu multi_tensor_lamb_stage_1.cu	1:14 (7%) 1:14 (8%)	view
11	x 2	apex/csrc apex/csrc	multi_tensor_lamb_stage_1.cu multi_tensor_lamb_stage_2.cu	1:14 (8%) 1:14 (11%)	view
11	x 2	apex/csrc apex/csrc	layer_norm_cuda_kernel.cu layer_norm_cuda_kernel.cu	101:111 (1%) 196:206 (1%)	view
11	x 2	apex/csrc apex/csrc	multi_tensor_axpby_kernel.cu multi_tensor_l2norm_kernel.cu	1:14 (10%) 1:14 (7%)	view
11	x 2	jukebox jukebox	hparams.py hparams.py	245:255 (2%) 275:285 (2%)	view
10	x 2	apex/csrc apex/csrc	layer_norm_cuda_kernel.cu layer_norm_cuda_kernel.cu	116:125 (1%) 211:220 (1%)	view
10	x 2	tensorboardX/tensorboardX tensorboardX/tensorboardX	visdom_writer.py visdom_writer.py	281:290 (6%) 314:323 (6%)	view
10	x 2	tensorboardX/tensorboardX/proto tensorboardX/tensorboardX/proto	proto layout.proto plugin_hparams.proto	3:14 (12%) 3:14 (16%)	view
10	x 2	tensorboardX/tensorboardX/proto tensorboardX/tensorboardX/proto	proto api.proto plugin_text.proto	3:14 (2%) 3:14 (47%)	view
10	x 2	tensorboardX/tensorboardX/proto tensorboardX/tensorboardX/proto	proto api.proto plugin_pr_curve.proto	3:14 (2%) 3:14 (55%)	view
10	x 2	tensorboardX/tensorboardX/proto tensorboardX/tensorboardX/proto	proto plugin_hparams.proto plugin_pr_curve.proto	3:14 (16%) 3:14 (55%)	view
10	x 2	tensorboardX/tensorboardX/proto tensorboardX/tensorboardX/proto	proto api.proto layout.proto	3:14 (2%) 3:14 (12%)	view
10	x 2	tensorboardX/tensorboardX tensorboardX/tensorboardX	summary.py summary.py	438:447 (2%) 455:464 (2%)	view
10	x 2	apex/csrc apex/csrc	welford.cu welford.cu	436:448 (<1%) 659:671 (<1%)	view
10	x 2	tensorboardX/tensorboardX/proto tensorboardX/tensorboardX/proto	proto plugin_hparams.proto plugin_text.proto	3:14 (16%) 3:14 (47%)	view
10	x 2	apex/apex/fp16_utils apex/apex/optimizers	fp16_optimizer.py fp16_optimizer.py	626:642 (4%) 193:209 (7%)	view
9	x 2	apex/csrc apex/csrc	layer_norm_cuda.cpp layer_norm_cuda.cpp	102:113 (5%) 163:174 (5%)	view
9	x 2	apex/apex/fp16_utils apex/apex/optimizers	fp16_optimizer.py fp16_optimizer.py	13:114 (4%) 4:48 (6%)	view
9	x 2	jukebox jukebox	lyricdict.py lyricdict.py	573:581 (1%) 584:592 (1%)	view
9	x 2	apex/csrc apex/csrc	multi_tensor_axpby_kernel.cu multi_tensor_l2norm_kernel.cu	26:36 (8%) 27:37 (6%)	view
9	x 2	apex/csrc apex/csrc	welford.cu welford.cu	1158:1168 (<1%) 1304:1314 (<1%)	view
9	x 2	apex/csrc apex/csrc	welford.cu welford.cu	1258:1266 (<1%) 1325:1333 (<1%)	view
9	x 2	apex/csrc apex/csrc	fused_adam_cuda_kernel.cu fused_adam_cuda_kernel.cu	106:114 (6%) 124:132 (6%)	view
9	x 2	apex/csrc apex/csrc	welford.cu welford.cu	897:905 (<1%) 1039:1047 (<1%)	view
9	x 2	apex/csrc apex/csrc	welford.cu welford.cu	1177:1185 (<1%) 1258:1266 (<1%)	view
9	x 2	apex/csrc apex/csrc	welford.cu welford.cu	899:907 (<1%) 1327:1335 (<1%)	view
9	x 2	apex/csrc apex/csrc	welford.cu welford.cu	899:907 (<1%) 1179:1187 (<1%)	view
9	x 2	apex/csrc apex/csrc	welford.cu welford.cu	598:608 (<1%) 795:805 (<1%)	view
8	x 2	jukebox jukebox	Interacting_with_Jukebox.ipynb Interacting_with_Jukebox.ipynb	338:345 (<1%) 951:958 (<1%)	view
8	x 2	apex/csrc apex/csrc	welford.cu welford.cu	325:332 (<1%) 638:645 (<1%)	view