aws / sagemaker-training-toolkit

Intro

For duplication, we look at places in code where there are 6 or more lines of code that are exactly the same.
Before duplication is calculated, the code is cleaned to remove empty lines, comments, and frequently duplicated constructs such as imports.
You should aim at having as little as possible (<5%) of duplicated code as high-level of duplication can lead to maintenance difficulties, poor factoring, and logical contradictions.

Learn more...

Duplication Overall

13% duplication:

2,863 cleaned lines of cleaned code (without empty lines, comments, and frequently duplicated constructs such as imports)
386 duplicated lines

41 duplicates

dependency graphs: 2D graph | 3D graph | 3D graph (with duplicates)...

Duplication per Extension

Duplication per Component (primary)

Longest Duplicates

The list of 41 longest duplicates.

See data for all 41 duplicates...

Size	#	Folders	Files	Lines	Code
19	x 2	src/sagemaker_training src/sagemaker_training	mpi.py smdataparallel.py	507:531 (5%) 417:441 (6%)	view
15	x 2	src/sagemaker_training src/sagemaker_training	mpi.py smdataparallel.py	430:446 (4%) 341:357 (5%)	view
13	x 2	src/sagemaker_training src/sagemaker_training	recordio.py recordio.py	116:128 (11%) 155:167 (11%)	view
12	x 2	src/sagemaker_training src/sagemaker_training	mpi.py smdataparallel.py	220:231 (3%) 72:83 (4%)	view
12	x 2	src/sagemaker_training src/sagemaker_training	mpi.py smdataparallel.py	324:335 (3%) 171:182 (4%)	view
11	x 2	src/sagemaker_training src/sagemaker_training	mpi.py smdataparallel.py	409:420 (3%) 318:330 (3%)	view
10	x 2	src/sagemaker_training src/sagemaker_training	mpi.py smdataparallel.py	448:488 (3%) 360:401 (3%)	view
9	x 2	src/sagemaker_training src/sagemaker_training	mpi.py smdataparallel.py	338:346 (2%) 186:194 (3%)	view
9	x 2	src/sagemaker_training src/sagemaker_training	mpi.py smdataparallel.py	264:276 (2%) 114:126 (3%)	view
9	x 2	src/sagemaker_training src/sagemaker_training	mpi.py smdataparallel.py	311:319 (2%) 162:170 (3%)	view
8	x 2	src/sagemaker_training src/sagemaker_training	runner.py runner.py	94:101 (6%) 129:136 (6%)	view
8	x 2	src/sagemaker_training src/sagemaker_training	smdataparallel.py torch_distributed.py	72:79 (2%) 39:46 (8%)	view
8	x 2	src/sagemaker_training src/sagemaker_training	mpi.py torch_distributed.py	220:227 (2%) 39:46 (8%)	view
8	x 2	src/sagemaker_training src/sagemaker_training	runner.py runner.py	104:111 (6%) 141:148 (6%)	view
7	x 2	src/sagemaker_training src/sagemaker_training	mpi.py smdataparallel.py	32:40 (2%) 29:37 (2%)	view
7	x 2	src/sagemaker_training src/sagemaker_training	mpi.py pytorch_xla.py	220:226 (2%) 37:43 (6%)	view
7	x 2	src/sagemaker_training src/sagemaker_training	pytorch_xla.py torch_distributed.py	37:43 (6%) 39:45 (7%)	view
7	x 2	src/sagemaker_training src/sagemaker_training	process.py smdataparallel.py	425:431 (2%) 324:330 (2%)	view
7	x 2	src/sagemaker_training src/sagemaker_training	mpi.py process.py	414:420 (2%) 425:431 (2%)	view
7	x 2	src/sagemaker_training src/sagemaker_training	environment.py environment.py	377:383 (1%) 395:401 (1%)	view
7	x 2	src/sagemaker_training src/sagemaker_training	pytorch_xla.py smdataparallel.py	37:43 (6%) 72:78 (2%)	view
7	x 2	src/sagemaker_training src/sagemaker_training	smdataparallel.py torch_distributed.py	326:332 (2%) 156:162 (7%)	view
6	x 2	src/sagemaker_training src/sagemaker_training	process.py process.py	425:430 (2%) 433:438 (2%)	view
6	x 2	src/sagemaker_training src/sagemaker_training	mpi.py smdataparallel.py	414:419 (1%) 332:337 (2%)	view
6	x 2	src/sagemaker_training src/sagemaker_training	runner.py runner.py	142:147 (5%) 151:156 (5%)	view
6	x 2	src/sagemaker_training src/sagemaker_training	smdataparallel.py smdataparallel.py	324:329 (2%) 332:337 (2%)	view
6	x 2	src/sagemaker_training src/sagemaker_training	process.py smdataparallel.py	433:438 (2%) 332:337 (2%)	view
6	x 2	src/sagemaker_training src/sagemaker_training	runner.py runner.py	94:99 (5%) 114:119 (5%)	view
6	x 2	src/sagemaker_training src/sagemaker_training	process.py smdataparallel.py	425:430 (2%) 332:337 (2%)	view
6	x 2	src/sagemaker_training src/sagemaker_training	mpi.py smdataparallel.py	303:308 (1%) 154:159 (2%)	view
6	x 2	src/sagemaker_training src/sagemaker_training	mpi.py process.py	422:427 (1%) 425:430 (2%)	view
6	x 2	src/sagemaker_training src/sagemaker_training	torch_distributed.py torch_distributed.py	154:159 (6%) 162:167 (6%)	view
6	x 2	src/sagemaker_training src/sagemaker_training	process.py smdataparallel.py	433:438 (2%) 324:329 (2%)	view
6	x 2	src/sagemaker_training src/sagemaker_training	runner.py runner.py	105:110 (5%) 151:156 (5%)	view
6	x 2	src/sagemaker_training src/sagemaker_training	mpi.py process.py	422:427 (1%) 433:438 (2%)	view
6	x 2	src/sagemaker_training src/sagemaker_training	runner.py runner.py	114:119 (5%) 129:134 (5%)	view
6	x 2	src/sagemaker_training src/sagemaker_training	mpi.py smdataparallel.py	422:427 (1%) 324:329 (2%)	view
6	x 2	src/sagemaker_training src/sagemaker_training	mpi.py process.py	414:419 (1%) 433:438 (2%)	view
6	x 2	src/sagemaker_training src/sagemaker_training	mpi.py smdataparallel.py	422:427 (1%) 332:337 (2%)	view
6	x 2	src/sagemaker_training src/sagemaker_training	smdataparallel.py torch_distributed.py	26:31 (2%) 22:27 (6%)	view
6	x 2	src/sagemaker_training src/sagemaker_training	mpi.py mpi.py	414:419 (1%) 422:427 (1%)	view