aws-samples / amazon-sagemaker-workshop-analytics-machine-learning
Duplication

Places in code with 6 or more lines that are exactly the same.

Intro
  • For duplication, we look at places in code where there are 6 or more lines of code that are exactly the same.
  • Before duplication is calculated, the code is cleaned to remove empty lines, comments, and frequently duplicated constructs such as imports.
  • You should aim at having as little as possible (<5%) of duplicated code as high-level of duplication can lead to maintenance difficulties, poor factoring, and logical contradictions.
Learn more...
Duplication Overall
  • 48% duplication:
    • 2,086 cleaned lines of cleaned code (without empty lines, comments, and frequently duplicated constructs such as imports)
    • 1,010 duplicated lines
  • 56 duplicates
system48% (1,010 lines)
Duplication per Extension
py61% (959 lines)
cfg100% (33 lines)
yml3% (18 lines)
Duplication per Component (primary)
5. MLOps SageMaker Project/sagemaker-workshop-preprocess-seedcode-v1/pipelines/preprocess62% (151 lines)
1. Amazon SageMaker Processing79% (130 lines)
5. MLOps SageMaker Project/sagemaker-workshop-preprocess-seedcode-v1/pipelines99% (120 lines)
5. MLOps SageMaker Project/sagemaker-workshop-build-seedcode-v1/pipelines99% (120 lines)
5. MLOps SageMaker Project/sagemaker-workshop-deploy-seedcode-v1/pipelines99% (120 lines)
glue92% (94 lines)
5. MLOps SageMaker Project/sagemaker-workshop-preprocess-seedcode-v1100% (63 lines)
5. MLOps SageMaker Project/sagemaker-workshop-deploy-seedcode-v1100% (63 lines)
5. MLOps SageMaker Project/sagemaker-workshop-build-seedcode-v1100% (63 lines)
5. MLOps SageMaker Project/sagemaker-workshop-deploy-seedcode-v1/pipelines/deploy13% (40 lines)
5. MLOps SageMaker Project/sagemaker-workshop-build-seedcode-v1/pipelines/train11% (28 lines)
5. MLOps SageMaker Project/cloud_formation3% (18 lines)

Duplication Between Components (50+ lines)

G 1. Amazon SageMaker Processing 1. Amazon SageMaker Processing 5. MLOps SageMaker Project/sagemaker-workshop-preprocess-seedcode-v1/pipelines/preprocess 5. MLOps SageMaker Project/sagemaker-workshop-preprocess-seedcode-v1/pipelines/preprocess 1. Amazon SageMaker Processing--5. MLOps SageMaker Project/sagemaker-workshop-preprocess-seedcode-v1/pipelines/preprocess 260 5. MLOps SageMaker Project/sagemaker-workshop-build-seedcode-v1/pipelines 5. MLOps SageMaker Project/sagemaker-workshop-build-seedcode-v1/pipelines 5. MLOps SageMaker Project/sagemaker-workshop-deploy-seedcode-v1/pipelines 5. MLOps SageMaker Project/sagemaker-workshop-deploy-seedcode-v1/pipelines 5. MLOps SageMaker Project/sagemaker-workshop-build-seedcode-v1/pipelines--5. MLOps SageMaker Project/sagemaker-workshop-deploy-seedcode-v1/pipelines 240 5. MLOps SageMaker Project/sagemaker-workshop-preprocess-seedcode-v1/pipelines 5. MLOps SageMaker Project/sagemaker-workshop-preprocess-seedcode-v1/pipelines 5. MLOps SageMaker Project/sagemaker-workshop-build-seedcode-v1/pipelines--5. MLOps SageMaker Project/sagemaker-workshop-preprocess-seedcode-v1/pipelines 240 5. MLOps SageMaker Project/sagemaker-workshop-deploy-seedcode-v1/pipelines--5. MLOps SageMaker Project/sagemaker-workshop-preprocess-seedcode-v1/pipelines 240 5. MLOps SageMaker Project/sagemaker-workshop-build-seedcode-v1 5. MLOps SageMaker Project/sagemaker-workshop-build-seedcode-v1 5. MLOps SageMaker Project/sagemaker-workshop-preprocess-seedcode-v1 5. MLOps SageMaker Project/sagemaker-workshop-preprocess-seedcode-v1 5. MLOps SageMaker Project/sagemaker-workshop-build-seedcode-v1--5. MLOps SageMaker Project/sagemaker-workshop-preprocess-seedcode-v1 126 5. MLOps SageMaker Project/sagemaker-workshop-deploy-seedcode-v1 5. MLOps SageMaker Project/sagemaker-workshop-deploy-seedcode-v1 5. MLOps SageMaker Project/sagemaker-workshop-build-seedcode-v1--5. MLOps SageMaker Project/sagemaker-workshop-deploy-seedcode-v1 126 5. MLOps SageMaker Project/sagemaker-workshop-deploy-seedcode-v1--5. MLOps SageMaker Project/sagemaker-workshop-preprocess-seedcode-v1 126 5. MLOps SageMaker Project/sagemaker-workshop-build-seedcode-v1/pipelines/train 5. MLOps SageMaker Project/sagemaker-workshop-build-seedcode-v1/pipelines/train 5. MLOps SageMaker Project/sagemaker-workshop-deploy-seedcode-v1/pipelines/deploy 5. MLOps SageMaker Project/sagemaker-workshop-deploy-seedcode-v1/pipelines/deploy 5. MLOps SageMaker Project/sagemaker-workshop-build-seedcode-v1/pipelines/train--5. MLOps SageMaker Project/sagemaker-workshop-deploy-seedcode-v1/pipelines/deploy 52

Download: SVG DOT (open online Graphviz editor)

Open 3D force graph...

Show more details on duplication between components...
Longest Duplicates
The list of 20 longest duplicates.
See data for all 56 duplicates...
Size#FoldersFilesLinesCode
93 x 2 1. Amazon SageMaker Processing
5. MLOps SageM...v1/pipelines/preprocess
preprocess.py
preprocess.py
 57:175 (57%)
36:154 (66%)
 view
66 x 2 5. MLOps SageM...d-seedcode-v1/pipelines
5. MLOps SageM...y-seedcode-v1/pipelines
run_pipeline.py
run_pipeline.py
 23:104 (100%)
23:104 (100%)
 view
66 x 2 5. MLOps SageM...y-seedcode-v1/pipelines
5. MLOps SageM...s-seedcode-v1/pipelines
run_pipeline.py
run_pipeline.py
 23:104 (100%)
23:104 (100%)
 view
66 x 2 5. MLOps SageM...d-seedcode-v1/pipelines
5. MLOps SageM...s-seedcode-v1/pipelines
run_pipeline.py
run_pipeline.py
 23:104 (100%)
23:104 (100%)
 view
52 x 2 5. MLOps SageM...kshop-build-seedcode-v1
5. MLOps SageM...-preprocess-seedcode-v1
setup.py
setup.py
 5:60 (100%)
5:60 (100%)
 view
52 x 2 5. MLOps SageM...shop-deploy-seedcode-v1
5. MLOps SageM...-preprocess-seedcode-v1
setup.py
setup.py
 5:60 (100%)
5:60 (100%)
 view
52 x 2 5. MLOps SageM...kshop-build-seedcode-v1
5. MLOps SageM...shop-deploy-seedcode-v1
setup.py
setup.py
 5:60 (100%)
5:60 (100%)
 view
41 x 2 5. MLOps SageM...d-seedcode-v1/pipelines
5. MLOps SageM...y-seedcode-v1/pipelines
get_pipeline_definition.py
get_pipeline_definition.py
 22:71 (100%)
22:71 (100%)
 view
41 x 2 5. MLOps SageM...d-seedcode-v1/pipelines
5. MLOps SageM...s-seedcode-v1/pipelines
get_pipeline_definition.py
get_pipeline_definition.py
 22:71 (100%)
22:71 (100%)
 view
41 x 2 5. MLOps SageM...y-seedcode-v1/pipelines
5. MLOps SageM...s-seedcode-v1/pipelines
get_pipeline_definition.py
get_pipeline_definition.py
 22:71 (100%)
22:71 (100%)
 view
23 x 2 1. Amazon SageMaker Processing
5. MLOps SageM...v1/pipelines/preprocess
preprocess.py
preprocess.py
 214:253 (14%)
174:213 (16%)
 view
20 x 2 glue
glue
greentaxi_etl.py
yellowtaxi_etl.py
 31:54 (39%)
31:54 (39%)
 view
19 x 2 5. MLOps SageM...ode-v1/pipelines/deploy
5. MLOps SageM...v1/pipelines/preprocess
pipeline.py
pipeline.py
 41:95 (15%)
32:86 (19%)
 view
16 x 2 5. MLOps SageM...code-v1/pipelines/train
5. MLOps SageM...v1/pipelines/preprocess
pipeline.py
pipeline.py
 43:84 (8%)
27:67 (16%)
 view
14 x 2 5. MLOps SageM...code-v1/pipelines/train
5. MLOps SageM...ode-v1/pipelines/deploy
pipeline.py
pipeline.py
 49:84 (7%)
41:76 (11%)
 view
13 x 2 glue
glue
greentaxi_etl.py
yellowtaxi_etl.py
 60:76 (25%)
60:76 (25%)
 view
12 x 2 5. MLOps SageM...code-v1/pipelines/train
5. MLOps SageM...ode-v1/pipelines/deploy
pipeline.py
pipeline.py
 99:120 (6%)
91:112 (10%)
 view
11 x 2 5. MLOps SageM...shop-deploy-seedcode-v1
5. MLOps SageM...-preprocess-seedcode-v1
cfg
setup.cfg
setup.cfg
 1:14 (100%)
1:14 (100%)
 view
11 x 2 5. MLOps SageM...kshop-build-seedcode-v1
5. MLOps SageM...shop-deploy-seedcode-v1
cfg
setup.cfg
setup.cfg
 1:14 (100%)
1:14 (100%)
 view
11 x 2 5. MLOps SageM...kshop-build-seedcode-v1
5. MLOps SageM...-preprocess-seedcode-v1
cfg
setup.cfg
setup.cfg
 1:14 (100%)
1:14 (100%)
 view
Duplicated Units
The list of top 11 duplicated units.
See data for all 11 unit duplicates...
Size#FoldersFilesLinesCode
68 x 3 5. MLOps SageM...s-seedcode-v1/pipelines
5. MLOps SageM...d-seedcode-v1/pipelines
5. MLOps SageM...y-seedcode-v1/pipelines
run_pipeline.py
run_pipeline.py
run_pipeline.py
 0:0 
0:0 
0:0 
 view
41 x 3 5. MLOps SageM...s-seedcode-v1/pipelines
5. MLOps SageM...d-seedcode-v1/pipelines
5. MLOps SageM...y-seedcode-v1/pipelines
get_pipeline_definition.py
get_pipeline_definition.py
get_pipeline_definition.py
 0:0 
0:0 
0:0 
 view
38 x 2 5. MLOps SageM...v1/pipelines/preprocess
1. Amazon SageMaker Processing
preprocess.py
preprocess.py
 0:0 
0:0 
 view
32 x 2 5. MLOps SageM...v1/pipelines/preprocess
1. Amazon SageMaker Processing
preprocess.py
preprocess.py
 0:0 
0:0 
 view
14 x 2 5. MLOps SageM...v1/pipelines/preprocess
1. Amazon SageMaker Processing
preprocess.py
preprocess.py
 0:0 
0:0 
 view
16 x 3 5. MLOps SageM...v1/pipelines/preprocess
5. MLOps SageM...code-v1/pipelines/train
5. MLOps SageM...ode-v1/pipelines/deploy
pipeline.py
pipeline.py
pipeline.py
 0:0 
0:0 
0:0 
 view
7 x 2 5. MLOps SageM...code-v1/pipelines/train
5. MLOps SageM...ode-v1/pipelines/deploy
pipeline.py
pipeline.py
 0:0 
0:0 
 view
9 x 2 5. MLOps SageM...v1/pipelines/preprocess
1. Amazon SageMaker Processing
preprocess.py
preprocess.py
 0:0 
0:0 
 view
10 x 3 5. MLOps SageM...v1/pipelines/preprocess
5. MLOps SageM...code-v1/pipelines/train
5. MLOps SageM...ode-v1/pipelines/deploy
pipeline.py
pipeline.py
pipeline.py
 0:0 
0:0 
0:0 
 view
11 x 3 5. MLOps SageM...s-seedcode-v1/pipelines
5. MLOps SageM...d-seedcode-v1/pipelines
5. MLOps SageM...y-seedcode-v1/pipelines
_utils.py
_utils.py
_utils.py
 0:0 
0:0 
0:0 
 view
8 x 2 5. MLOps SageM...v1/pipelines/preprocess
1. Amazon SageMaker Processing
preprocess.py
preprocess.py
 0:0 
0:0 
 view