Places in code with 6 or more lines that are exactly the same.
Intro
- For duplication, we look at places in code where there are 6 or more lines of code that are exactly the same.
- Before duplication is calculated, the code is cleaned to remove empty lines, comments, and frequently duplicated constructs such as imports.
- You should aim at having as little as possible (<5%) of duplicated code as high-level of duplication can lead to maintenance difficulties, poor factoring, and logical contradictions.
- To learn more about duplications and techniques for eliminating duplication, Sokrates recommends the following resources:
- Avoid Repetition, MartinFlower.com (IEEE Software article)
- Refactoring duplicated code, sourcemaking.com
- Beck Design Rules, MartinFowler.com
- DRY (Don't Repeat Yourself) Principle, Wikipedia
- The Cut-and-Paste Programming Software Development Anti-Pattern, sourcemaking.com
- Code Smells / Duplicated Code, Jeff Atwood, codinghorror.com:
- "Duplicated code is the bane of software development. Stamp out duplication whenever possible. You should always be on the lookout for more subtle cases of near-duplication, too. Don't Repeat Yourself!"
- Reducing Coupling, MartinFlower.com (IEEE Software article):
- "There are several ways to describe coupling, but it boils down to this: If changing one module in a program requires changing another module, then coupling exists. ... Duplication always implies coupling, because changing one piece of duplicate code implies changing the other."
Duplication Overall
- 67% duplication:
- 33,761 cleaned lines of cleaned code (without empty lines, comments, and frequently duplicated constructs such as imports)
- 22,817 duplicated lines
- 7,491 duplicates
Duplication per Extension
Duplication per Component (primary)
Duplication Between Components (50+ lines)
Open 3D force graph...
Show more details on duplication between components...
From Component
--> To Component
Duplicated
Lines
File Pairs
Details
sdlf-team (157%)
--> sdlf-utils (20%)
8255
58 file pairs
details...
sdlf-stageA (367%)
--> sdlf-utils (13%)
5747
76 file pairs
details...
sdlf-stageB (343%)
--> sdlf-utils (13%)
5641
114 file pairs
details...
sdlf-foundations (26%)
--> sdlf-utils (5%)
2298
97 file pairs
details...
sdlf-cicd (52%)
--> sdlf-utils (4%)
1943
59 file pairs
details...
sdlf-pipeline (221%)
--> sdlf-utils (3%)
1455
15 file pairs
details...
sdlf-datalakeLibrary (33%)
--> sdlf-utils (3%)
1248
14 file pairs
details...
sdlf-stageA (62%)
--> sdlf-stageB (59%)
971
13 file pairs
details...
sdlf-cicd (23%)
--> sdlf-team (16%)
865
11 file pairs
details...
sdlf-dataset (229%)
--> sdlf-utils (1%)
633
9 file pairs
details...
sdlf-cicd (10%)
--> sdlf-foundations (4%)
406
14 file pairs
details...
sdlf-foundations (4%)
--> sdlf-team (7%)
390
9 file pairs
details...
sdlf-foundations (4%)
--> sdlf-stageA (24%)
378
9 file pairs
details...
sdlf-foundations (4%)
--> sdlf-stageB (22%)
372
8 file pairs
details...
sdlf-stageA (10%)
--> sdlf-team (3%)
168
2 file pairs
details...
sdlf-pipeline (25%)
--> sdlf-team (3%)
168
3 file pairs
details...
sdlf-stageB (8%)
--> sdlf-team (2%)
142
2 file pairs
details...
sdlf-cicd (2%)
--> sdlf-stageA (5%)
89
2 file pairs
details...
sdlf-cicd (2%)
--> sdlf-stageB (5%)
89
2 file pairs
details...
sdlf-cicd (1%)
--> sdlf-pipeline (9%)
65
3 file pairs
details...
| From Component --> To Component |
Duplicated Lines |
File Pairs | Details |
|---|---|---|---|
| sdlf-team (157%) --> sdlf-utils (20%) |
8255 | 58 file pairs | details... |
| sdlf-stageA (367%) --> sdlf-utils (13%) |
5747 | 76 file pairs | details... |
| sdlf-stageB (343%) --> sdlf-utils (13%) |
5641 | 114 file pairs | details... |
| sdlf-foundations (26%) --> sdlf-utils (5%) |
2298 | 97 file pairs | details... |
| sdlf-cicd (52%) --> sdlf-utils (4%) |
1943 | 59 file pairs | details... |
| sdlf-pipeline (221%) --> sdlf-utils (3%) |
1455 | 15 file pairs | details... |
| sdlf-datalakeLibrary (33%) --> sdlf-utils (3%) |
1248 | 14 file pairs | details... |
| sdlf-stageA (62%) --> sdlf-stageB (59%) |
971 | 13 file pairs | details... |
| sdlf-cicd (23%) --> sdlf-team (16%) |
865 | 11 file pairs | details... |
| sdlf-dataset (229%) --> sdlf-utils (1%) |
633 | 9 file pairs | details... |
| sdlf-cicd (10%) --> sdlf-foundations (4%) |
406 | 14 file pairs | details... |
| sdlf-foundations (4%) --> sdlf-team (7%) |
390 | 9 file pairs | details... |
| sdlf-foundations (4%) --> sdlf-stageA (24%) |
378 | 9 file pairs | details... |
| sdlf-foundations (4%) --> sdlf-stageB (22%) |
372 | 8 file pairs | details... |
| sdlf-stageA (10%) --> sdlf-team (3%) |
168 | 2 file pairs | details... |
| sdlf-pipeline (25%) --> sdlf-team (3%) |
168 | 3 file pairs | details... |
| sdlf-stageB (8%) --> sdlf-team (2%) |
142 | 2 file pairs | details... |
| sdlf-cicd (2%) --> sdlf-stageA (5%) |
89 | 2 file pairs | details... |
| sdlf-cicd (2%) --> sdlf-stageB (5%) |
89 | 2 file pairs | details... |
| sdlf-cicd (1%) --> sdlf-pipeline (9%) |
65 | 3 file pairs | details... |
Longest Duplicates
The list of 20 longest duplicates.
| Size | # | Folders | Files | Lines | Code |
|---|---|---|---|---|---|
| 1307 | x 2 | sdlf-utils/ingestion-exa...sdlf-team/nested-stackssdlf-utils/pipeline-exam...sdlf-team/nested-stacks |  template-iam.yamltemplate-iam.yaml |
1:1322 (100%)1:1322 (100%) | view |
| 947 | x 2 | sdlf-team/nested-stackssdlf-utils/ingestion-exa...sdlf-team/nested-stacks | template-cicd.yamltemplate-cicd.yaml |
1:979 (100%)1:979 (100%) | view |
| 536 | x 2 | sdlf-team/nested-stackssdlf-utils/pipeline-examples/topic-modelling | template-iam.yamltemplate-iam.yaml |
610:1153 (47%)643:1186 (45%) | view |
| 380 | x 2 | sdlf-utils/pipeline-exam...sdlf-team/nested-stackssdlf-utils/pipeline-examples/topic-modelling | template-iam.yamltemplate-iam.yaml |
936:1322 (29%)800:1186 (32%) | view |
| 380 | x 2 | sdlf-utils/ingestion-exa...sdlf-team/nested-stackssdlf-utils/pipeline-examples/topic-modelling | template-iam.yamltemplate-iam.yaml |
936:1322 (29%)800:1186 (32%) | view |
| 380 | x 2 | sdlf-team/nested-stackssdlf-utils/pipeline-exam...sdlf-team/nested-stacks | template-iam.yamltemplate-iam.yaml |
767:1153 (33%)936:1322 (29%) | view |
| 380 | x 2 | sdlf-team/nested-stackssdlf-utils/ingestion-exa...sdlf-team/nested-stacks | template-iam.yamltemplate-iam.yaml |
767:1153 (33%)936:1322 (29%) | view |
| 359 | x 2 | sdlf-team/nested-stackssdlf-utils/pipeline-examples/topic-modelling | template-iam.yamltemplate-iam.yaml |
248:609 (31%)280:641 (30%) | view |
| 319 | x 2 | sdlf-stageAsdlf-utils/pipeline-exam.../topic-modelling/stageA | template.yamltemplate.yaml |
102:441 (53%)101:440 (53%) | view |
| 246 | x 2 | sdlf-utils/pipeline-examples/manifests/stageAsdlf-utils/pipeline-exam.../topic-modelling/stageA | template.yamltemplate.yaml |
95:355 (27%)98:358 (41%) | view |
| 244 | x 2 | sdlf-stageAsdlf-utils/pipeline-examples/manifests/stageA | template.yamltemplate.yaml |
102:359 (41%)98:355 (27%) | view |
| 222 | x 2 | sdlf-datalakeLibrary/pyt...lake_library/interfacessdlf-utils/pipeline-exam...lake_library/interfaces |  dynamo_interface.pydynamo_interface.py |
47:312 (88%)58:323 (86%) | view |
| 212 | x 2 | sdlf-utils/ingestion-exa...-pipeline/nested-stackssdlf-utils/pipeline-exam...-pipeline/nested-stacks | template-statemachine.yamltemplate-statemachine.yaml |
1:218 (100%)1:218 (100%) | view |
| 198 | x 2 | sdlf-utils/ingestion-exa...sdlf-team/nested-stackssdlf-utils/pipeline-examples/topic-modelling | template-iam.yamltemplate-iam.yaml |
379:576 (15%)323:520 (16%) | view |
| 198 | x 2 | sdlf-team/nested-stackssdlf-utils/ingestion-exa...sdlf-team/nested-stacks | template-iam.yamltemplate-iam.yaml |
291:488 (17%)379:576 (15%) | view |
| 198 | x 2 | sdlf-team/nested-stackssdlf-utils/pipeline-exam...sdlf-team/nested-stacks | template-iam.yamltemplate-iam.yaml |
291:488 (17%)379:576 (15%) | view |
| 198 | x 2 | sdlf-utils/pipeline-exam...sdlf-team/nested-stackssdlf-utils/pipeline-examples/topic-modelling | template-iam.yamltemplate-iam.yaml |
379:576 (15%)323:520 (16%) | view |
| 181 | x 2 | sdlf-team/nested-stackssdlf-utils/pipeline-examples/topic-modelling | template-iam.yamltemplate-iam.yaml |
1:184 (15%)1:184 (15%) | view |
| 180 | x 2 | sdlf-utils/ingestion-exa...sqoop/sdlf-team/scriptssdlf-utils/pipeline-exam...ncies/sdlf-team/scripts | template-team-repos.yamltemplate-team-repos.yaml |
1:186 (100%)1:186 (100%) | view |
| 175 | x 2 | sdlf-utils/ingestion-examples/sqoop/sdlf-pipelinesdlf-utils/pipeline-exam...lf-engineering-pipeline | template.yamltemplate.yaml |
1:181 (100%)1:181 (92%) | view |
Duplicated Units
The list of top 20 duplicated units.
| Size | # | Folders | Files | Lines | Code |
|---|---|---|---|---|---|
| 125 | x 2 | sdlf-utils/pipeline-exam...orms/stage_a_transformssdlf-utils/pipeline-exam...orms/stage_a_transforms | light_transform_athena_ctas_sqoop.pylight_transform_athena_ctas.py |
0:0 0:0 | view |
| 64 | x 2 | sdlf-utils/pipeline-exam...lake_library/interfacessdlf-datalakeLibrary/pyt...lake_library/interfaces | dynamo_interface.pydynamo_interface.py |
0:0 0:0 | view |
| 63 | x 2 | sdlf-utils/pipeline-exam...lake_library/interfacessdlf-datalakeLibrary/pyt...lake_library/interfaces | dynamo_interface.pydynamo_interface.py |
0:0 0:0 | view |
| 63 | x 2 | sdlf-stageB/lambda/stage-b-routing/srcsdlf-utils/pipeline-exam...bda/stage-b-routing/src | lambda_function.pylambda_function.py |
0:0 0:0 | view |
| 46 | x 3 | sdlf-utils/pipeline-exam...-preupdate-metadata/srcsdlf-utils/pipeline-exam...-preupdate-metadata/srcsdlf-stageA/lambda/stage-a-preupdate-metadata/src | lambda_function.pylambda_function.pylambda_function.py |
0:0 0:0 0:0 | view |
| 36 | x 2 | sdlf-stageB/lambda/stage-b-crawl-data/srcsdlf-utils/pipeline-exam.../stage-b-crawl-data/src | lambda_function.pylambda_function.py |
0:0 0:0 | view |
| 26 | x 2 | sdlf-stageB/lambda/stage-b-redrive/srcsdlf-utils/pipeline-exam...bda/stage-b-redrive/src | lambda_function.pylambda_function.py |
0:0 0:0 | view |
| 28 | x 2 | sdlf-utils/data_lake_testing_pipeline/src/lambdassdlf-utils/pipeline-testing/src/lambdas | pickTest.pypickTest.py |
0:0 0:0 | view |
| 26 | x 2 | sdlf-utils/pipeline-exam...lake_library/interfacessdlf-datalakeLibrary/pyt...lake_library/interfaces | dynamo_interface.pydynamo_interface.py |
0:0 0:0 | view |
| 30 | x 2 | sdlf-utils/pipeline-exam...lake_library/interfacessdlf-datalakeLibrary/pyt...lake_library/interfaces | sqs_interface.pysqs_interface.py |
0:0 0:0 | view |
| 24 | x 2 | sdlf-utils/data_lake_testing_pipeline/src/lambdassdlf-utils/pipeline-testing/src/lambdas | nextTest.pynextTest.py |
0:0 0:0 | view |
| 23 | x 2 | sdlf-utils/pipeline-exam...postupdate-metadata/srcsdlf-utils/pipeline-exam...a/stage-b-check-job/src | lambda_function.pylambda_function.py |
0:0 0:0 | view |
| 19 | x 2 | sdlf-utils/pipeline-exam...bda/stage-a-redrive/srcsdlf-stageA/lambda/stage-a-redrive/src | lambda_function.pylambda_function.py |
0:0 0:0 | view |
| 24 | x 2 | sdlf-foundations/lambda/elasticsearch/srcsdlf-foundations/lambda/topic/src | lambda_function.pylambda_function.py |
0:0 0:0 | view |
| 17 | x 2 | sdlf-foundations/lambda/catalog-redrive/srcsdlf-foundations/lambda/routing-redrive/src | lambda_function.pylambda_function.py |
0:0 0:0 | view |
| 17 | x 2 | sdlf-utils/pipeline-exam...lake_library/interfacessdlf-datalakeLibrary/pyt...lake_library/interfaces | sqs_interface.pysqs_interface.py |
0:0 0:0 | view |
| 14 | x 2 | sdlf-utils/pipeline-exam...bda/stage-a-routing/srcsdlf-stageA/lambda/stage-a-routing/src | lambda_function.pylambda_function.py |
0:0 0:0 | view |
| 13 | x 2 | sdlf-foundations/scripts/deequ/resourcessdlf-utils/ingestion-exa...dms-replication/scripts | deequ-controller.pyDMSCDC_Controller.py |
0:0 0:0 | view |
| 13 | x 2 | sdlf-utils/pipeline-exam...lake_library/interfacessdlf-utils/ingestion-exa...lake_library/interfaces | states_interface.pystates_interface.py |
0:0 0:0 | view |
| 12 | x 2 | sdlf-foundations/scripts/deequ/resourcessdlf-foundations/scripts/deequ/resources |  deequ-profile-runner.scaladeequ-analysis-verification-runner.scala |
93:106 305:318 | view |