aws-solutions / discovering-hot-topics-using-machine-learning
Duplication

Places in code with 6 or more lines that are exactly the same.

Intro
  • For duplication, we look at places in code where there are 6 or more lines of code that are exactly the same.
  • Before duplication is calculated, the code is cleaned to remove empty lines, comments, and frequently duplicated constructs such as imports.
  • You should aim at having as little as possible (<5%) of duplicated code as high-level of duplication can lead to maintenance difficulties, poor factoring, and logical contradictions.
Learn more...
Duplication Overall
  • 20% duplication:
    • 6,742 cleaned lines of cleaned code (without empty lines, comments, and frequently duplicated constructs such as imports)
    • 1,392 duplicated lines
  • 747 duplicates
system20% (1,392 lines)
Duplication per Extension
ts24% (798 lines)
js28% (460 lines)
py7% (134 lines)
Duplication per Component (primary)
source/lib/visualization59% (404 lines)
source/lib/ingestion19% (135 lines)
source/lambda/quicksight-custom-resources10% (92 lines)
source/lambda/firehose-text-proxy29% (83 lines)
source/lambda/wf-detect-moderation-labels54% (82 lines)
source/lib/integration29% (65 lines)
source/lambda/wf-extract-text-in-image37% (57 lines)
source/lib/topic-analysis-workflow23% (53 lines)
source/lib/text-analysis-workflow10% (45 lines)
source/lambda/wf-analyze-text38% (37 lines)
source/lambda/wf-submit-topic-model23% (33 lines)
source/lambda/wf-detect-language46% (29 lines)
source/lambda/capture_news_feed7% (28 lines)
source/lib/storage12% (26 lines)
source/lib/quicksight-custom-resources17% (26 lines)
source/lib5% (21 lines)
source/lambda/ingestion-consumer58% (21 lines)
source/lambda/wf-check-topic-model31% (16 lines)
source/lambda/create-partition15% (16 lines)
source/lambda/wf-translate-text14% (16 lines)
source/lambda/ingestion-producer7% (16 lines)
source/lambda/storage-firehose-processor51% (15 lines)
source/lambda/integration51% (15 lines)
source/lambda/wf-publish-text-inference40% (15 lines)
source/lambda/wf_publish_topic_model7% (14 lines)
source/lib/awsnodejs-lambda-layer15% (13 lines)
source/lib/aspects29% (10 lines)
source/lambda/layers12% (9 lines)
source0% (0 lines)
source/lib/solution-helper0% (0 lines)
source/lambda/solution_helper0% (0 lines)
source/lambda/firehose_topic_proxy0% (0 lines)
source/lambda/ingestion-youtube0% (0 lines)
deployment/cdk-solution-helper0% (0 lines)
deployment0% (0 lines)

Duplication Between Components (50+ lines)

G source/lib/ingestion source/lib/ingestion source/lib/visualization source/lib/visualization source/lib/ingestion--source/lib/visualization 257 source/lib/integration source/lib/integration source/lib/ingestion--source/lib/integration 162 source/lib/storage source/lib/storage source/lib/ingestion--source/lib/storage 120 source/lib/quicksight-custom-resources source/lib/quicksight-custom-resources source/lib/ingestion--source/lib/quicksight-custom-resources 117 source/lib/integration--source/lib/visualization 225 source/lib/integration--source/lib/storage 79 source/lib/integration--source/lib/quicksight-custom-resources 85 source/lib/awsnodejs-lambda-layer source/lib/awsnodejs-lambda-layer source/lib/awsnodejs-lambda-layer--source/lib/ingestion 101 source/lib/awsnodejs-lambda-layer--source/lib/visualization 182 source/lib/awsnodejs-lambda-layer--source/lib/integration 69 source/lib source/lib source/lib--source/lib/ingestion 117 source/lib--source/lib/visualization 182 source/lib--source/lib/integration 69 source/lib/storage--source/lib/visualization 150 source/lib/quicksight-custom-resources--source/lib/visualization 150 source/lib/aspects source/lib/aspects source/lib/aspects--source/lib/ingestion 89 source/lib/aspects--source/lib/visualization 140 source/lib/aspects--source/lib/integration 60 source/lambda/wf-detect-moderation-labels source/lambda/wf-detect-moderation-labels source/lambda/wf-extract-text-in-image source/lambda/wf-extract-text-in-image source/lambda/wf-detect-moderation-labels--source/lambda/wf-extract-text-in-image 112 source/lambda/wf-submit-topic-model source/lambda/wf-submit-topic-model source/lambda/wf-detect-moderation-labels--source/lambda/wf-submit-topic-model 66 source/lambda/wf-analyze-text source/lambda/wf-analyze-text source/lambda/wf-analyze-text--source/lambda/wf-detect-moderation-labels 62 source/lambda/wf-detect-language source/lambda/wf-detect-language source/lambda/wf-analyze-text--source/lambda/wf-detect-language 56

Download: SVG DOT (open online Graphviz editor)

Open 3D force graph...

Show more details on duplication between components...
Longest Duplicates
The list of 20 longest duplicates.
See data for all 747 duplicates...
Size#FoldersFilesLinesCode
23 x 2 source/lambda/wf-detect-moderation-labels/util
source/lambda/wf-extract-text-in-image/util
stream-analyzer.js
stream-analyzer.js
 14:63 (100%)
14:58 (100%)
 view
22 x 2 source/lib/visualization
source/lib/visualization
keyphrase-table-construct.ts
sentiment-table-construct.ts
 21:45 (47%)
20:43 (44%)
 view
22 x 2 source/lib/visualization
source/lib/visualization
entity-table-construct.ts
keyphrase-table-construct.ts
 21:44 (44%)
21:45 (47%)
 view
22 x 2 source/lib/visualization
source/lib/visualization
entity-table-construct.ts
sentiment-table-construct.ts
 21:44 (44%)
20:43 (44%)
 view
19 x 2 source/lib/visualization
source/lib/visualization
sentiment-table-construct.ts
text-in-img-sentiment-table-construct.ts
 20:40 (38%)
21:42 (38%)
 view
19 x 2 source/lib/visualization
source/lib/visualization
keyphrase-table-construct.ts
text-in-image-entities-table-construc...
 21:42 (41%)
21:41 (38%)
 view
19 x 2 source/lib/visualization
source/lib/visualization
text-in-image-entities-table-construc...
text-in-img-keyphrase-table-construct.ts
 21:41 (38%)
21:41 (41%)
 view
19 x 2 source/lib/visualization
source/lib/visualization
entity-table-construct.ts
text-in-image-entities-table-construc...
 21:41 (38%)
21:41 (38%)
 view
19 x 2 source/lib/visualization
source/lib/visualization
keyphrase-table-construct.ts
text-in-img-sentiment-table-construct.ts
 21:42 (41%)
21:42 (38%)
 view
19 x 2 source/lib/visualization
source/lib/visualization
entity-table-construct.ts
text-in-img-keyphrase-table-construct.ts
 21:41 (38%)
21:41 (41%)
 view
19 x 2 source/lib/visualization
source/lib/visualization
text-in-img-keyphrase-table-construct.ts
text-in-img-sentiment-table-construct.ts
 21:41 (41%)
21:42 (38%)
 view
19 x 2 source/lib/visualization
source/lib/visualization
sentiment-table-construct.ts
text-in-img-keyphrase-table-construct.ts
 20:40 (38%)
21:41 (41%)
 view
19 x 2 source/lib/visualization
source/lib/visualization
entity-table-construct.ts
text-in-img-sentiment-table-construct.ts
 21:41 (38%)
21:42 (38%)
 view
19 x 2 source/lib/visualization
source/lib/visualization
keyphrase-table-construct.ts
text-in-img-keyphrase-table-construct.ts
 21:42 (41%)
21:41 (41%)
 view
19 x 2 source/lib/visualization
source/lib/visualization
text-in-image-entities-table-construc...
text-in-img-sentiment-table-construct.ts
 21:41 (38%)
21:42 (38%)
 view
19 x 2 source/lib/visualization
source/lib/visualization
sentiment-table-construct.ts
text-in-image-entities-table-construc...
 20:40 (38%)
21:41 (38%)
 view
16 x 2 source/lambda/firehose-text-proxy
source/lambda/wf-analyze-text
jest.config.js
jest.config.js
 14:29 (100%)
14:29 (100%)
 view
16 x 2 source/lambda/create-partition
source/lambda/wf-check-topic-model
jest.config.js
jest.config.js
 14:29 (100%)
14:29 (100%)
 view
16 x 2 source/lib/visualization
source/lib/visualization
moderation-labels-table-construct.ts
text-in-image-entities-table-construc...
 21:38 (43%)
21:38 (32%)
 view
16 x 2 source/lambda/create-partition
source/lambda/wf-translate-text
jest.config.js
jest.config.js
 14:29 (100%)
14:29 (100%)
 view
Duplicated Units
The list of top 3 duplicated units.
See data for all 3 unit duplicates...
Size#FoldersFilesLinesCode
14 x 2 source/lambda/wf-detect-moderation-labels/util
source/lambda/wf-extract-text-in-image/util
stream-analyzer.js
stream-analyzer.js
 26:41 
26:41 
 view
10 x 2 source/lambda/wf-detect-moderation-labels/util
source/lambda/wf-extract-text-in-image/util
stream-analyzer.js
stream-analyzer.js
 48:60 
43:55 
 view
6 x 4 source/lambda/wf-detect-moderation-labels
source/lambda/wf-extract-text-in-image
source/lambda/wf-detect-language
source/lambda/wf-analyze-text
index.js
index.js
index.js
index.js
 108:114 
119:125 
76:82 
136:142 
 view