elastic / cloudbeat - Duplication

Intro

For duplication, we look at places in code where there are 6 or more lines of code that are exactly the same.
Before duplication is calculated, the code is cleaned to remove empty lines, comments, and frequently duplicated constructs such as imports.
You should aim at having as little as possible (<5%) of duplicated code as high-level of duplication can lead to maintenance difficulties, poor factoring, and logical contradictions.

Learn more...

Duplication Overall

20% duplication:

55,880 cleaned lines of cleaned code (without empty lines, comments, and frequently duplicated constructs such as imports)
11,532 duplicated lines

4,428 duplicates

dependency graphs: 2D graph | 3D graph | 3D graph (with duplicates)...

Duplication per Extension

Duplication per Component (primary)

Longest Duplicates

The list of 50 longest duplicates.

See data for all 4,428 duplicates...

Size	#	Folders	Files	Lines	Code
216	x 2	deploy/asset-inventory-arm deploy/azure	generate_dev_template.py generate_dev_template.py	18:305 (100%) 17:304 (100%)	view
54	x 2	deploy/asset-inventory-cloudformation deploy/cloudformation	gomain.go gomain.go	125:204 (44%) 129:208 (42%)	view
48	x 2	deploy/asset-inventory-cloudformation deploy/cloudformation	config.go config.go	41:115 (80%) 50:124 (69%)	view
44	x 2	deploy/asset-inventory-cloudformation deploy/cloudformation	gomain.go gomain.go	63:119 (36%) 71:127 (34%)	view
31	x 2	internal/dataprovider/providers/cloud internal/dataprovider/providers/k8s	mock_option.go mock_option.go	25:81 (100%) 25:81 (100%)	view
28	x 2	deploy/test-environments/cdr deploy/test-environments/cis	variables.tf variables.tf	71:104 (33%) 32:64 (53%)	view
26	x 2	deploy/test-environments/elk-stack deploy/test-environments/modules/aws/cloudtrail	variables.tf variables.tf	62:94 (33%) 31:62 (50%)	view
26	x 2	internal/flavors/benchmark/builder internal/uniqueness	mock_manager.go mock_manager.go	66:112 (54%) 129:175 (32%)	view
26	x 2	deploy/test-environments/cdr deploy/test-environments/modules/aws/cloudtrail	variables.tf variables.tf	73:104 (31%) 31:62 (50%)	view
26	x 2	deploy/test-environments/cdr deploy/test-environments/elk-stack	variables.tf variables.tf	73:104 (31%) 62:94 (33%)	view
26	x 2	deploy/test-environments/cis deploy/test-environments/elk-stack	variables.tf variables.tf	34:64 (50%) 62:94 (33%)	view
26	x 2	deploy/test-environments/cis deploy/test-environments/modules/aws/cloudtrail	variables.tf variables.tf	34:64 (50%) 31:62 (50%)	view
24	x 2	security-policies/bundle...e/cis_gcp/rules/cis_1_5 security-policies/bundle...e/cis_gcp/rules/cis_1_6	data.yaml data.yaml	81:104 (21%) 67:90 (25%)	view
20	x 2	security-policies/bundle...e/cis_aws/rules/cis_4_3 security-policies/bundle...e/cis_aws/rules/cis_4_4	data.yaml data.yaml	10:45 (23%) 10:45 (23%)	view
20	x 2	internal/resources/providers/azurelib/inventory internal/resources/providers/azurelib/inventory	postgresql_provider.go postgresql_provider.go	173:197 (13%) 207:231 (13%)	view
20	x 2	security-policies/bundle..._azure/rules/cis_2_1_19 security-policies/bundle..._azure/rules/cis_2_1_20	data.yaml data.yaml	53:72 (27%) 48:67 (28%)	view
19	x 2	security-policies/bundle.../cis_aws/rules/cis_4_12 security-policies/bundle...e/cis_aws/rules/cis_4_8	data.yaml data.yaml	11:41 (22%) 10:40 (22%)	view
19	x 2	security-policies/bundle.../cis_aws/rules/cis_4_11 security-policies/bundle.../cis_aws/rules/cis_4_14	data.yaml data.yaml	12:42 (21%) 11:44 (22%)	view
19	x 2	security-policies/bundle.../cis_aws/rules/cis_4_10 security-policies/bundle...e/cis_aws/rules/cis_4_8	data.yaml data.yaml	11:41 (22%) 10:40 (22%)	view
19	x 2	security-policies/bundle.../cis_aws/rules/cis_4_14 security-policies/bundle...e/cis_aws/rules/cis_4_8	data.yaml data.yaml	11:44 (22%) 10:40 (22%)	view
19	x 2	security-policies/bundle...e/cis_aws/rules/cis_4_7 security-policies/bundle...e/cis_aws/rules/cis_4_9	data.yaml data.yaml	10:40 (22%) 10:40 (22%)	view
19	x 2	security-policies/bundle.../cis_aws/rules/cis_4_13 security-policies/bundle...e/cis_aws/rules/cis_4_8	data.yaml data.yaml	11:44 (22%) 10:40 (22%)	view
19	x 2	security-policies/bundle.../cis_aws/rules/cis_4_10 security-policies/bundle.../cis_aws/rules/cis_4_14	data.yaml data.yaml	11:41 (22%) 11:44 (22%)	view
19	x 2	security-policies/bundle.../cis_aws/rules/cis_4_14 security-policies/bundle...e/cis_aws/rules/cis_4_7	data.yaml data.yaml	11:44 (22%) 10:40 (22%)	view
19	x 2	security-policies/bundle.../cis_aws/rules/cis_4_11 security-policies/bundle...e/cis_aws/rules/cis_4_5	data.yaml data.yaml	12:42 (21%) 10:43 (22%)	view
19	x 2	security-policies/bundle.../cis_aws/rules/cis_4_11 security-policies/bundle...e/cis_aws/rules/cis_4_9	data.yaml data.yaml	12:42 (21%) 10:40 (22%)	view
19	x 2	security-policies/bundle...e/cis_aws/rules/cis_4_6 security-policies/bundle...e/cis_aws/rules/cis_4_9	data.yaml data.yaml	11:41 (22%) 10:40 (22%)	view
19	x 2	security-policies/bundle.../cis_aws/rules/cis_4_12 security-policies/bundle...e/cis_aws/rules/cis_4_7	data.yaml data.yaml	11:41 (22%) 10:40 (22%)	view
19	x 2	security-policies/bundle.../cis_aws/rules/cis_4_13 security-policies/bundle...e/cis_aws/rules/cis_4_7	data.yaml data.yaml	11:44 (22%) 10:40 (22%)	view
19	x 2	security-policies/bundle.../cis_aws/rules/cis_4_10 security-policies/bundle...e/cis_aws/rules/cis_4_9	data.yaml data.yaml	11:41 (22%) 10:40 (22%)	view
19	x 2	security-policies/bundle...e/cis_aws/rules/cis_4_5 security-policies/bundle...e/cis_aws/rules/cis_4_9	data.yaml data.yaml	10:43 (22%) 10:40 (22%)	view
19	x 2	security-policies/bundle.../cis_aws/rules/cis_4_10 security-policies/bundle...e/cis_aws/rules/cis_4_7	data.yaml data.yaml	11:41 (22%) 10:40 (22%)	view
19	x 2	security-policies/bundle.../cis_aws/rules/cis_4_11 security-policies/bundle...e/cis_aws/rules/cis_4_8	data.yaml data.yaml	12:42 (21%) 10:40 (22%)	view
19	x 2	security-policies/bundle.../cis_aws/rules/cis_4_14 security-policies/bundle...e/cis_aws/rules/cis_4_9	data.yaml data.yaml	11:44 (22%) 10:40 (22%)	view
19	x 2	security-policies/bundle.../cis_aws/rules/cis_4_13 security-policies/bundle...e/cis_aws/rules/cis_4_9	data.yaml data.yaml	11:44 (22%) 10:40 (22%)	view
19	x 2	security-policies/bundle.../cis_aws/rules/cis_4_12 security-policies/bundle...e/cis_aws/rules/cis_4_9	data.yaml data.yaml	11:41 (22%) 10:40 (22%)	view
19	x 2	security-policies/bundle.../cis_aws/rules/cis_4_13 security-policies/bundle.../cis_aws/rules/cis_4_14	data.yaml data.yaml	11:44 (22%) 11:44 (22%)	view
19	x 2	security-policies/bundle...e/cis_aws/rules/cis_4_5 security-policies/bundle...e/cis_aws/rules/cis_4_7	data.yaml data.yaml	10:43 (22%) 10:40 (22%)	view
19	x 2	security-policies/bundle.../cis_aws/rules/cis_4_12 security-policies/bundle...e/cis_aws/rules/cis_4_6	data.yaml data.yaml	11:41 (22%) 11:41 (22%)	view
19	x 2	security-policies/bundle...e/cis_aws/rules/cis_4_8 security-policies/bundle...e/cis_aws/rules/cis_4_9	data.yaml data.yaml	10:40 (22%) 10:40 (22%)	view
19	x 2	security-policies/bundle.../cis_aws/rules/cis_4_13 security-policies/bundle...e/cis_aws/rules/cis_4_6	data.yaml data.yaml	11:44 (22%) 11:41 (22%)	view
19	x 2	security-policies/bundle.../cis_aws/rules/cis_4_14 security-policies/bundle...e/cis_aws/rules/cis_4_6	data.yaml data.yaml	11:44 (22%) 11:41 (22%)	view
19	x 2	security-policies/bundle.../cis_aws/rules/cis_4_14 security-policies/bundle...e/cis_aws/rules/cis_4_5	data.yaml data.yaml	11:44 (22%) 10:43 (22%)	view
19	x 2	security-policies/bundle.../cis_aws/rules/cis_4_13 security-policies/bundle...e/cis_aws/rules/cis_4_5	data.yaml data.yaml	11:44 (22%) 10:43 (22%)	view
19	x 2	security-policies/bundle.../cis_aws/rules/cis_4_10 security-policies/bundle.../cis_aws/rules/cis_4_11	data.yaml data.yaml	11:41 (22%) 12:42 (21%)	view
19	x 2	security-policies/bundle.../cis_aws/rules/cis_4_12 security-policies/bundle.../cis_aws/rules/cis_4_14	data.yaml data.yaml	11:41 (22%) 11:44 (22%)	view
19	x 2	security-policies/bundle.../cis_aws/rules/cis_4_11 security-policies/bundle.../cis_aws/rules/cis_4_12	data.yaml data.yaml	12:42 (21%) 11:41 (22%)	view
19	x 2	security-policies/bundle...e/cis_aws/rules/cis_4_6 security-policies/bundle...e/cis_aws/rules/cis_4_8	data.yaml data.yaml	11:41 (22%) 10:40 (22%)	view
19	x 2	security-policies/bundle.../cis_aws/rules/cis_4_11 security-policies/bundle...e/cis_aws/rules/cis_4_6	data.yaml data.yaml	12:42 (21%) 11:41 (22%)	view
19	x 2	security-policies/bundle.../cis_aws/rules/cis_4_10 security-policies/bundle.../cis_aws/rules/cis_4_12	data.yaml data.yaml	11:41 (22%) 11:41 (22%)	view

Duplicated Units

The list of top 36 duplicated units.

See data for all 36 unit duplicates...

Size	#	Folders	Files	Lines	Code
46	x 2	deploy/azure deploy/asset-inventory-arm	generate_dev_template.py generate_dev_template.py	0:0 0:0	view
31	x 2	deploy/cloudformation deploy/asset-inventory-cloudformation	gomain.go gomain.go	133:170 129:166	view
30	x 2	deploy/cloudformation deploy/asset-inventory-cloudformation	gomain.go gomain.go	172:209 168:205	view
22	x 2	internal/inventory/azurefetcher internal/resources/providers/msgraph	mock_activedirectory_provider.go mock_provider_api.go	43:70 43:70	view
22	x 2	internal/inventory/azurefetcher internal/resources/providers/msgraph	mock_activedirectory_provider.go mock_provider_api.go	101:128 101:128	view
22	x 2	internal/inventory/azurefetcher internal/resources/providers/msgraph	mock_activedirectory_provider.go mock_provider_api.go	217:244 217:244	view
22	x 3	internal/inventory/azurefetcher internal/inventory/azurefetcher internal/resources/providers/azurelib	mock_account_provider.go mock_storage_provider.go mock_provider_api.go	43:70 574:601 2016:2043	view
22	x 2	internal/inventory/azurefetcher internal/resources/providers/azurelib	mock_storage_provider.go mock_provider_api.go	43:70 1308:1335	view
22	x 2	internal/inventory/azurefetcher internal/resources/providers/azurelib	mock_storage_provider.go mock_provider_api.go	515:542 1780:1807	view
22	x 9	internal/inventory/awsfetcher internal/inventory/awsfetcher internal/inventory/awsfetcher internal/inventory/awsfetcher internal/inventory/awsfetcher internal/inventory/awsfetcher internal/inventory/awsfetcher internal/resources/providers/awslib/lambda internal/resources/providers/awslib/elb	mock_network_describe_func.go mock_elb_describe_func.go mock_lambda_describe_func.go mock_v1_provider.go mock_v2_provider.go mock_lambda_provider.go mock_networking_provider.go mock_lambda.go mock_load_balancer_describer.go	44:71 44:71 44:71 44:71 44:71 104:131 44:71 44:71 46:73	view
22	x 14	internal/inventory/awsfetcher internal/inventory/awsfetcher internal/inventory/awsfetcher internal/inventory/awsfetcher internal/inventory/awsfetcher internal/resources/providers/aws_cis/logging internal/resources/providers/awslib/elb_v2 internal/resources/providers/awslib/sns internal/resources/providers/awslib/ec2 internal/resources/providers/awslib/rds ...	mock_s3_bucket_provider.go mock_rds_provider.go mock_iam_policy_provider.go mock_sns_provider.go mock_iam_user_provider.go mock_client.go mock_load_balancer_describer.go mock_sns.go mock_elastic_compute.go mock_rds.go ...	44:71 44:71 44:71 44:71 44:71 44:71 44:71 164:191 46:73 44:71 ...	view
22	x 2	internal/inventory/awsfetcher internal/vulnerability	mock_ec2_instances_provider.go mock_instances_provider.go	43:70 43:70	view
22	x 2	internal/inventory/awsfetcher internal/resources/providers/awslib/lambda	mock_lambda_provider.go mock_lambda.go	44:71 218:245	view
22	x 2	internal/dataprovider internal/resources/fetching	mock_elastic_common_data_provider.go mock_resource.go	38:65 85:112	view
22	x 3	internal/resources/providers/azurelib/inventory internal/resources/providers/azurelib/inventory internal/resources/providers/azurelib/inventory	mock_postgresql_provider_api.go mock_sql_provider_api.go mock_mysql_provider_api.go	42:69 42:69 42:69	view
21	x 2	deploy/cloudformation deploy/asset-inventory-cloudformation	config.go config.go	82:104 73:95	view
19	x 2	deploy/cloudformation deploy/asset-inventory-cloudformation	config.go config.go	55:80 46:71	view
15	x 2	internal/flavors/benchmark internal/flavors/benchmark	aws.go aws_org.go	86:105 220:240	view
15	x 2	deploy/cloudformation deploy/asset-inventory-cloudformation	gomain.go gomain.go	93:109 85:101	view
13	x 2	internal/resources/fetching/registry internal/resources/fetching	mock_registry.go mock_resource.go	43:60 142:159	view
11	x 3	internal/flavors internal/flavors/benchmark internal/resources/providers/gcplib/inventory	mock_repeater_func.go mock_strategy.go mock_service_api.go	38:53 107:122 42:57	view
11	x 2	internal/uniqueness internal/resources/fetching	mock_manager.go mock_condition.go	42:57 38:53	view
11	x 2	internal/uniqueness internal/vulnerability	mock_manager.go mock_runner_interface.go	87:102 45:60	view
11	x 3	internal/resources/fetching internal/resources/providers/awslib internal/processor/add_cluster_id	mock_condition.go mock_aws_resource.go mock_cluster_helper.go	83:98 38:53 38:53	view
9	x 2	internal/flavors/benchmark internal/flavors/benchmark	eks.go k8s.go	51:62 44:55	view
9	x 4	internal/resources/fetching/fetchers/gcp internal/resources/fetching/fetchers/gcp internal/resources/fetching/fetchers/gcp internal/resources/fetching/fetchers/gcp	monitoring_fetcher.go policies_fetcher.go log_sink_fetcher.go service_usage_fetcher.go	88:98 82:92 83:93 83:93	view
8	x 2	internal/flavors/benchmark internal/flavors/benchmark	aws.go aws_org.go	44:54 55:65	view
8	x 2	deploy/cloudformation deploy/asset-inventory-cloudformation	gomain.go gomain.go	47:57 39:49	view
7	x 2	internal/flavors/benchmark internal/flavors/assetinventory	mock_strategy.go mock_strategy.go	153:163 104:114	view
7	x 4	internal/flavors/benchmark/builder internal/dataprovider/providers/cloud internal/dataprovider/providers/k8s internal/resources/fetching/registry	mock_option.go mock_option.go mock_option.go mock_option.go	72:82 72:82 72:82 72:82	view
7	x 2	internal/flavors/benchmark/builder internal/uniqueness	mock_manager.go mock_manager.go	103:113 166:176	view
7	x 2	internal/resources/providers/gcplib/auth internal/resources/providers/azurelib/auth	mock_config_provider_api.go mock_config_provider_api.go	106:116 100:110	view
7	x 2	internal/resources/providers/gcplib/inventory internal/resources/providers/azurelib	mock_provider_initializer_api.go mock_provider_initializer_api.go	106:116 103:113	view
7	x 17	internal/resources/providers/aws_cis/logging internal/resources/providers/aws_cis/monitoring internal/resources/providers/awslib/ecr internal/resources/providers/awslib/elb_v2 internal/resources/providers/awslib/sns internal/resources/providers/awslib/ec2 internal/resources/providers/awslib/rds internal/resources/providers/awslib/cloudtrail internal/resources/providers/awslib/lambda internal/resources/providers/awslib/kms ...	mock_client.go mock_client.go mock_client.go mock_client.go mock_client.go mock_client.go mock_client.go mock_client.go mock_client.go mock_client.go ...	103:113 101:111 118:128 192:202 192:202 1376:1386 118:128 266:276 340:350 266:276 ...	view
7	x 3	internal/resources/providers/msgraph internal/resources/providers/azurelib internal/resources/providers/azurelib/governance	mock_provider_api.go mock_provider_api.go mock_provider_api.go	276:286 2133:2143 103:113	view
7	x 2	internal/resources/providers/awslib/elb_v2 internal/resources/providers/awslib/elb	mock_load_balancer_describer.go mock_load_balancer_describer.go	103:113 164:174	view