elastic / crawler - Duplication

Intro

For duplication, we look at places in code where there are 6 or more lines of code that are exactly the same.
Before duplication is calculated, the code is cleaned to remove empty lines, comments, and frequently duplicated constructs such as imports.
You should aim at having as little as possible (<5%) of duplicated code as high-level of duplication can lead to maintenance difficulties, poor factoring, and logical contradictions.

Learn more...

Duplication Overall

10% duplication:

14,118 cleaned lines of cleaned code (without empty lines, comments, and frequently duplicated constructs such as imports)
1,456 duplicated lines

159 duplicates

dependency graphs: 2D graph | 3D graph | 3D graph (with duplicates)...

Duplication per Extension

Duplication per Component (primary)

Duplication Between Components (50+ lines)

Download: SVG DOT (open online Graphviz editor)

From Component --> To Component	Duplicated Lines	File Pairs	Details
lib (<1%) --> spec (<1%)	64	7 file pairs	details...

Open 2D force graph... Open 3D force graph...

Show more details on duplication between components...

Longest Duplicates

The list of 50 longest duplicates.

See data for all 159 duplicates...

Size	#	Folders	Files	Lines	Code
40	x 2	lib/crawler/logging/handler lib/crawler/logging/handler	file.rb stdout.rb	23:69 (80%) 21:67 (81%)	view
18	x 2	spec/lib/crawler/cli spec/lib/crawler/cli	crawl_spec.rb validate_spec.rb	9:30 (28%) 9:30 (33%)	view
18	x 2	spec/lib/crawler/data/crawl_result spec/lib/crawler/data/crawl_result	sitemap_spec.rb sitemap_spec.rb	75:98 (9%) 207:230 (9%)	view
17	x 2	spec/lib/crawler/output_sink spec/lib/es	elasticsearch_spec.rb client_spec.rb	523:539 (2%) 479:496 (3%)	view
16	x 2	spec/lib/crawler/output_sink spec/lib/crawler/output_sink	elasticsearch_spec.rb elasticsearch_spec.rb	250:266 (2%) 355:371 (2%)	view
15	x 2	spec/lib/crawler spec/lib/crawler	http_executor_spec.rb http_executor_spec.rb	375:389 (4%) 424:438 (4%)	view
14	x 2	lib/crawler lib/crawler	http_executor.rb http_executor.rb	296:309 (4%) 315:328 (4%)	view
14	x 2	spec/lib/crawler spec/lib/crawler	document_mapper_spec.rb document_mapper_spec.rb	74:89 (6%) 197:212 (6%)	view
13	x 2	spec/lib/crawler spec/lib/crawler	coordinator_spec.rb coordinator_spec.rb	196:208 (1%) 260:272 (1%)	view
13	x 2	spec/lib/crawler spec/lib/crawler	http_client_spec.rb http_client_spec.rb	341:355 (4%) 363:377 (4%)	view
12	x 2	lib/crawler lib/crawler	http_executor.rb http_executor.rb	317:328 (4%) 340:351 (4%)	view
12	x 2	spec/lib/crawler/data/crawl_result spec/lib/crawler/data	html_spec.rb crawl_result_spec.rb	14:26 (3%) 13:25 (16%)	view
12	x 2	spec/lib/crawler/output_sink spec/lib/crawler/output_sink	elasticsearch_spec.rb elasticsearch_spec.rb	11:22 (2%) 302:313 (2%)	view
12	x 2	spec/lib/crawler/output_sink spec/lib/crawler/output_sink	elasticsearch_spec.rb elasticsearch_spec.rb	278:289 (2%) 302:313 (2%)	view
12	x 2	spec/lib/crawler/output_sink spec/lib/crawler/output_sink	elasticsearch_spec.rb elasticsearch_spec.rb	11:22 (2%) 278:289 (2%)	view
12	x 2	lib/crawler lib/crawler	http_executor.rb http_executor.rb	298:309 (4%) 340:351 (4%)	view
11	x 2	spec/integration spec/integration	sitemap_xxe_spec.rb sitemap_xxe_spec.rb	50:62 (15%) 75:87 (15%)	view
11	x 2	spec/integration spec/integration	robots_txt_spec.rb robots_txt_spec.rb	82:94 (6%) 125:137 (6%)	view
11	x 2	spec/integration spec/integration	robots_txt_spec.rb robots_txt_spec.rb	82:94 (6%) 164:176 (6%)	view
11	x 2	spec/integration spec/integration	robots_txt_spec.rb robots_txt_spec.rb	164:176 (6%) 195:207 (6%)	view
11	x 2	spec/integration spec/integration	robots_txt_spec.rb robots_txt_spec.rb	82:94 (6%) 195:207 (6%)	view
11	x 2	spec/integration spec/integration	robots_txt_spec.rb robots_txt_spec.rb	125:137 (6%) 164:176 (6%)	view
11	x 2	spec/integration spec/integration	robots_txt_spec.rb robots_txt_spec.rb	125:137 (6%) 195:207 (6%)	view
11	x 2	spec/lib/crawler/output_sink spec/lib/crawler/output_sink	elasticsearch_spec.rb elasticsearch_spec.rb	395:407 (1%) 494:507 (1%)	view
11	x 2	spec/factories spec/factories	crawl_results.rb crawl_results.rb	87:98 (10%) 101:112 (10%)	view
10	x 2	spec/lib/crawler/output_sink spec/lib/crawler/output_sink	elasticsearch_spec.rb elasticsearch_spec.rb	511:520 (1%) 561:570 (1%)	view
10	x 2	spec/lib/crawler spec/lib/crawler	document_mapper_spec.rb document_mapper_spec.rb	95:105 (4%) 218:228 (4%)	view
10	x 2	spec/lib/crawler/cli spec/lib/crawler/cli	crawl_spec.rb schedule_spec.rb	79:90 (15%) 63:74 (20%)	view
10	x 2	spec/factories spec/factories	crawl_results.rb crawl_results.rb	44:54 (9%) 58:68 (9%)	view
9	x 2	spec/lib/crawler/output_sink spec/lib/crawler/output_sink	elasticsearch_spec.rb elasticsearch_spec.rb	254:262 (1%) 325:333 (1%)	view
9	x 2	spec/lib/crawler spec/lib/crawler	event_generator_spec.rb event_generator_spec.rb	97:105 (6%) 114:122 (6%)	view
9	x 2	spec/lib/crawler/output_sink spec/lib/crawler/output_sink	elasticsearch_spec.rb elasticsearch_spec.rb	325:333 (1%) 359:367 (1%)	view
9	x 2	spec/lib/crawler spec/lib/crawler	coordinator_spec.rb coordinator_spec.rb	257:266 (1%) 277:286 (1%)	view
9	x 2	spec/lib/crawler spec/lib/crawler	robots_txt_parser_spec.rb robots_txt_parser_spec.rb	22:32 (6%) 39:49 (6%)	view
9	x 2	spec/integration spec/integration	headers_spec.rb headers_spec.rb	64:73 (12%) 81:90 (12%)	view
9	x 2	spec/lib/crawler/output_sink spec/lib/crawler/output_sink	elasticsearch_spec.rb elasticsearch_spec.rb	230:238 (1%) 254:262 (1%)	view
9	x 2	spec/lib/crawler/output_sink spec/lib/crawler/output_sink	elasticsearch_spec.rb elasticsearch_spec.rb	230:238 (1%) 359:367 (1%)	view
9	x 2	spec/lib/crawler/data/crawl_result spec/lib/crawler/data/crawl_result	sitemap_spec.rb sitemap_spec.rb	50:61 (4%) 182:193 (4%)	view
9	x 2	spec/lib/crawler spec/lib/crawler	http_client_spec.rb http_client_spec.rb	160:168 (2%) 199:207 (2%)	view
9	x 2	lib/crawler/output_sink spec/lib/crawler/output_sink	elasticsearch.rb elasticsearch_spec.rb	128:136 (4%) 564:572 (1%)	view
9	x 2	spec/integration spec/integration	robots_txt_spec.rb robots_txt_spec.rb	32:42 (5%) 57:67 (5%)	view
9	x 2	spec/lib/crawler/output_sink spec/lib/crawler/output_sink	elasticsearch_spec.rb elasticsearch_spec.rb	230:238 (1%) 325:333 (1%)	view
8	x 2	spec/lib/crawler spec/lib/crawler	http_executor_spec.rb http_executor_spec.rb	233:241 (2%) 288:296 (2%)	view
8	x 2	spec/lib/crawler/api spec/lib/crawler	crawl_spec.rb output_sink_spec.rb	18:25 (4%) 37:44 (22%)	view
8	x 2	spec/lib/crawler spec/lib/crawler	coordinator_spec.rb coordinator_spec.rb	213:220 (1%) 230:237 (1%)	view
8	x 2	lib/crawler/output_sink lib/crawler/output_sink	elasticsearch.rb elasticsearch.rb	105:112 (3%) 127:134 (3%)	view
8	x 2	spec/lib/es spec/lib/es	client_spec.rb client_spec.rb	350:357 (1%) 375:382 (1%)	view
8	x 2	spec/lib/crawler/output_sink spec/lib/crawler/output_sink	elasticsearch_spec.rb elasticsearch_spec.rb	131:139 (1%) 164:172 (1%)	view
8	x 2	spec/lib/crawler/output_sink spec/lib/crawler/output_sink	elasticsearch_spec.rb elasticsearch_spec.rb	131:139 (1%) 147:155 (1%)	view
8	x 2	spec/lib/crawler spec/lib/crawler	coordinator_spec.rb coordinator_spec.rb	260:267 (1%) 386:393 (1%)	view

Duplicated Units

The list of top 2 duplicated units.

See data for all 2 unit duplicates...

Size	#	Folders	Files	Lines	Code
14	x 2	lib/crawler/logging/handler lib/crawler/logging/handler	file.rb stdout.rb	29:45 27:43	view
8	x 2	lib/crawler/logging/handler lib/crawler/logging/handler	file.rb stdout.rb	51:61 49:59	view