elastic / crawler
File Age & Freshness

File age measurements show the distribution of file ages (days since the first commit) and the file freshness (days since the latest commit).

Summary
File Change History Overall
File Age Distribution Overall
Days since first update
  • There are 168 files with 14,128 lines of code in files.
    • 109 files that are 366+ days old (9,615 lines of code)
    • 41 files that are 181-365 days old (3,135 lines of code)
    • 10 files that are 91-180 days old (886 lines of code)
    • 7 files that are 31-90 days old (296 lines of code)
    • 1 files that are 1-30 days old (196 lines of code)
68% | 22% | 6% | 2% | 1%
Legend:
366+
181-365
91-180
31-90
1-30

explore: grouped by folders | grouped by age
File Freshness Distribution Overall
Days since last update
  • There are 168 files with 14,128 lines of code in files.
    • 62 files have been last changed 366+ days ago (2,308 lines of code)
    • 57 files have been last changed 181-365 days ago (3,471 lines of code)
    • 13 files have been last changed 91-180 days ago (1,636 lines of code)
    • 21 files have been last changed 31-90 days ago (3,230 lines of code)
    • 15 files have been last changed 1-30 days ago (3,483 lines of code)
16% | 24% | 11% | 22% | 24%
Legend:
366+
181-365
91-180
31-90
1-30

explore: grouped by folders | grouped by freshness
File Change History per File Extension
rb, md, yaml, txt, sh, xml, json, html, gitignore, gemspec
File Age Distribution per Extension
Days since first update
366+
181-365
91-180
31-90
1-30
rb67% | 22% | 6% | 2% | 1%
yaml100% | 0% | 0% | 0% | 0%
xml100% | 0% | 0% | 0% | 0%
File Freshness Distribution per Extension
Days since last update
366+
181-365
91-180
31-90
1-30
rb16% | 24% | 11% | 23% | 24%
xml100% | 0% | 0% | 0% | 0%
yaml0% | 100% | 0% | 0% | 0%
File Change History per Logical Decomposition
primary
primary (file age distribution)
Days since first update
366+
181-365
91-180
31-90
1-30
spec64% | 20% | 10% | 1% | 2%
lib74% | 22% | 0% | 3% | 0%
ROOT100% | 0% | 0% | 0% | 0%
script10% | 89% | 0% | 0% | 0%
primary (file freshness distribution)
Days since last update
366+
181-365
91-180
31-90
1-30
spec17% | 18% | 19% | 23% | 19%
lib14% | 30% | <1% | 22% | 32%
script10% | 89% | 0% | 0% | 0%
ROOT0% | 100% | 0% | 0% | 0%
Oldest Files (Top 50)
File# lines# unitscreatedlast modified# changes
(days)		# contributorsfirst
contributor		latest
contributor
coordinator_spec.rb
in spec/lib/crawler
 708 8 2024-04-02 2025-02-06 12 3 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
coordinator.rb
in lib/crawler
 563 33 2024-04-02 2025-05-09 14 2 13634519+navarone-feekery@u... matt.nowzari@elastic.co
http_executor_spec.rb
in spec/lib/crawler
 366 - 2024-04-02 2025-03-28 7 2 13634519+navarone-feekery@u... matt.nowzari@elastic.co
html_spec.rb
in spec/lib/crawler/data/crawl_result
 365 - 2024-04-02 2025-04-16 5 2 13634519+navarone-feekery@u... matt.nowzari@elastic.co
config.rb
in lib/crawler/api
 341 23 2024-04-02 2025-05-08 27 5 13634519+navarone-feekery@u... williamseaston@gmail.com
http_client_spec.rb
in spec/lib/crawler
 318 7 2024-04-02 2025-03-28 5 2 13634519+navarone-feekery@u... matt.nowzari@elastic.co
http_executor.rb
in lib/crawler
 286 17 2024-04-02 2025-03-28 9 2 13634519+navarone-feekery@u... matt.nowzari@elastic.co
event_generator.rb
in lib/crawler
 279 31 2024-04-02 2025-03-28 11 3 13634519+navarone-feekery@u... matt.nowzari@elastic.co
http_client.rb
in lib/crawler
 276 22 2024-04-02 2025-04-03 9 2 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
config_spec.rb
in spec/lib/crawler/api
 218 1 2024-04-02 2025-05-08 9 4 13634519+navarone-feekery@u... williamseaston@gmail.com
sitemap_spec.rb
in spec/lib/crawler/data/crawl_result
 194 - 2024-04-02 2024-05-13 4 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
crawl.rb
in lib/crawler/api
 185 17 2024-04-02 2025-05-09 12 3 13634519+navarone-feekery@u... matt.nowzari@elastic.co
crawl_spec.rb
in spec/lib/crawler/api
 183 - 2024-04-02 2025-05-07 12 3 13634519+navarone-feekery@u... matt.nowzari@elastic.co
base_spec.rb
in spec/lib/crawler/rule_engine
 182 - 2024-04-02 2024-07-24 6 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
robots_txt_spec.rb
in spec/integration
 169 - 2024-04-02 2025-03-28 6 2 13634519+navarone-feekery@u... matt.nowzari@elastic.co
faux_crawl.rb
in spec/support/faux
 153 14 2024-04-02 2024-08-29 6 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
html.rb
in lib/crawler/data/crawl_result
 153 22 2024-04-02 2025-04-16 8 3 13634519+navarone-feekery@u... matt.nowzari@elastic.co
event_generator_spec.rb
in spec/lib/crawler
 150 - 2024-04-02 2025-03-28 7 2 13634519+navarone-feekery@u... matt.nowzari@elastic.co
robots_txt_parser_spec.rb
in spec/lib/crawler
 133 - 2024-04-02 2024-05-13 4 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
link_spec.rb
in spec/lib/crawler/data
 119 - 2024-04-02 2024-05-13 4 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
rule_engine_outcome.rb
in lib/crawler/data
 90 19 2024-04-02 2024-08-05 6 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
stats_spec.rb
in spec/lib/crawler
 85 - 2024-04-02 2024-05-06 2 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
headers_spec.rb
in spec/integration
 75 3 2024-04-02 2024-05-06 3 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
73 2 2024-04-02 2024-05-23 5 2 13634519+navarone-feekery@u... vidokx@gmail.com
crawl_result_spec.rb
in spec/lib/crawler/data
 72 - 2024-04-02 2024-05-13 4 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
robots_txt_service.rb
in lib/crawler
 72 12 2024-04-02 2024-05-23 5 2 13634519+navarone-feekery@u... vidokx@gmail.com
legacy_sitemaps_spec.rb
in spec/integration
 69 1 2024-04-02 2024-08-05 4 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
sitemap_xxe_spec.rb
in spec/integration
 69 2 2024-04-02 2024-08-05 5 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
base.rb
in lib/crawler/rule_engine
 68 7 2024-04-02 2024-08-05 7 2 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
stats.rb
in lib/crawler
 67 12 2024-04-02 2024-05-06 3 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
request_timeout_spec.rb
in spec/integration/timeouts
 65 3 2024-04-02 2024-06-28 5 2 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
crawl_task.rb
in lib/crawler/data
 61 12 2024-04-02 2024-05-06 3 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
link.rb
in lib/crawler/data
 60 11 2024-04-02 2024-05-23 4 2 13634519+navarone-feekery@u... vidokx@gmail.com
url_spec.rb
in spec/lib/crawler/data
 55 1 2024-04-02 2024-05-06 3 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
base.rb
in lib/crawler/data/url_queue
 55 13 2024-04-02 2024-05-06 3 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
memory_only_spec.rb
in spec/lib/crawler/data/url_queue
 54 - 2024-04-02 2024-06-28 5 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
robots_txt_parser.rb
in lib/crawler
 54 10 2024-04-02 2024-05-06 3 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
memory_only.rb
in lib/crawler/data/url_queue
 52 5 2024-04-02 2024-05-21 4 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
sitemap.rb
in lib/crawler/data/crawl_result
 50 5 2024-04-02 2024-05-06 3 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
46 1 2024-04-02 2024-05-06 3 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
url.rb
in lib/crawler/data
 46 9 2024-04-02 2024-07-23 4 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
content_extraction_spec.rb
in spec/integration
 44 - 2024-04-02 2024-05-06 3 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
charset_spec.rb
in spec/integration
 44 - 2024-04-02 2024-05-06 3 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
base.rb
in lib/crawler/output_sink
 41 11 2024-04-02 2024-08-22 12 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
redirects_spec.rb
in spec/integration
 40 - 2024-04-02 2024-08-05 5 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
38 - 2024-04-02 2024-05-13 6 2 13634519+navarone-feekery@u... vidokx@gmail.com
redirect.rb
in lib/crawler/data/crawl_result
 37 5 2024-04-02 2024-05-13 4 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
sitemap_spec.rb
in spec/integration
 36 - 2024-04-02 2024-05-06 4 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
output_sink_spec.rb
in spec/lib/crawler
 36 - 2024-04-02 2025-02-06 9 3 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
response_limits_spec.rb
in spec/integration
 34 2 2024-04-02 2024-05-06 3 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
Files Not Recently Changed (Top 50)
File# lines# unitscreatedlast modified# changes
(days)		# contributorsfirst
contributor		latest
contributor
sitemap_no_urls.xml
in spec/fixtures/sitemap
 3 - 2024-04-02 2024-04-02 1 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
sitemap_index.xml
in spec/fixtures/sitemap
 11 - 2024-04-02 2024-04-02 1 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
sitemap_urlset.xml
in spec/fixtures/sitemap
 27 - 2024-04-02 2024-04-02 1 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
string_colors.rb
in script/support
 14 4 2024-04-08 2024-04-15 2 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
core_ext.rb
in lib/crawler
 5 1 2024-04-02 2024-05-06 3 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
5 - 2024-04-02 2024-05-06 3 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
null.rb
in lib/crawler/output_sink
 9 1 2024-04-02 2024-05-06 3 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
9 - 2024-04-08 2024-05-06 4 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
robots_txt.rb
in lib/crawler/data/crawl_result
 10 - 2024-04-02 2024-05-06 3 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
executor.rb
in lib/crawler
 10 2 2024-04-02 2024-05-06 3 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
mock_event_logger.rb
in lib/crawler
 13 2 2024-04-02 2024-05-06 2 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
output_sink.rb
in lib/crawler
 14 2 2024-04-02 2024-05-06 3 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
seed_spec.rb
in spec/integration
 15 - 2024-04-02 2024-05-06 4 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
redirect_error.rb
in lib/crawler/data/crawl_result
 16 1 2024-04-02 2024-05-06 3 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
crawler_spec.rb
in spec/lib
 16 - 2024-04-02 2024-05-06 2 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
console.rb
in lib/crawler/output_sink
 17 1 2024-04-02 2024-05-06 4 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
mock.rb
in lib/crawler/output_sink
 17 2 2024-04-02 2024-05-06 3 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
all_trusting_trust_manager.rb
in lib/crawler/http_utils
 17 3 2024-04-15 2024-05-06 2 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
fixtures.rb
in spec/support
 17 4 2024-04-02 2024-05-06 2 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
results_collection.rb
in spec/support/faux
 17 4 2024-04-02 2024-05-06 3 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
config_spec.rb
in spec/lib/crawler/http_utils
 18 - 2024-04-15 2024-05-06 2 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
url_queue.rb
in lib/crawler/data
 19 2 2024-04-02 2024-05-06 3 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
rule_spec.rb
in spec/lib/crawler/data
 22 - 2024-04-02 2024-05-06 3 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
url_fragments_spec.rb
in spec/integration
 22 - 2024-04-02 2024-05-06 3 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
socket_timeout_spec.rb
in spec/integration/timeouts
 23 1 2024-04-02 2024-05-06 3 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
seen_urls.rb
in lib/crawler/data
 28 7 2024-04-02 2024-05-06 3 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
28 - 2024-04-02 2024-05-06 3 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
nofollow_spec.rb
in spec/integration
 30 - 2024-04-02 2024-05-06 3 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
domain_spec.rb
in spec/lib/crawler/data
 31 1 2024-04-02 2024-05-06 2 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
domain.rb
in lib/crawler/data
 33 6 2024-04-02 2024-05-06 2 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
response_limits_spec.rb
in spec/integration
 34 2 2024-04-02 2024-05-06 3 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
sitemap_spec.rb
in spec/integration
 36 - 2024-04-02 2024-05-06 4 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
charset_spec.rb
in spec/integration
 44 - 2024-04-02 2024-05-06 3 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
content_extraction_spec.rb
in spec/integration
 44 - 2024-04-02 2024-05-06 3 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
46 1 2024-04-02 2024-05-06 3 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
sitemap.rb
in lib/crawler/data/crawl_result
 50 5 2024-04-02 2024-05-06 3 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
robots_txt_parser.rb
in lib/crawler
 54 10 2024-04-02 2024-05-06 3 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
base.rb
in lib/crawler/data/url_queue
 55 13 2024-04-02 2024-05-06 3 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
url_spec.rb
in spec/lib/crawler/data
 55 1 2024-04-02 2024-05-06 3 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
crawl_task.rb
in lib/crawler/data
 61 12 2024-04-02 2024-05-06 3 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
stats.rb
in lib/crawler
 67 12 2024-04-02 2024-05-06 3 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
headers_spec.rb
in spec/integration
 75 3 2024-04-02 2024-05-06 3 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
stats_spec.rb
in spec/lib/crawler
 85 - 2024-04-02 2024-05-06 2 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
exceptions.rb
in lib/crawler/http_utils
 150 28 2024-04-15 2024-05-06 2 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
version_spec.rb
in spec/lib/crawler/cli
 9 - 2024-05-13 2024-05-13 1 1 vidokx@gmail.com vidokx@gmail.com
crawl_task_spec.rb
in spec/lib/crawler/data
 10 - 2024-04-02 2024-05-13 4 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
mock_response.rb
in spec/support
 11 2 2024-04-02 2024-05-13 3 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
version.rb
in lib/crawler/cli
 12 1 2024-05-13 2024-05-13 1 1 vidokx@gmail.com vidokx@gmail.com
12 2 2024-04-02 2024-05-13 3 2 13634519+navarone-feekery@u... vidokx@gmail.com
success.rb
in lib/crawler/data/crawl_result
 19 1 2024-04-02 2024-05-13 4 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
Most Recently Created Files (Top 50)
File# lines# unitscreatedlast modified# changes
(days)		# contributorsfirst
contributor		latest
contributor
helpers_spec.rb
in spec/lib/crawler/cli
 196 - 2025-04-21 2025-04-21 1 1 williamseaston@gmail.com williamseaston@gmail.com
urltest_spec.rb
in spec/lib/crawler/cli
 49 - 2025-04-11 2025-04-11 1 1 matt.nowzari@elastic.co matt.nowzari@elastic.co
urltest.rb
in lib/crawler/cli
 17 1 2025-04-11 2025-04-11 1 1 matt.nowzari@elastic.co matt.nowzari@elastic.co
crawllogger_spec.rb
in spec/lib/crawler/logging
 68 - 2025-03-28 2025-03-28 1 1 matt.nowzari@elastic.co matt.nowzari@elastic.co
file.rb
in lib/crawler/logging/handler
 50 5 2025-03-28 2025-03-28 1 1 matt.nowzari@elastic.co matt.nowzari@elastic.co
stdout.rb
in lib/crawler/logging/handler
 49 5 2025-03-28 2025-03-28 1 1 matt.nowzari@elastic.co matt.nowzari@elastic.co
logger.rb
in lib/crawler/logging
 44 11 2025-03-28 2025-03-28 1 1 matt.nowzari@elastic.co matt.nowzari@elastic.co
base.rb
in lib/crawler/logging/handler
 19 3 2025-03-28 2025-03-28 1 1 matt.nowzari@elastic.co matt.nowzari@elastic.co
url_request_check_spec.rb
in spec/lib/crawler/url_validator
 165 - 2025-01-02 2025-01-02 1 1 837854+bsantanna@users.nore... 837854+bsantanna@users.nore...
url_content_check_spec.rb
in spec/lib/crawler/url_validator
 148 - 2025-01-02 2025-01-02 1 1 837854+bsantanna@users.nore... 837854+bsantanna@users.nore...
url_validator_spec.rb
in spec/lib/crawler
 132 1 2025-01-02 2025-01-02 1 1 837854+bsantanna@users.nore... 837854+bsantanna@users.nore...
robots_txt_check_spec.rb
in spec/lib/crawler/url_validator
 104 - 2025-01-02 2025-01-02 1 1 837854+bsantanna@users.nore... 837854+bsantanna@users.nore...
tcp_check_spec.rb
in spec/lib/crawler/url_validator
 86 - 2025-01-02 2025-01-02 1 1 837854+bsantanna@users.nore... 837854+bsantanna@users.nore...
url_check_spec.rb
in spec/lib/crawler/url_validator
 74 - 2025-01-02 2025-01-02 1 1 837854+bsantanna@users.nore... 837854+bsantanna@users.nore...
dns_check_spec.rb
in spec/lib/crawler/url_validator
 62 - 2025-01-02 2025-01-02 1 1 837854+bsantanna@users.nore... 837854+bsantanna@users.nore...
crawl_rules_check_spec.rb
in spec/lib/crawler/url_validator
 49 - 2025-01-02 2025-01-02 1 1 837854+bsantanna@users.nore... 837854+bsantanna@users.nore...
domain_access_check_spec.rb
in spec/lib/crawler/url_validator
 33 - 2025-01-02 2025-01-02 1 1 837854+bsantanna@users.nore... 837854+bsantanna@users.nore...
domain_uniqueness_check_spec.rb
in spec/lib/crawler/url_validator
 33 - 2025-01-02 2025-01-02 1 1 837854+bsantanna@users.nore... 837854+bsantanna@users.nore...
schedule_spec.rb
in spec/lib/crawler/cli
 50 - 2024-08-28 2024-08-28 1 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
schedule.rb
in lib/crawler/cli
 34 2 2024-08-28 2024-08-28 1 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
document_mapper_spec.rb
in spec/lib/crawler
 219 - 2024-08-21 2025-04-16 3 2 13634519+navarone-feekery@u... matt.nowzari@elastic.co
client_spec.rb
in spec/lib/es
 480 1 2024-07-29 2025-05-05 6 3 13634519+navarone-feekery@u... jedrazb@gmail.com
client.rb
in lib/es
 198 15 2024-07-29 2025-05-05 6 3 13634519+navarone-feekery@u... jedrazb@gmail.com
bulk_queue_spec.rb
in spec/lib/es
 124 - 2024-07-29 2024-07-29 1 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
bulk_queue.rb
in lib/es
 70 8 2024-07-29 2024-07-29 1 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
utils.rb
in lib/crawler
 22 3 2024-07-24 2024-07-24 1 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
transformer_spec.rb
in spec/lib/crawler/content_engine
 213 1 2024-07-16 2024-07-16 1 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
extractor_spec.rb
in spec/lib/crawler/content_engine
 188 - 2024-07-16 2024-07-23 2 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
utils.rb
in lib/crawler/content_engine
 67 3 2024-07-16 2024-07-16 1 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
extractor.rb
in lib/crawler/content_engine
 49 5 2024-07-16 2024-07-23 2 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
transformer.rb
in lib/crawler/content_engine
 38 3 2024-07-16 2024-07-16 1 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
utils_spec.rb
in spec/lib/crawler/content_engine
 35 - 2024-07-16 2024-07-16 1 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
rule_spec.rb
in spec/lib/crawler/data/extraction
 162 - 2024-07-12 2025-04-07 3 2 13634519+navarone-feekery@u... matt.nowzari@elastic.co
rule.rb
in lib/crawler/data/extraction
 98 9 2024-07-12 2025-04-07 4 2 13634519+navarone-feekery@u... matt.nowzari@elastic.co
ruleset_spec.rb
in spec/lib/crawler/data/extraction
 61 - 2024-07-12 2024-07-16 2 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
url_filter_spec.rb
in spec/lib/crawler/data/extraction
 58 - 2024-07-12 2024-07-12 1 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
ruleset.rb
in lib/crawler/data/extraction
 52 5 2024-07-12 2024-07-24 3 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
url_filter.rb
in lib/crawler/data/extraction
 37 3 2024-07-12 2024-07-16 2 1 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
29 - 2024-07-12 2025-04-16 3 2 13634519+navarone-feekery@u... matt.nowzari@elastic.co
errors.rb
in lib
 6 - 2024-06-17 2025-02-05 3 2 13634519+navarone-feekery@u... matt.nowzari@elastic.co
url_validator.rb
in lib/crawler
 124 15 2024-05-23 2024-05-29 2 2 vidokx@gmail.com 13634519+navarone-feekery@u...
url_request_check_concern.rb
in lib/crawler/url_validator
 122 3 2024-05-23 2024-05-29 2 2 vidokx@gmail.com 13634519+navarone-feekery@u...
url_content_check_concern.rb
in lib/crawler/url_validator
 57 3 2024-05-23 2024-05-23 1 1 vidokx@gmail.com vidokx@gmail.com
validate_spec.rb
in spec/lib/crawler/cli
 53 - 2024-05-23 2024-05-23 1 1 vidokx@gmail.com vidokx@gmail.com
robots_txt_check_concern.rb
in lib/crawler/url_validator
 50 1 2024-05-23 2024-05-23 1 1 vidokx@gmail.com vidokx@gmail.com
helpers.rb
in lib/crawler/cli
 43 5 2024-05-23 2025-04-21 3 3 vidokx@gmail.com williamseaston@gmail.com
validate.rb
in lib/crawler/cli
 31 2 2024-05-23 2024-05-23 1 1 vidokx@gmail.com vidokx@gmail.com
dns_check_concern.rb
in lib/crawler/url_validator
 30 1 2024-05-23 2024-05-23 1 1 vidokx@gmail.com vidokx@gmail.com
tcp_check_concern.rb
in lib/crawler/url_validator
 24 1 2024-05-23 2024-05-23 1 1 vidokx@gmail.com vidokx@gmail.com
result.rb
in lib/crawler/url_validator
 19 3 2024-05-23 2024-05-23 1 1 vidokx@gmail.com vidokx@gmail.com
Most Recently Changed Files (Top 50)
File# lines# unitscreatedlast modified# changes
(days)		# contributorsfirst
contributor		latest
contributor
coordinator.rb
in lib/crawler
 563 33 2024-04-02 2025-05-09 14 2 13634519+navarone-feekery@u... matt.nowzari@elastic.co
crawl.rb
in lib/crawler/api
 185 17 2024-04-02 2025-05-09 12 3 13634519+navarone-feekery@u... matt.nowzari@elastic.co
config.rb
in lib/crawler/api
 341 23 2024-04-02 2025-05-08 27 5 13634519+navarone-feekery@u... williamseaston@gmail.com
config_spec.rb
in spec/lib/crawler/api
 218 1 2024-04-02 2025-05-08 9 4 13634519+navarone-feekery@u... williamseaston@gmail.com
elasticsearch.rb
in lib/crawler/output_sink
 222 22 2024-04-08 2025-05-07 21 4 13634519+navarone-feekery@u... matt.nowzari@elastic.co
crawl_spec.rb
in spec/lib/crawler/api
 183 - 2024-04-02 2025-05-07 12 3 13634519+navarone-feekery@u... matt.nowzari@elastic.co
client_spec.rb
in spec/lib/es
 480 1 2024-07-29 2025-05-05 6 3 13634519+navarone-feekery@u... jedrazb@gmail.com
client.rb
in lib/es
 198 15 2024-07-29 2025-05-05 6 3 13634519+navarone-feekery@u... jedrazb@gmail.com
helpers_spec.rb
in spec/lib/crawler/cli
 196 - 2025-04-21 2025-04-21 1 1 williamseaston@gmail.com williamseaston@gmail.com
helpers.rb
in lib/crawler/cli
 43 5 2024-05-23 2025-04-21 3 3 vidokx@gmail.com williamseaston@gmail.com
html_spec.rb
in spec/lib/crawler/data/crawl_result
 365 - 2024-04-02 2025-04-16 5 2 13634519+navarone-feekery@u... matt.nowzari@elastic.co
document_mapper_spec.rb
in spec/lib/crawler
 219 - 2024-08-21 2025-04-16 3 2 13634519+navarone-feekery@u... matt.nowzari@elastic.co
html.rb
in lib/crawler/data/crawl_result
 153 22 2024-04-02 2025-04-16 8 3 13634519+navarone-feekery@u... matt.nowzari@elastic.co
document_mapper.rb
in lib/crawler
 88 11 2024-04-05 2025-04-16 10 2 13634519+navarone-feekery@u... matt.nowzari@elastic.co
29 - 2024-07-12 2025-04-16 3 2 13634519+navarone-feekery@u... matt.nowzari@elastic.co
urltest_spec.rb
in spec/lib/crawler/cli
 49 - 2025-04-11 2025-04-11 1 1 matt.nowzari@elastic.co matt.nowzari@elastic.co
file_spec.rb
in spec/lib/crawler/output_sink
 26 1 2024-04-02 2025-04-11 6 2 13634519+navarone-feekery@u... matt.nowzari@elastic.co
urltest.rb
in lib/crawler/cli
 17 1 2025-04-11 2025-04-11 1 1 matt.nowzari@elastic.co matt.nowzari@elastic.co
cli.rb
in lib/crawler
 11 - 2024-05-13 2025-04-11 5 3 13634519+navarone-feekery@u... matt.nowzari@elastic.co
rule_spec.rb
in spec/lib/crawler/data/extraction
 162 - 2024-07-12 2025-04-07 3 2 13634519+navarone-feekery@u... matt.nowzari@elastic.co
rule.rb
in lib/crawler/data/extraction
 98 9 2024-07-12 2025-04-07 4 2 13634519+navarone-feekery@u... matt.nowzari@elastic.co
http_client.rb
in lib/crawler
 276 22 2024-04-02 2025-04-03 9 2 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
http_executor_spec.rb
in spec/lib/crawler
 366 - 2024-04-02 2025-03-28 7 2 13634519+navarone-feekery@u... matt.nowzari@elastic.co
http_client_spec.rb
in spec/lib/crawler
 318 7 2024-04-02 2025-03-28 5 2 13634519+navarone-feekery@u... matt.nowzari@elastic.co
http_executor.rb
in lib/crawler
 286 17 2024-04-02 2025-03-28 9 2 13634519+navarone-feekery@u... matt.nowzari@elastic.co
event_generator.rb
in lib/crawler
 279 31 2024-04-02 2025-03-28 11 3 13634519+navarone-feekery@u... matt.nowzari@elastic.co
robots_txt_spec.rb
in spec/integration
 169 - 2024-04-02 2025-03-28 6 2 13634519+navarone-feekery@u... matt.nowzari@elastic.co
event_generator_spec.rb
in spec/lib/crawler
 150 - 2024-04-02 2025-03-28 7 2 13634519+navarone-feekery@u... matt.nowzari@elastic.co
crawllogger_spec.rb
in spec/lib/crawler/logging
 68 - 2025-03-28 2025-03-28 1 1 matt.nowzari@elastic.co matt.nowzari@elastic.co
file.rb
in lib/crawler/logging/handler
 50 5 2025-03-28 2025-03-28 1 1 matt.nowzari@elastic.co matt.nowzari@elastic.co
stdout.rb
in lib/crawler/logging/handler
 49 5 2025-03-28 2025-03-28 1 1 matt.nowzari@elastic.co matt.nowzari@elastic.co
logger.rb
in lib/crawler/logging
 44 11 2025-03-28 2025-03-28 1 1 matt.nowzari@elastic.co matt.nowzari@elastic.co
base.rb
in lib/crawler/logging/handler
 19 3 2025-03-28 2025-03-28 1 1 matt.nowzari@elastic.co matt.nowzari@elastic.co
crawl_spec.rb
in spec/lib/crawler/cli
 63 - 2024-05-15 2025-03-19 2 2 vidokx@gmail.com matt.nowzari@elastic.co
elasticsearch_spec.rb
in spec/lib/crawler/output_sink
 594 - 2024-04-09 2025-03-06 19 3 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
response.rb
in lib/crawler/http_utils
 136 21 2024-04-15 2025-03-04 7 3 13634519+navarone-feekery@u... matt.nowzari@elastic.co
coordinator_spec.rb
in spec/lib/crawler
 708 8 2024-04-02 2025-02-06 12 3 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
output_sink_spec.rb
in spec/lib/crawler
 36 - 2024-04-02 2025-02-06 9 3 13634519+navarone-feekery@u... 13634519+navarone-feekery@u...
errors.rb
in lib
 6 - 2024-06-17 2025-02-05 3 2 13634519+navarone-feekery@u... matt.nowzari@elastic.co
url_request_check_spec.rb
in spec/lib/crawler/url_validator
 165 - 2025-01-02 2025-01-02 1 1 837854+bsantanna@users.nore... 837854+bsantanna@users.nore...
url_content_check_spec.rb
in spec/lib/crawler/url_validator
 148 - 2025-01-02 2025-01-02 1 1 837854+bsantanna@users.nore... 837854+bsantanna@users.nore...
url_validator_spec.rb
in spec/lib/crawler
 132 1 2025-01-02 2025-01-02 1 1 837854+bsantanna@users.nore... 837854+bsantanna@users.nore...
robots_txt_check_spec.rb
in spec/lib/crawler/url_validator
 104 - 2025-01-02 2025-01-02 1 1 837854+bsantanna@users.nore... 837854+bsantanna@users.nore...
tcp_check_spec.rb
in spec/lib/crawler/url_validator
 86 - 2025-01-02 2025-01-02 1 1 837854+bsantanna@users.nore... 837854+bsantanna@users.nore...
url_check_spec.rb
in spec/lib/crawler/url_validator
 74 - 2025-01-02 2025-01-02 1 1 837854+bsantanna@users.nore... 837854+bsantanna@users.nore...
dns_check_spec.rb
in spec/lib/crawler/url_validator
 62 - 2025-01-02 2025-01-02 1 1 837854+bsantanna@users.nore... 837854+bsantanna@users.nore...
crawl_rules_check_spec.rb
in spec/lib/crawler/url_validator
 49 - 2025-01-02 2025-01-02 1 1 837854+bsantanna@users.nore... 837854+bsantanna@users.nore...
domain_access_check_spec.rb
in spec/lib/crawler/url_validator
 33 - 2025-01-02 2025-01-02 1 1 837854+bsantanna@users.nore... 837854+bsantanna@users.nore...
domain_uniqueness_check_spec.rb
in spec/lib/crawler/url_validator
 33 - 2025-01-02 2025-01-02 1 1 837854+bsantanna@users.nore... 837854+bsantanna@users.nore...
59 - 2024-04-08 2024-09-27 8 3 13634519+navarone-feekery@u... klim.markelov@gmail.com