huggingface / OBELICS File Age & Freshness

File age measurements show the distribution of file ages (days since the first commit) and the file freshness (days since the latest commit).

Summary

Number of files: 58
Daily file updates (only one update per file and date counted): 120
First update: 2023-06-07
Latest update: 2024-08-28
Days between first and latest update: 449 (64 weeks, estimated 320 working days)
Active days (at least one file change): 7
Data:

Organized per file

File Change History Overall

File Age Distribution Overall

Days since first update

There are 58 files with 7,737 lines of code in files.

58 files that are 366+ days old (7,737 lines of code)
0 files that are 181-365 days old (0 lines of code)
0 files that are 91-180 days old (0 lines of code)
0 files that are 31-90 days old (0 lines of code)
0 files that are 1-30 days old (0 lines of code)

Legend:

366+

181-365

91-180

31-90

1-30

explore: grouped by folders | grouped by age

File Freshness Distribution Overall

Days since last update

There are 58 files with 7,737 lines of code in files.

58 files have been last changed 366+ days ago (7,737 lines of code)
0 files have been last changed 181-365 days ago (0 lines of code)
0 files have been last changed 91-180 days ago (0 lines of code)
0 files have been last changed 31-90 days ago (0 lines of code)
0 files have been last changed 1-30 days ago (0 lines of code)

Legend:

366+

181-365

91-180

31-90

1-30

explore: grouped by folders | grouped by freshness

File Change History per File Extension

py, txt, yaml, gitignore, md, html

File Age Distribution per Extension

Days since first update

366+

181-365

91-180

31-90

1-30

File Freshness Distribution per Extension

Days since last update

366+

181-365

91-180

31-90

1-30

File Change History per Logical Decomposition

primary

primary (file age distribution)

Days since first update

366+

181-365

91-180

31-90

1-30

primary (file freshness distribution)

Days since last update

366+

181-365

91-180

31-90

1-30

Oldest Files (Top 50)

File	# lines	# units	created	last modified	# changes (days)	# contributors	first contributor	latest contributor
web_document_filtering.py in obelics/processors	1002	38	2023-06-07	2023-08-10	3	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
web_document_and_filtering_visualization.py in obelics/visualization	675	8	2023-06-07	2023-08-10	3	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
filtering_utils.py in obelics/utils	427	-	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
web_document_extractor.py in obelics/processors	371	20	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
global_visualization.py in obelics/visualization	321	7	2023-06-07	2023-08-10	3	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
13_final_processing.py in build_obelics	276	11	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
05_filtering_web_docs.py in build_obelics	225	1	2023-06-07	2023-08-10	3	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
choose_filtering_parameters_web_documents_node_level.py in obelics/visualization	205	3	2023-06-07	2023-08-10	3	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
filter_web_documents.py in obelics/callers	196	1	2023-06-07	2023-08-10	3	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
dom_tree_simplificator.py in obelics/processors	195	16	2023-06-07	2023-08-10	3	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
extract_web_documents.py in obelics/callers	176	1	2023-06-07	2023-08-10	3	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
simplification_utils.py in obelics/utils	176	6	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
07_01_nsfw_image_filtering.py in build_obelics	175	8	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
DOM_tree_viz.html in obelics/visualization/assets	172	-	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
03_dl_images_create_dataset.py in build_obelics	163	1	2023-06-07	2023-08-10	3	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
pre_extraction_simplificator.py in obelics/processors	152	11	2023-06-07	2023-08-10	3	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
02_extract_html_get_image_urls.py in build_obelics	144	1	2023-06-07	2023-08-10	3	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
04_merge_web_docs_with_images.py in build_obelics	140	2	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
02_bis_extract_html_get_image_urls_new_rules.py in build_obelics	140	1	2023-06-07	2023-08-10	3	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
06_03_remove_image_duplicates.py in build_obelics	123	4	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
web_document_line_deduplication.py in obelics/processors	121	6	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
02_parallel_extract_html_get_image_urls.py in build_obelics	113	-	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
10_final_cleaning.py in build_obelics	113	2	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
11_03_set_img_urls_dedup.py in build_obelics	94	3	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
09_06_line_dedup.py in build_obelics	91	3	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
09_04_get_domain_to_duplicated_texts.py in build_obelics	90	1	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
06_01_create_set_image_urls_in_webdocs.py in build_obelics	90	1	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
01_download_warc.py in build_obelics	89	1	2023-06-07	2023-08-10	3	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
09_05_merge_domain_to_duplicated_texts_sharded.py in build_obelics	86	-	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
08_02_urldedup.py in build_obelics	82	3	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
tags_attributes.py in obelics/utils	82	-	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
12_01_find_opt_out_images.py in build_obelics	79	1	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
07_03_nsfw_image_removal.py in build_obelics	77	3	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
11_02_get_docs_to_remove_by_set_img_urls_dedup.py in build_obelics	75	-	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
12_02_remove_opt_out_images.py in build_obelics	74	3	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
07_02_nsfw_image_visualization.py in build_obelics	68	1	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
line_deduplicate_web_documents.py in obelics/callers	65	1	2023-06-07	2023-08-10	3	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
08_01_prepare_urldedup.py in build_obelics	64	1	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
09_07_merge_web_docs_texts_only_and_rest.py in build_obelics	64	-	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
config_filter_web_documents.yaml in obelics/configs	64	-	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
download_warc.py in obelics/callers	63	1	2023-06-07	2023-08-10	3	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
09_02_get_domain_to_positions.py in build_obelics	62	1	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
web_document_visualization.py in obelics/visualization	59	6	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
extract_html.py in obelics/callers	53	1	2023-06-07	2023-08-10	3	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
html_extractor.py in obelics/processors	52	2	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
11_01_create_set_img_urls.py in build_obelics	50	-	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
09_01_create_web_docs_texts_only.py in build_obelics	49	-	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
06_02_merge_sets_image_urls_in_webdocs.py in build_obelics	48	-	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
warc_downloader.py in obelics/processors	36	4	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
09_03_split_domain_to_positions.py in build_obelics	34	-	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com

Files Not Recently Changed (Top 50)

File	# lines	# units	created	last modified	# changes (days)	# contributors	first contributor	latest contributor
__init__.py in obelics/visualization	1	-	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
__init__.py in obelics	1	-	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
__init__.py in obelics/callers	1	-	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
utils.py in obelics/utils	4	1	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
__init__.py in obelics/processors	11	-	2023-06-07	2023-08-10	3	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
config_extract_web_documents.yaml in obelics/configs	23	-	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
__init__.py in obelics/utils	24	-	2023-06-07	2023-08-10	3	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
03_parallel_dl_images_create_dataset.py in build_obelics	31	-	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
09_03_split_domain_to_positions.py in build_obelics	34	-	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
warc_downloader.py in obelics/processors	36	4	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
06_02_merge_sets_image_urls_in_webdocs.py in build_obelics	48	-	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
09_01_create_web_docs_texts_only.py in build_obelics	49	-	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
11_01_create_set_img_urls.py in build_obelics	50	-	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
html_extractor.py in obelics/processors	52	2	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
extract_html.py in obelics/callers	53	1	2023-06-07	2023-08-10	3	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
web_document_visualization.py in obelics/visualization	59	6	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
09_02_get_domain_to_positions.py in build_obelics	62	1	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
download_warc.py in obelics/callers	63	1	2023-06-07	2023-08-10	3	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
config_filter_web_documents.yaml in obelics/configs	64	-	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
09_07_merge_web_docs_texts_only_and_rest.py in build_obelics	64	-	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
08_01_prepare_urldedup.py in build_obelics	64	1	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
line_deduplicate_web_documents.py in obelics/callers	65	1	2023-06-07	2023-08-10	3	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
07_02_nsfw_image_visualization.py in build_obelics	68	1	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
12_02_remove_opt_out_images.py in build_obelics	74	3	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
11_02_get_docs_to_remove_by_set_img_urls_dedup.py in build_obelics	75	-	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
07_03_nsfw_image_removal.py in build_obelics	77	3	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
12_01_find_opt_out_images.py in build_obelics	79	1	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
tags_attributes.py in obelics/utils	82	-	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
08_02_urldedup.py in build_obelics	82	3	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
09_05_merge_domain_to_duplicated_texts_sharded.py in build_obelics	86	-	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
01_download_warc.py in build_obelics	89	1	2023-06-07	2023-08-10	3	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
06_01_create_set_image_urls_in_webdocs.py in build_obelics	90	1	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
09_04_get_domain_to_duplicated_texts.py in build_obelics	90	1	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
09_06_line_dedup.py in build_obelics	91	3	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
11_03_set_img_urls_dedup.py in build_obelics	94	3	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
10_final_cleaning.py in build_obelics	113	2	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
02_parallel_extract_html_get_image_urls.py in build_obelics	113	-	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
web_document_line_deduplication.py in obelics/processors	121	6	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
06_03_remove_image_duplicates.py in build_obelics	123	4	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
02_bis_extract_html_get_image_urls_new_rules.py in build_obelics	140	1	2023-06-07	2023-08-10	3	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
04_merge_web_docs_with_images.py in build_obelics	140	2	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
02_extract_html_get_image_urls.py in build_obelics	144	1	2023-06-07	2023-08-10	3	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
pre_extraction_simplificator.py in obelics/processors	152	11	2023-06-07	2023-08-10	3	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
03_dl_images_create_dataset.py in build_obelics	163	1	2023-06-07	2023-08-10	3	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
DOM_tree_viz.html in obelics/visualization/assets	172	-	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
07_01_nsfw_image_filtering.py in build_obelics	175	8	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
simplification_utils.py in obelics/utils	176	6	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
extract_web_documents.py in obelics/callers	176	1	2023-06-07	2023-08-10	3	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
dom_tree_simplificator.py in obelics/processors	195	16	2023-06-07	2023-08-10	3	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
filter_web_documents.py in obelics/callers	196	1	2023-06-07	2023-08-10	3	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com

Most Recently Created Files (Top 50)

File	# lines	# units	created	last modified	# changes (days)	# contributors	first contributor	latest contributor
web_document_filtering.py in obelics/processors	1002	38	2023-06-07	2023-08-10	3	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
web_document_and_filtering_visualization.py in obelics/visualization	675	8	2023-06-07	2023-08-10	3	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
filtering_utils.py in obelics/utils	427	-	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
web_document_extractor.py in obelics/processors	371	20	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
global_visualization.py in obelics/visualization	321	7	2023-06-07	2023-08-10	3	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
13_final_processing.py in build_obelics	276	11	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
05_filtering_web_docs.py in build_obelics	225	1	2023-06-07	2023-08-10	3	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
choose_filtering_parameters_web_documents_node_level.py in obelics/visualization	205	3	2023-06-07	2023-08-10	3	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
filter_web_documents.py in obelics/callers	196	1	2023-06-07	2023-08-10	3	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
dom_tree_simplificator.py in obelics/processors	195	16	2023-06-07	2023-08-10	3	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
extract_web_documents.py in obelics/callers	176	1	2023-06-07	2023-08-10	3	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
simplification_utils.py in obelics/utils	176	6	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
07_01_nsfw_image_filtering.py in build_obelics	175	8	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
DOM_tree_viz.html in obelics/visualization/assets	172	-	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
03_dl_images_create_dataset.py in build_obelics	163	1	2023-06-07	2023-08-10	3	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
pre_extraction_simplificator.py in obelics/processors	152	11	2023-06-07	2023-08-10	3	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
02_extract_html_get_image_urls.py in build_obelics	144	1	2023-06-07	2023-08-10	3	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
04_merge_web_docs_with_images.py in build_obelics	140	2	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
02_bis_extract_html_get_image_urls_new_rules.py in build_obelics	140	1	2023-06-07	2023-08-10	3	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
06_03_remove_image_duplicates.py in build_obelics	123	4	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
web_document_line_deduplication.py in obelics/processors	121	6	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
02_parallel_extract_html_get_image_urls.py in build_obelics	113	-	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
10_final_cleaning.py in build_obelics	113	2	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
11_03_set_img_urls_dedup.py in build_obelics	94	3	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
09_06_line_dedup.py in build_obelics	91	3	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
09_04_get_domain_to_duplicated_texts.py in build_obelics	90	1	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
06_01_create_set_image_urls_in_webdocs.py in build_obelics	90	1	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
01_download_warc.py in build_obelics	89	1	2023-06-07	2023-08-10	3	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
09_05_merge_domain_to_duplicated_texts_sharded.py in build_obelics	86	-	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
08_02_urldedup.py in build_obelics	82	3	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
tags_attributes.py in obelics/utils	82	-	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
12_01_find_opt_out_images.py in build_obelics	79	1	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
07_03_nsfw_image_removal.py in build_obelics	77	3	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
11_02_get_docs_to_remove_by_set_img_urls_dedup.py in build_obelics	75	-	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
12_02_remove_opt_out_images.py in build_obelics	74	3	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
07_02_nsfw_image_visualization.py in build_obelics	68	1	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
line_deduplicate_web_documents.py in obelics/callers	65	1	2023-06-07	2023-08-10	3	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
08_01_prepare_urldedup.py in build_obelics	64	1	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
09_07_merge_web_docs_texts_only_and_rest.py in build_obelics	64	-	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
config_filter_web_documents.yaml in obelics/configs	64	-	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
download_warc.py in obelics/callers	63	1	2023-06-07	2023-08-10	3	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
09_02_get_domain_to_positions.py in build_obelics	62	1	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
web_document_visualization.py in obelics/visualization	59	6	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
extract_html.py in obelics/callers	53	1	2023-06-07	2023-08-10	3	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
html_extractor.py in obelics/processors	52	2	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
11_01_create_set_img_urls.py in build_obelics	50	-	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
09_01_create_web_docs_texts_only.py in build_obelics	49	-	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
06_02_merge_sets_image_urls_in_webdocs.py in build_obelics	48	-	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
warc_downloader.py in obelics/processors	36	4	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
09_03_split_domain_to_positions.py in build_obelics	34	-	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com

Most Recently Changed Files (Top 50)

File	# lines	# units	created	last modified	# changes (days)	# contributors	first contributor	latest contributor
web_document_filtering.py in obelics/processors	1002	38	2023-06-07	2023-08-10	3	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
web_document_and_filtering_visualization.py in obelics/visualization	675	8	2023-06-07	2023-08-10	3	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
filtering_utils.py in obelics/utils	427	-	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
web_document_extractor.py in obelics/processors	371	20	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
global_visualization.py in obelics/visualization	321	7	2023-06-07	2023-08-10	3	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
13_final_processing.py in build_obelics	276	11	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
05_filtering_web_docs.py in build_obelics	225	1	2023-06-07	2023-08-10	3	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
choose_filtering_parameters_web_documents_node_level.py in obelics/visualization	205	3	2023-06-07	2023-08-10	3	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
filter_web_documents.py in obelics/callers	196	1	2023-06-07	2023-08-10	3	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
dom_tree_simplificator.py in obelics/processors	195	16	2023-06-07	2023-08-10	3	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
extract_web_documents.py in obelics/callers	176	1	2023-06-07	2023-08-10	3	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
simplification_utils.py in obelics/utils	176	6	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
07_01_nsfw_image_filtering.py in build_obelics	175	8	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
DOM_tree_viz.html in obelics/visualization/assets	172	-	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
03_dl_images_create_dataset.py in build_obelics	163	1	2023-06-07	2023-08-10	3	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
pre_extraction_simplificator.py in obelics/processors	152	11	2023-06-07	2023-08-10	3	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
02_extract_html_get_image_urls.py in build_obelics	144	1	2023-06-07	2023-08-10	3	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
04_merge_web_docs_with_images.py in build_obelics	140	2	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
02_bis_extract_html_get_image_urls_new_rules.py in build_obelics	140	1	2023-06-07	2023-08-10	3	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
06_03_remove_image_duplicates.py in build_obelics	123	4	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
web_document_line_deduplication.py in obelics/processors	121	6	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
02_parallel_extract_html_get_image_urls.py in build_obelics	113	-	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
10_final_cleaning.py in build_obelics	113	2	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
11_03_set_img_urls_dedup.py in build_obelics	94	3	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
09_06_line_dedup.py in build_obelics	91	3	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
09_04_get_domain_to_duplicated_texts.py in build_obelics	90	1	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
06_01_create_set_image_urls_in_webdocs.py in build_obelics	90	1	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
01_download_warc.py in build_obelics	89	1	2023-06-07	2023-08-10	3	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
09_05_merge_domain_to_duplicated_texts_sharded.py in build_obelics	86	-	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
08_02_urldedup.py in build_obelics	82	3	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
tags_attributes.py in obelics/utils	82	-	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
12_01_find_opt_out_images.py in build_obelics	79	1	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
07_03_nsfw_image_removal.py in build_obelics	77	3	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
11_02_get_docs_to_remove_by_set_img_urls_dedup.py in build_obelics	75	-	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
12_02_remove_opt_out_images.py in build_obelics	74	3	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
07_02_nsfw_image_visualization.py in build_obelics	68	1	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
line_deduplicate_web_documents.py in obelics/callers	65	1	2023-06-07	2023-08-10	3	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
08_01_prepare_urldedup.py in build_obelics	64	1	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
09_07_merge_web_docs_texts_only_and_rest.py in build_obelics	64	-	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
config_filter_web_documents.yaml in obelics/configs	64	-	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
download_warc.py in obelics/callers	63	1	2023-06-07	2023-08-10	3	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
09_02_get_domain_to_positions.py in build_obelics	62	1	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
web_document_visualization.py in obelics/visualization	59	6	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
extract_html.py in obelics/callers	53	1	2023-06-07	2023-08-10	3	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
html_extractor.py in obelics/processors	52	2	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
11_01_create_set_img_urls.py in build_obelics	50	-	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
09_01_create_web_docs_texts_only.py in build_obelics	49	-	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
06_02_merge_sets_image_urls_in_webdocs.py in build_obelics	48	-	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
warc_downloader.py in obelics/processors	36	4	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com
09_03_split_domain_to_positions.py in build_obelics	34	-	2023-06-07	2023-08-10	2	1	hugo.laurencon@gmail.com	hugo.laurencon@gmail.com