huggingface / OBELICS Conditional Complexity

Intro

Conditional complexity (also called cyclomatic complexity) is a term used to measure the complexity of software. The term refers to the number of possible paths through a program function. A higher value ofter means higher maintenance and testing costs (infosecinstitute.com).
Conditional complexity is calculated by counting all conditions in the program that can affect the execution path (e.g. if statement, loops, switches, and/or operators, try and catch blocks...).
Conditional complexity is measured at the unit level (methods, functions...).
Units are classified in four categories based on the measured McCabe index: 1-5 (simple units), 6-10 (medium complex units), 11-25 (complex units), 26+ (very complex units).

Learn more...

Conditional Complexity Overall

There are 186 units with 3,523 lines of code in units (45.5% of code).

0 very complex units (0 lines of code)
4 complex units (470 lines of code)
13 medium complex units (367 lines of code)
22 simple units (400 lines of code)
147 very simple units (2,286 lines of code)

Legend:

51+

26-50

11-25

6-10

1-5

Alternative Visuals

3D view of all units

Conditional Complexity per Extension

51+

26-50

11-25

6-10

1-5

Conditional Complexity per Logical Component

primary logical decomposition

51+

26-50

11-25

6-10

1-5

Most Complex Units

Top 50 most complex units

Unit	# lines	McCabe index	# params
def __call__() in obelics/processors/web_document_filtering.py	158	47	2
def get_media_src() in obelics/utils/simplification_utils.py	49	29	1
def extraction_mode() in obelics/visualization/global_visualization.py	170	28	1
def __call__() in obelics/processors/web_document_filtering.py	93	27	2
def _merge_consecutive_text_nodes() in obelics/processors/pre_extraction_simplificator.py	48	20	2
def get_domain_to_duplicated_texts() in build_obelics/09_04_get_domain_to_duplicated_texts.py	27	17	1
def simplify_media_node() in obelics/utils/simplification_utils.py	38	17	2
def urls_to_images() in obelics/processors/web_document_extractor.py	50	17	5
def line_deduplicate_web_documents() in obelics/processors/web_document_line_deduplication.py	39	13	1
def _strip_special_divs() in obelics/processors/dom_tree_simplificator.py	19	12	2
def _remove_empty_leaves() in obelics/processors/dom_tree_simplificator.py	23	12	2
def get_html_from_warc() in obelics/processors/html_extractor.py	33	12	2
def get_domain_to_duplicated_texts() in obelics/processors/web_document_line_deduplication.py	23	12	1
def urls_to_images() in build_obelics/04_merge_web_docs_with_images.py	22	11	5
def __call__() in build_obelics/12_02_remove_opt_out_images.py	10	11	2
def remove_duplicated_images() in build_obelics/13_final_processing.py	15	11	3
def __call__() in build_obelics/06_03_remove_image_duplicates.py	20	11	2
def _remove_dates() in obelics/processors/dom_tree_simplificator.py	14	10	2
def get_domain_to_positions() in obelics/processors/web_document_line_deduplication.py	25	10	1
def func_map_final_cleaning_node_level() in build_obelics/10_final_cleaning.py	34	9	1
def merge_consecutive_END_OF_DOCUMENT_TOKEN_TO_BE_REPLACED() in build_obelics/13_final_processing.py	22	9	3
def _format_texts() in obelics/processors/pre_extraction_simplificator.py	16	9	2
def get_exs_and_stats() in obelics/visualization/choose_filtering_parameters_web_documents_node_level.py	19	9	4
def final_cleaning_node_level() in build_obelics/13_final_processing.py	28	8	3
def process_one_tar() in obelics/processors/web_document_extractor.py	31	8	1
def display_document() in obelics/visualization/web_document_and_filtering_visualization.py	26	8	1
def func_filter_final_cleaning_doc_level() in build_obelics/10_final_cleaning.py	7	7	1
def final_cleaning_doc_level() in build_obelics/13_final_processing.py	7	7	1
def _unnest_nodes() in obelics/processors/dom_tree_simplificator.py	15	7	2
def func_html_to_web_documents() in obelics/processors/web_document_extractor.py	43	7	1
def get_words_from_text() in obelics/processors/web_document_filtering.py	10	7	4
def load_dataset() in obelics/visualization/web_document_and_filtering_visualization.py	17	7	1
def __call__() in build_obelics/09_06_line_dedup.py	14	6	2
def __call__() in build_obelics/08_02_urldedup.py	10	6	2
def get_domain_to_positions() in build_obelics/09_02_get_domain_to_positions.py	15	6	0
def _only_text_image_nodes() in obelics/processors/pre_extraction_simplificator.py	7	6	2
def _remove_nodes_matching_css_rules() in obelics/processors/dom_tree_simplificator.py	13	6	2
def get_image_urls() in obelics/processors/web_document_extractor.py	10	6	3
def strip() in obelics/processors/web_document_filtering.py	17	6	2
def __call__() in build_obelics/07_03_nsfw_image_removal.py	5	5	2
def remove_spam_paragraphs() in build_obelics/13_final_processing.py	13	5	3
def remove_end_END_OF_DOCUMENT_TOKEN_TO_BE_REPLACED() in build_obelics/13_final_processing.py	10	5	3
def compute_character_repetition_ratio() in obelics/processors/web_document_filtering.py	17	5	2
def compute_word_repetition_ratio() in obelics/processors/web_document_filtering.py	14	5	3
def select_mode() in obelics/visualization/web_document_and_filtering_visualization.py	15	5	1
def process_image() in build_obelics/07_01_nsfw_image_filtering.py	12	4	2
def compute_spam_word_ratio() in build_obelics/13_final_processing.py	8	4	1
def remove_texts_only_END_OF_DOCUMENT_TOKEN_TO_BE_REPLACED() in build_obelics/13_final_processing.py	4	4	1
def format_filename() in obelics/utils/simplification_utils.py	19	4	1
def format_relative_to_absolute_path() in obelics/utils/simplification_utils.py	11	4	2