huggingface / datatrove Conditional Complexity

Intro

Conditional complexity (also called cyclomatic complexity) is a term used to measure the complexity of software. The term refers to the number of possible paths through a program function. A higher value ofter means higher maintenance and testing costs (infosecinstitute.com).
Conditional complexity is calculated by counting all conditions in the program that can affect the execution path (e.g. if statement, loops, switches, and/or operators, try and catch blocks...).
Conditional complexity is measured at the unit level (methods, functions...).
Units are classified in four categories based on the measured McCabe index: 1-5 (simple units), 6-10 (medium complex units), 11-25 (complex units), 26+ (very complex units).

Learn more...

Conditional Complexity Overall

There are 548 units with 5,296 lines of code in units (41.4% of code).

0 very complex units (0 lines of code)
2 complex units (126 lines of code)
35 medium complex units (1,338 lines of code)
48 simple units (997 lines of code)
463 very simple units (2,835 lines of code)

Legend:

51+

26-50

11-25

6-10

1-5

Alternative Visuals

3D view of all units

Conditional Complexity per Extension

51+

26-50

11-25

6-10

1-5

Conditional Complexity per Logical Component

primary logical decomposition

51+

26-50

11-25

6-10

1-5

Most Complex Units

Top 50 most complex units

Unit	# lines	McCabe index	# params
def filter() in src/datatrove/pipeline/filters/gopher_quality_filter.py	40	35	2
def run() in src/datatrove/pipeline/dedup/minhash.py	86	26	4
def filter() in src/datatrove/pipeline/filters/c4_filters.py	38	24	2
def run() in src/datatrove/pipeline/tokens/merger.py	97	24	4
def run() in src/datatrove/pipeline/dedup/sentence_dedup.py	61	21	4
def run() in src/datatrove/pipeline/dedup/sentence_dedup.py	59	21	4
def run() in src/datatrove/pipeline/dedup/url_dedup.py	66	21	4
def run() in src/datatrove/pipeline/dedup/minhash.py	52	21	4
def process_record() in src/datatrove/pipeline/readers/warc.py	41	20	1
def remove_dup_sentences() in src/datatrove/pipeline/dedup/sentence_dedup.py	47	20	3
def run() in src/datatrove/pipeline/dedup/minhash.py	57	20	4
def load_tokenizer_assignments() in src/datatrove/utils/word_tokenizers.py	54	20	0
def get_dtokens_and_spaces() in src/datatrove/utils/japanese_tokenizer.py	37	19	3
def main() in src/datatrove/tools/inspect_data.py	58	18	0
def _get_dtokens() in src/datatrove/utils/japanese_tokenizer.py	24	17	3
def launch_job() in src/datatrove/executor/slurm.py	46	17	1
def run() in src/datatrove/pipeline/stats/base.py	34	16	4
def run() in src/datatrove/pipeline/readers/huggingface.py	36	15	4
def _get_pos_from_index_file() in src/datatrove/utils/dataset.py	24	15	2
def filter() in src/datatrove/pipeline/filters/gopher_repetition_filter.py	27	14	2
def filter() in src/datatrove/pipeline/filters/fineweb_quality_filter.py	19	14	2
def extract_stats() in src/datatrove/pipeline/stats/line_stats.py	22	14	2
def check_can_skip_sig_writing() in src/datatrove/pipeline/dedup/minhash.py	18	14	2
def run() in src/datatrove/executor/ray.py	68	14	1
def run() in src/datatrove/pipeline/decont/n_grams.py	37	12	4
def extract_stats() in src/datatrove/pipeline/stats/doc_stats.py	10	12	2
def run() in src/datatrove/pipeline/dedup/url_dedup.py	39	12	4
def run() in src/datatrove/pipeline/dedup/minhash.py	37	12	4
def write_unshuffled() in src/datatrove/pipeline/tokens/tokenizer.py	26	12	3
def simplify_text() in src/datatrove/utils/text.py	16	12	2
def split_into_parts() in src/datatrove/utils/text.py	27	12	3
def filter() in src/datatrove/pipeline/filters/fasttext_filter.py	27	11	2
def extract_stats() in src/datatrove/pipeline/stats/paragraph_stats.py	19	11	2
def get_duplicate_range() in src/datatrove/pipeline/dedup/exact_substrings.py	30	11	2
def run() in src/datatrove/pipeline/tokens/tokenizer.py	37	11	4
def to_dict() in src/datatrove/utils/stats.py	19	11	1
def __call__() in src/datatrove/utils/japanese_tokenizer.py	29	11	2
def filter() in src/datatrove/pipeline/filters/language_filter.py	17	10	2
def filter() in src/datatrove/pipeline/filters/url_filter.py	21	10	2
def _get_badwords() in src/datatrove/pipeline/filters/c4_filters.py	27	10	2
def run() in src/datatrove/pipeline/filters/base_filter.py	22	10	4
def extract_stats() in src/datatrove/pipeline/stats/word_stats.py	21	10	2
def run() in src/datatrove/pipeline/dedup/minhash.py	27	10	4
def _load_file_list() in src/datatrove/utils/dataset.py	32	10	1
def run() in src/datatrove/executor/local.py	52	10	1
def main() in src/datatrove/tools/jobs_status.py	52	10	0
def format() in src/datatrove/pipeline/formatters/symbol_lines_remover.py	13	9	2
def run() in src/datatrove/pipeline/stats/merger.py	23	9	4
def read_files_shard() in src/datatrove/pipeline/readers/base.py	31	9	2
def __init__() in src/datatrove/pipeline/tokens/tokenizer.py	17	9	10