huggingface / fineweb-2 Conditional Complexity

Intro

Conditional complexity (also called cyclomatic complexity) is a term used to measure the complexity of software. The term refers to the number of possible paths through a program function. A higher value ofter means higher maintenance and testing costs (infosecinstitute.com).
Conditional complexity is calculated by counting all conditions in the program that can affect the execution path (e.g. if statement, loops, switches, and/or operators, try and catch blocks...).
Conditional complexity is measured at the unit level (methods, functions...).
Units are classified in four categories based on the measured McCabe index: 1-5 (simple units), 6-10 (medium complex units), 11-25 (complex units), 26+ (very complex units).

Learn more...

Conditional Complexity Overall

There are 53 units with 781 lines of code in units (39.6% of code).

0 very complex units (0 lines of code)
0 complex units (0 lines of code)
2 medium complex units (61 lines of code)
13 simple units (372 lines of code)
38 very simple units (348 lines of code)

Legend:

51+

26-50

11-25

6-10

1-5

Alternative Visuals

3D view of all units

Conditional Complexity per Extension

51+

26-50

11-25

6-10

1-5

Conditional Complexity per Logical Component

primary logical decomposition

51+

26-50

11-25

6-10

1-5

Most Complex Units

Top 50 most complex units

Unit	# lines	McCabe index	# params
def run() in misc/precision_filtering/slurm_count_word.py	37	14	4
def get_checkpoints_to_run() in ablations/evaluation/launch_evals.py	24	12	6
def read_files_shard() in misc/reference_datasets/monolingual/zh/download_mapcc.py	29	9	2
def read_files_shard() in misc/reference_datasets/multilingual/part jsons.py	29	9	2
def print_differences() in ablations/training/launch_exp.py	14	8	2
def run() in misc/reference_datasets/multilingual/copy_raw_data.py	20	8	4
def __init__() in misc/precision_filtering/run_precision_filtering.py	31	8	5
def url_filter() in misc/precision_filtering/run_precision_filtering.py	19	8	2
def load_words_from_txt() in misc/precision_filtering/wordlist_score.py	13	8	1
def filter_score() in misc/precision_filtering/wordlist_score.py	7	8	2
def run() in misc/reference_datasets/multilingual/download_hplt.py	18	7	4
def read_file() in misc/reference_datasets/multilingual/copy_raw_data.py	23	7	2
def filter_by_ratio() in misc/precision_filtering/wordlist_gen.py	9	7	3
def checkpoint_exists() in ablations/evaluation/launch_evals.py	13	6	4
def run() in misc/reference_datasets/multilingual/download_cc-100.py	147	6	4
def read() in misc/reference_datasets/monolingual/zh/download_mapcc.py	13	5	2
def read() in misc/reference_datasets/multilingual/part jsons.py	13	5	2
def read_file() in misc/reference_datasets/multilingual/part jsons.py	17	5	2
def load_and_save_tokenizer_freq() in misc/precision_filtering/count_common.py	16	5	2
def run() in misc/reference_datasets/monolingual/zh/download_mapcc.py	23	4	4
def read_file() in misc/reference_datasets/monolingual/ar/download_arabicweb24.py	16	4	2
def read_file() in misc/reference_datasets/monolingual/fr/download_croissant.py	16	4	2
def wordlist() in misc/precision_filtering/run_precision_filtering.py	6	4	1
def filter_top_percentile() in misc/precision_filtering/wordlist_gen.py	8	4	2
def parse_date() in ablations/evaluation/launch_evals.py	7	3	1
def _open_next_file() in misc/reference_datasets/monolingual/zh/download_mapcc.py	9	3	1
def run() in misc/reference_datasets/monolingual/zh/download_mapcc.py	6	3	4
def _open_next_file() in misc/reference_datasets/multilingual/part jsons.py	9	3	1
def launch_slurm_job() in ablations/evaluation/launch_evals.py	9	2	2
def launch_slurm_job() in ablations/evaluation/launch_random_evals.py	9	2	2
def close() in misc/reference_datasets/monolingual/zh/download_mapcc.py	3	2	1
def adapter() in misc/reference_datasets/multilingual/download_mc4.py	10	2	4
def close() in misc/reference_datasets/multilingual/part jsons.py	3	2	1
def wordlist_filter() in misc/precision_filtering/run_precision_filtering.py	7	2	2
def filter() in misc/precision_filtering/run_precision_filtering.py	2	2	2
def save() in misc/precision_filtering/wordlist_gen.py	14	2	2
def launch_slurm_job() in ablations/training/launch_exp.py	5	1	2
def above_lang_threshold() in fineweb-2-pipeline.py	2	1	2
def __init__() in misc/reference_datasets/monolingual/zh/download_mapcc.py	5	1	2
def __init__() in misc/reference_datasets/monolingual/zh/download_mapcc.py	9	1	0
def open_concatenated_gzip_files() in misc/reference_datasets/monolingual/zh/download_mapcc.py	4	1	1
def __init__() in misc/reference_datasets/monolingual/ar/download_arabicweb24.py	17	1	0
def __init__() in misc/reference_datasets/monolingual/fr/download_croissant.py	17	1	0
def adapter() in misc/reference_datasets/multilingual/download_culturax.py	7	1	4
def __init__() in misc/reference_datasets/multilingual/part jsons.py	5	1	2
def open_concatenated_gzip_files() in misc/reference_datasets/multilingual/part jsons.py	4	1	1
def __init__() in misc/reference_datasets/multilingual/part jsons.py	9	1	0
def adapter() in misc/reference_datasets/multilingual/copy_raw_data.py	7	1	4
def __init__() in misc/reference_datasets/multilingual/copy_raw_data.py	32	1	15
def generate_tokenizer_identifier() in misc/precision_filtering/slurm_count_word.py	2	1	1