huggingface / tokenizers Conditional Complexity

Intro

Conditional complexity (also called cyclomatic complexity) is a term used to measure the complexity of software. The term refers to the number of possible paths through a program function. A higher value ofter means higher maintenance and testing costs (infosecinstitute.com).
Conditional complexity is calculated by counting all conditions in the program that can affect the execution path (e.g. if statement, loops, switches, and/or operators, try and catch blocks...).
Conditional complexity is measured at the unit level (methods, functions...).
Units are classified in four categories based on the measured McCabe index: 1-5 (simple units), 6-10 (medium complex units), 11-25 (complex units), 26+ (very complex units).

Learn more...

Conditional Complexity Overall

There are 1,522 units with 15,667 lines of code in units (53.9% of code).

0 very complex units (0 lines of code)
0 complex units (0 lines of code)
10 medium complex units (613 lines of code)
26 simple units (878 lines of code)
1,486 very simple units (14,176 lines of code)

Legend:

51+

26-50

11-25

6-10

1-5

Alternative Visuals

3D view of all units

Conditional Complexity per Extension

51+

26-50

11-25

6-10

1-5

Conditional Complexity per Logical Component

primary logical decomposition

51+

26-50

11-25

6-10

1-5

Most Complex Units

Top 50 most complex units

Unit	# lines	McCabe index	# params
fn prune_sentence_pieces() in tokenizers/src/models/unigram/trainer.rs	116	22	4
def check_details() in bindings/python/scripts/spm_parity_check.py	48	19	5
fn encode_optimized() in tokenizers/src/models/unigram/model.rs	84	16	2
fn merge_word() in tokenizers/src/models/bpe/model.rs	76	16	2
fn find_matches() in tokenizers/src/tokenizer/added_vocabulary.rs	42	14	3
fn tokenize_words() in tokenizers/src/models/bpe/trainer.rs	43	13	5
fn decode_chain() in tokenizers/src/decoders/byte_fallback.rs	36	12	2
def pyi_file() in bindings/python/stub.py	40	12	2
fn finalize() in tokenizers/src/models/unigram/trainer.rs	63	11	3
fn make_seed_sentence_pieces() in tokenizers/src/models/unigram/trainer.rs	65	11	3
fn visit_map() in tokenizers/src/models/bpe/serialization.rs	77	10	2
def __make_html() in bindings/python/py_src/tokenizers/tools/visualizer.py	51	9	3
def write() in bindings/python/stub.py	35	9	4
fn tokenize() in tokenizers/src/models/wordpiece/mod.rs	53	8	2
fn compute_alphabet() in tokenizers/src/models/bpe/trainer.rs	47	8	4
fn count_pairs() in tokenizers/src/models/bpe/trainer.rs	37	8	4
def check_train() in bindings/python/scripts/spm_parity_check.py	27	8	1
def check_encode() in bindings/python/scripts/spm_parity_check.py	43	8	1
fn apply_to_bpe_builder() in bindings/node/src/models.rs	24	8	2
fn normalize() in tokenizers/src/normalizers/bert.rs	16	7	2
fn lrstrip() in tokenizers/src/tokenizer/normalizer.rs	35	7	3
fn validate() in tokenizers/src/processors/template.rs	53	7	1
def __make_char_states() in bindings/python/py_src/tokenizers/tools/visualizer.py	12	7	3
fn extract_bound() in bindings/python/src/tokenizer.rs	21	7	1
fn extract_bound() in bindings/python/src/processors.rs	27	7	1
def fn_predicate() in bindings/python/stub.py	7	7	1
fn normalize() in tokenizers/src/normalizers/precompiled.rs	26	6	2
fn encode_unoptimized() in tokenizers/src/models/unigram/model.rs	26	6	2
fn tokenize_with_cache() in tokenizers/src/models/bpe/model.rs	22	6	2
fn visit_map() in tokenizers/src/tokenizer/serialization.rs	59	6	2
def from_spm() in bindings/python/py_src/tokenizers/implementations/sentencepiece_unigram.py	41	6	1
fn new() in bindings/python/src/models.rs	36	6	4
fn refill() in bindings/python/src/utils/iterators.rs	28	6	1
fn enable_padding() in bindings/python/src/tokenizer.rs	52	6	2
def extract() in bindings/python/scripts/sentencepiece_extractor.py	13	6	1
def check_diff() in bindings/python/scripts/spm_parity_check.py	10	6	4
fn normalize() in tokenizers/src/normalizers/strip.rs	13	5	2
fn find_matches() in tokenizers/src/utils/fancy.rs	24	5	2
fn pre_tokenize() in tokenizers/src/pre_tokenizers/unicode_scripts/pre_tokenizer.rs	35	5	2
fn pre_tokenize() in tokenizers/src/pre_tokenizers/byte_level.rs	28	5	2
fn pre_tokenize() in tokenizers/src/pre_tokenizers/metaspace.rs	25	5	2
fn run_m_step() in tokenizers/src/models/unigram/trainer.rs	35	5	3
fn tokenize() in tokenizers/src/models/unigram/model.rs	35	5	2
fn decode_chain() in tokenizers/src/decoders/wordpiece.rs	15	5	2
fn decode_chain() in tokenizers/src/decoders/strip.rs	29	5	2
fn find_matches() in tokenizers/src/tokenizer/pattern.rs	18	5	2
fn find_matches() in tokenizers/src/tokenizer/pattern.rs	18	5	2
fn find_matches() in tokenizers/src/tokenizer/pattern.rs	28	5	2
fn validate_range + Clone>() in tokenizers/src/tokenizer/normalizer.rs	27	5	2
fn process_encodings() in tokenizers/src/processors/roberta.rs	151	5	3