openai / tiktoken - Conditional Complexity

Intro

Conditional complexity (also called cyclomatic complexity) is a term used to measure the complexity of software. The term refers to the number of possible paths through a program function. A higher value ofter means higher maintenance and testing costs (infosecinstitute.com).
Conditional complexity is calculated by counting all conditions in the program that can affect the execution path (e.g. if statement, loops, switches, and/or operators, try and catch blocks...).
Conditional complexity is measured at the unit level (methods, functions...).
Units are classified in four categories based on the measured McCabe index: 1-5 (simple units), 6-10 (medium complex units), 11-25 (complex units), 26+ (very complex units).

Learn more...

Conditional Complexity Overall

There are 85 units with 778 lines of code in units (54.6% of code).

0 very complex units (0 lines of code)
0 complex units (0 lines of code)
1 medium complex units (40 lines of code)
6 simple units (136 lines of code)
78 very simple units (602 lines of code)

Legend:

51+

26-50

11-25

6-10

1-5

Alternative Visuals

3D view of all units

Conditional Complexity per Extension

51+

26-50

11-25

6-10

1-5

Conditional Complexity per Logical Component

primary logical decomposition

51+

26-50

11-25

6-10

1-5

Most Complex Units

Top 50 most complex units

Unit	# lines	McCabe index	# params
def read_file_cached() in tiktoken/load.py	40	13	2
def redact_file() in scripts/redact.py	28	9	2
fn _byte_pair_merge() in src/lib.rs	40	8	3
def _find_constructors() in tiktoken/registry.py	24	7	0
def encoding_name_for_model() in tiktoken/model.py	14	7	1
def get_encoding() in tiktoken/registry.py	21	6	1
def decode_with_offsets() in tiktoken/core.py	9	6	2
def visualise_tokens() in tiktoken/_educational.py	14	5	1
def read_file() in tiktoken/load.py	14	5	1
fn _encode_bytes() in src/py.rs	32	5	3
fn _increase_last_piece_token_len() in src/lib.rs	30	5	3
def load_tiktoken_bpe() in tiktoken/load.py	12	4	2
def redact() in scripts/redact.py	11	4	1
fn encode_single_token() in src/py.rs	11	4	2
def _available_plugin_modules() in tiktoken/registry.py	6	3	0
def dump_tiktoken_bpe() in tiktoken/load.py	10	3	3
def main() in scripts/redact.py	7	3	0
fn decode_single_token_bytes() in src/py.rs	9	3	3
def list_encoding_names() in tiktoken/registry.py	6	2	0
def __init__() in tiktoken/_educational.py	5	2	5
def encode() in tiktoken/_educational.py	8	2	3
def decode_bytes() in tiktoken/_educational.py	2	2	2
def decode_tokens_bytes() in tiktoken/_educational.py	2	2	2
def from_tiktoken() in tiktoken/_educational.py	6	2	1
def encode_ordinary() in tiktoken/core.py	6	2	2
def encode_single_token() in tiktoken/core.py	4	2	2
def decode_tokens_bytes() in tiktoken/core.py	2	2	2
def _encode_single_piece() in tiktoken/core.py	4	2	2
def _encode_only_native_bpe() in tiktoken/core.py	6	2	2
def __getstate__() in tiktoken/core.py	10	2	1
def __setstate__() in tiktoken/core.py	6	2	2
def _special_token_regex() in tiktoken/core.py	3	2	1
def decode_data_gym() in tiktoken/load.py	2	2	1
fn py_encode_ordinary() in src/py.rs	3	2	3
fn py_encode() in src/py.rs	12	2	4
fn encode_to_tiktoken_buffer() in src/py.rs	14	2	4
fn py_encode_with_unstable() in src/py.rs	19	2	4
fn encode_single_piece() in src/py.rs	6	2	2
fn py_decode_bytes() in src/py.rs	6	2	3
fn decode_bytes() in src/lib.rs	14	2	2
def gpt2() in tiktoken_ext/openai_public.py	14	1	0
def r50k_base() in tiktoken_ext/openai_public.py	12	1	0
def p50k_base() in tiktoken_ext/openai_public.py	12	1	0
def p50k_edit() in tiktoken_ext/openai_public.py	12	1	0
def cl100k_base() in tiktoken_ext/openai_public.py	18	1	0
def o200k_base() in tiktoken_ext/openai_public.py	23	1	0
def decode() in tiktoken/_educational.py	2	1	2
def train() in tiktoken/_educational.py	3	1	3
def bpe_encode() in tiktoken/_educational.py	2	1	0
def bpe_train() in tiktoken/_educational.py	2	1	0