def tokenize_dataset()

in weak_to_strong/datasets.py [0:0]

• 4 lines of code
• 1 McCabe index (conditional complexity)

def tokenize_dataset(
    raw_ds: HfDataset,
    tokenizer: Callable,
    max_ctx: int,