def preprocess_data()

in src/train_intent.py [0:0]

• 3 lines of code
• 1 McCabe index (conditional complexity)

    def preprocess_data(self, dataset_dict):
        self.data_collator = DataCollatorWithPadding(tokenizer=self.tokenizer)
        return dataset_dict.map(self._tokenize_function, batched=True)