utils_nlp/models/transformers/question_answering.py [561:629]: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - ): """ Fine-tune pre-trained transofmer models for question answering. Args: train_dataloader (Dataloader): A PyTorch DataLoader to be used for training. num_epochs (int, optional): Number of training epochs. Defaults to 1. max_steps (int, optional): Total number of training steps. If set to a positive value, it overrides num_epochs. Otherwise, it's determined by the dataset length, gradient_accumulation_steps, and num_epochs. Defualts to -1. gradient_accumulation_steps (int, optional): Number of steps to accumulate before performing a backward/update pass. Default to 1. num_gpus (int, optional): The number of GPUs to use. If None, all available GPUs will be used. If set to 0 or GPUs are not available, CPU device will be used. Defaults to None. gpu_ids (list): List of GPU IDs to be used. If set to None, the first num_gpus GPUs will be used. Defaults to None. local_rank (int, optional): Local_rank for distributed training on GPUs. Defaults to -1, which means non-distributed training. weight_decay (float, optional): Weight decay to apply after each parameter update. Defaults to 0.0. learning_rate (float, optional): Learning rate of the AdamW optimizer. Defaults to 5e-5. adam_epsilon (float, optional): Epsilon of the AdamW optimizer. Defaults to 1e-8. warmup_steps (int, optional): Number of steps taken to increase learning rate from 0 to `learning rate`. Defaults to 0. fp16 (bool): Whether to use 16-bit mixed precision through Apex Defaults to False fp16_opt_level (str): Apex AMP optimization level for fp16. One of in ['O0', 'O1', 'O2', and 'O3'] See https://nvidia.github.io/apex/amp.html" Defaults to "01" checkpoint_state_dict (dict): Checkpoint states of model and optimizer. If specified, the model and optimizer's parameters are loaded using checkpoint_state_dict["model"] and checkpoint_state_dict["optimizer"] Defaults to None. verbose (bool, optional): Whether to print out the training log. Defaults to True. seed (int, optional): Random seed used to improve reproducibility. Defaults to None. cache_model (bool, optional): Whether to save the fine-tuned model. If True, the fine-tuned model is saved to a `fine_tuned` folder under of the `cache_dir` of AnswerExtractor. Defaults to True. """ # init device and optimizer device, num_gpus, amp = self.prepare_model_and_optimizer( num_gpus=num_gpus, gpu_ids=gpu_ids, local_rank=local_rank, weight_decay=weight_decay, learning_rate=learning_rate, adam_epsilon=adam_epsilon, fp16=fp16, fp16_opt_level=fp16_opt_level, checkpoint_state_dict=checkpoint_state_dict, ) # compute the max number of training steps max_steps = compute_training_steps( - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - utils_nlp/models/transformers/sequence_classification.py [242:305]: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - ): """ Fine-tunes a pre-trained sequence classification model. Args: train_dataloader (Dataloader): A PyTorch DataLoader to be used for training. num_epochs (int, optional): Number of training epochs. Defaults to 1. max_steps (int, optional): Total number of training steps. If set to a positive value, it overrides num_epochs. Otherwise, it's determined by the dataset length, gradient_accumulation_steps, and num_epochs. Defualts to -1. gradient_accumulation_steps (int, optional): Number of steps to accumulate before performing a backward/update pass. Default to 1. num_gpus (int, optional): The number of GPUs to use. If None, all available GPUs will be used. If set to 0 or GPUs are not available, CPU device will be used. Defaults to None. gpu_ids (list): List of GPU IDs to be used. If set to None, the first num_gpus GPUs will be used. Defaults to None. local_rank (int, optional): Local_rank for distributed training on GPUs. Defaults to -1, which means non-distributed training. weight_decay (float, optional): Weight decay to apply after each parameter update. Defaults to 0.0. learning_rate (float, optional): Learning rate of the AdamW optimizer. Defaults to 5e-5. adam_epsilon (float, optional): Epsilon of the AdamW optimizer. Defaults to 1e-8. warmup_steps (int, optional): Number of steps taken to increase learning rate from 0 to `learning rate`. Defaults to 0. fp16 (bool): Whether to use 16-bit mixed precision through Apex Defaults to False fp16_opt_level (str): Apex AMP optimization level for fp16. One of in ['O0', 'O1', 'O2', and 'O3'] See https://nvidia.github.io/apex/amp.html" Defaults to "01" checkpoint_state_dict (dict): Checkpoint states of model and optimizer. If specified, the model and optimizer's parameters are loaded using checkpoint_state_dict["model"] and checkpoint_state_dict["optimizer"] Defaults to None. verbose (bool, optional): Whether to print out the training log. Defaults to True. seed (int, optional): Random seed used to improve reproducibility. Defaults to None. """ # init device and optimizer device, num_gpus, amp = self.prepare_model_and_optimizer( num_gpus=num_gpus, gpu_ids=gpu_ids, local_rank=local_rank, weight_decay=weight_decay, learning_rate=learning_rate, adam_epsilon=adam_epsilon, fp16=fp16, fp16_opt_level=fp16_opt_level, checkpoint_state_dict=checkpoint_state_dict, ) # compute the max number of training steps max_steps = compute_training_steps( - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -