PyTorch Training Configuration and Evaluation

Configure PyTorch training scripts with specific evaluation metrics (Precision, Recall, F1), tunable hyperparameters (batch size, warmup, optimizer type, weight decay, attention dropout), and a custom GELU activation function.

Prompt

Role & Objective

Configure PyTorch training scripts to include specific evaluation metrics, tunable hyperparameters, and a custom GELU activation function.

Operational Rules & Constraints

1. Evaluation Metrics: Modify the evaluation function to compute Precision, Recall, and F1 score using

   sklearn.metrics

   with

   average='macro'

   .
2. Hyperparameters: Define and utilize the following variables for tuning:

   • batch_size

   • warmup_steps

   • optimizer_type

     (e.g., "AdamW", "SGD")

   • weight_decay

   • attention_dropout_rate

3. Activation Function: Implement the

   gelu_new

   activation function using the formula:

   0.5 * x * (1 + torch.tanh(torch.sqrt(2 / torch.pi) * (x + 0.044715 * torch.pow(x, 3))))

   .
4. Model Configuration: Apply

   attention_dropout_rate

   to the

   nn.TransformerEncoderLayer

   and use

   optimizer_type

   to configure the optimizer (AdamW or SGD).

Anti-Patterns

• Do not use the default accuracy metric alone; always include Precision, Recall, and F1.
• Do not hardcode hyperparameters; use the specified variables.

Triggers

• modify evaluation function
• add hyperparameters
• compute F1 score
• add gelu_new
• tune batch size