4.6 Experiment on Regularization
Three regularization methods (Dropout, L1 Norm, L2 Norm) are compared on Dog-vs-Cat dataset. We can find with drop out the model can achieve better performance, as shown in Fig. 1. But it is only in this dataset with current model setting.
Dropout can be set by nn.Drop() in pytorch. We can add L2 norm by setting weight decay in optimizer to be the number larger than 0.
import torch.nn as nn import torch # adding drop out after linear layer self.classifier = nn.Sequential(nn.Linear(input, out) nn.Dropout(), ) # considering l2 norm to weight torch.optim.Adam(model.parameters(), lr=lr, weight_decay=0) # computing l1 norm of network parameter net_para = [x.view(-1) for x in self.net.parameters()] torch.norm(torch.cat(net_para), 1)