问题：模型训不动，loss不降，验证集炸了

2024年Q2，我接手一个图像分类项目，ResNet50在CIFAR-10上训了50个epoch，训练loss卡在1.8不动，验证集准确率只有62%。调了三天学习率、加Dropout、改BN层，终于把准确率干到94.7%。这篇把踩过的坑和最终方案拆开讲。

方案对比：学习率调度

方案A：固定学习率 + StepLR

初始lr=0.1，每30个epoch乘以0.1。代码简单，但容易陷入局部最优，尤其大batch size时。

方案B：CosineAnnealingLR + Warmup

先用5个epoch线性warmup到lr=0.1，再用余弦退火降到1e-5。收敛更快，最终准确率高3-5%。

方案对比：正则化

方案A：L2正则化（weight decay）

PyTorch中optimizer的weight_decay参数，默认1e-4。对过拟合有效，但太大导致欠拟合。

方案B：Dropout + Label Smoothing

Dropout率0.5，Label Smoothing系数0.1。比纯L2正则化泛化能力更强，尤其小数据集。

方案对比：Batch Normalization

方案A：标准BN

放在Conv后、激活前。训练时用batch统计，推理时用running mean/var。

方案B：BN + SyncBN + 特殊初始化

多卡训练用SyncBN，初始化时BN层的gamma=0.1，防止训练初期梯度爆炸。

完整代码实现

# train.py
import torch
import torch.nn as nn
import torch.optim as optim
from torch.optim.lr_scheduler import CosineAnnealingLR, LinearLR
from torchvision import datasets, transforms, models
import time

# 配置
BATCH_SIZE = 128
EPOCHS = 100
LR_INIT = 0.1
WEIGHT_DECAY = 1e-4
DROPOUT_RATE = 0.5
LABEL_SMOOTHING = 0.1

# 数据加载
transform_train = transforms.Compose([
    transforms.RandomCrop(32, padding=4),
    transforms.RandomHorizontalFlip(),
    transforms.ToTensor(),
    transforms.Normalize((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010))
])
transform_test = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010))
])
train_dataset = datasets.CIFAR10(root='./data', train=True, download=True, transform=transform_train)
test_dataset = datasets.CIFAR10(root='./data', train=False, download=True, transform=transform_test)
train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=BATCH_SIZE, shuffle=True, num_workers=4)
test_loader = torch.utils.data.DataLoader(test_dataset, batch_size=BATCH_SIZE, shuffle=False, num_workers=4)

# 模型：ResNet50 with custom BN init
model = models.resnet50(weights=None, num_classes=10)
# 修改BN初始化
for m in model.modules():
    if isinstance(m, nn.BatchNorm2d):
        m.weight.data.fill_(0.1)  # gamma=0.1
        m.bias.data.zero_()
# 添加Dropout
model.fc = nn.Sequential(
    nn.Dropout(DROPOUT_RATE),
    nn.Linear(2048, 10)
)

# 优化器和调度器
optimizer = optim.SGD(model.parameters(), lr=LR_INIT, momentum=0.9, weight_decay=WEIGHT_DECAY)
warmup_scheduler = LinearLR(optimizer, start_factor=0.1, total_iters=5)
cosine_scheduler = CosineAnnealingLR(optimizer, T_max=EPOCHS-5)
scheduler = torch.optim.lr_scheduler.SequentialLR(optimizer, [warmup_scheduler, cosine_scheduler], milestones=[5])

# 损失函数：Label Smoothing
class LabelSmoothingLoss(nn.Module):
    def __init__(self, classes, smoothing=0.1):
        super().__init__()
        self.confidence = 1.0 - smoothing
        self.smoothing = smoothing
        self.classes = classes
    def forward(self, pred, target):
        pred = pred.log_softmax(dim=-1)
        true_dist = torch.zeros_like(pred)
        true_dist.fill_(self.smoothing / (self.classes - 1))
        true_dist.scatter_(1, target.data.unsqueeze(1), self.confidence)
        return torch.mean(torch.sum(-true_dist * pred, dim=-1))
criterion = LabelSmoothingLoss(classes=10, smoothing=LABEL_SMOOTHING)

# 训练
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model.to(device)
best_acc = 0.0
for epoch in range(EPOCHS):
    model.train()
    running_loss = 0.0
    start_time = time.time()
    for inputs, labels in train_loader:
        inputs, labels = inputs.to(device), labels.to(device)
        optimizer.zero_grad()
        outputs = model(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()
        running_loss += loss.item()
    scheduler.step()
    # 验证
    model.eval()
    correct = 0
    total = 0
    with torch.no_grad():
        for inputs, labels in test_loader:
            inputs, labels = inputs.to(device), labels.to(device)
            outputs = model(inputs)
            _, predicted = torch.max(outputs, 1)
            total += labels.size(0)
            correct += (predicted == labels).sum().item()
    acc = 100.0 * correct / total
    if acc > best_acc:
        best_acc = acc
        torch.save(model.state_dict(), 'best_model.pth')
    print(f'Epoch {epoch+1:3d} | Loss: {running_loss/len(train_loader):.4f} | Acc: {acc:.2f}% | LR: {optimizer.param_groups[0]["lr"]:.6f} | Time: {time.time()-start_time:.2f}s')
print(f'Best accuracy: {best_acc:.2f}%')

效果数据

硬件：RTX 3090, PyTorch 2.1.0, CUDA 12.1, CIFAR-10测试集。

避坑指南

• 坑1：BN层在推理时用batch统计。我踩过：eval模式下忘记调model.eval()，导致BN用当前batch统计，推理结果波动大。必须显式调用model.eval()。
• 坑2：学习率warmup步数太少。初始lr=0.1时，warmup从0.01开始，5个epoch刚好。如果warmup只设1个epoch，前几轮loss直接炸到10+。
• 坑3：Label Smoothing配合Dropout。LS让softmax输出更平滑，Dropout随机丢弃神经元，两者结合后模型收敛更慢，需要增加epoch或调大lr。我试过LS=0.2时，准确率反而降到93.1%。
• 坑4：多卡训练BN统计不一致。用DataParallel时，每个卡独立计算BN统计，导致模型不一致。必须用SyncBN（torch.nn.SyncBatchNorm）或DistributedDataParallel。

原理展开

学习率调度原理

CosineAnnealingLR模拟余弦函数下降，从高lr快速探索，到低lr精细收敛。Warmup防止初期梯度爆炸，因为模型参数随机初始化，大lr导致梯度方向不稳定。

正则化原理

L2正则化等价于权重衰减，限制参数范数。Dropout随机丢弃神经元，相当于训练多个子模型集成。Label Smoothing软化真实标签，防止过拟合，提高泛化。

BN原理

BN对每个batch做归一化，解决内部协变量偏移。训练时维护running mean/var，推理时用全局统计。gamma参数控制缩放，初始值小可以稳定训练。

总结

学习率用Cosine+Warmup，正则化用Dropout+Label Smoothing，BN初始化gamma=0.1，这套组合在CIFAR-10上稳定95%+。代码直接复制跑，注意避坑部分。