Classification, segmentation and detection in PyTorch

A CNN with three convolutional layers, trained from scratch on CIFAR-100, reached 30.62% to 42.13% test accuracy across three optimiser settings, while its training accuracy rose to about 80% in the two stronger runs.

I then adapted ResNet18, pretrained on ImageNet, to the 37 breeds of Oxford-IIIT Pet, in three runs of ten epochs:

Feature extraction

backbone frozen; only a new 37-class output layer trained

Test accuracy 87.24%

Fine-tuning

every layer trained at a learning rate of 0.0001, continuing from the run above

Test accuracy 90.32%

Augmentation

random rotations and horizontal flips, fine-tuned again from the ImageNet weights

Test accuracy 88.28%

Fine-tuning gave the best result, with training accuracy at 100%. Augmentation did not raise test accuracy, but that run restarted from the ImageNet weights, so the comparison is not clean.

The last exercise compared a Vision Transformer, ViT-B/16, with a CNN, VGG16-BN, on CIFAR-10. Both started from ImageNet weights with a new 10-class output layer, and I trained all parameters of each the same way: 224 × 224 inputs, no augmentation, Adam at 0.0001, batches of 64.

ViT-B/16 finished 2.4 to 3.0 percentage points ahead, at about 2.7 times the time per epoch. Over ten epochs, the test loss of VGG16-BN rose from 0.216 after the second epoch to 0.270, while that of ViT-B/16 stayed below 0.20. That holds for this dataset and setup; it does not rank the two architectures.

Segmentation

I built a U-Net from scratch and trained it for 15 epochs on SBD, at 256 × 256 pixels in 21 classes, holding out a fifth of the 8,498 training images for evaluation.

Pixel accuracy 75.79%

Detection

I fine-tuned torchvision’s Faster R-CNN, with a ResNet-50 FPN backbone and COCO weights, on the 2,501 images of the VOC 2007 train split, with a new predictor for its 20 classes plus background.

Training loss, SGD, 10 epochs 0.644 to 0.161

Of three optimiser setups, AdamW reached the lowest training loss, 0.144. But training loss is not detection quality: I computed no mAP and only checked predictions by eye. Likewise, pixel accuracy includes the background, and without IoU it does not show how well each class was segmented.

What I took from these exercises: write the metric and the split beside every number, and compare models only under one setup.

Two things I would change. In the classification exercises the test split doubled as the validation set checked after each epoch, so I report the last epoch, not the best; a separate validation split would fix that. And the comparison had one run per model without a fixed seed: after five epochs, two runs of the same VGG16-BN setup stood at 92.84% and 93.35%. Several seeds, plus IoU and mAP, would make these comparisons firmer.