Segmentation, Four Ways

UNet, autoencoder transfer, CLIP fusion & SAM2

Project poster — “One point finds the form.” A blue halftone dog’s head cut out of a red risograph ground, captioned SAM2 single-point segmentation.

05
Year — 2025

Role

MSc project — University of Edinburgh
End-to-end design & evaluation (team of 2)

Do modern shortcuts — self-supervised pre-training, foundation-model features, promptable segmentation — actually beat a classic architecture engineered with care? Four approaches, one dataset, identical evaluation: a custom 20.1M-parameter UNet, an autoencoder-pretrained variant, a CLIP-fused hybrid, and SAM2 trained from scratch. The carefully built UNet more than doubled the baseline. The study’s sharpest insight came from the one that failed.

Problem

Pixel-level segmentation is unforgiving: every ear, tail, and fuzzy outline classified against cluttered backgrounds, 37 breeds, and a dataset riddled with real-world flaws — a 2:1 class imbalance, inconsistent masks, corrupt files, ambiguous boundaries. Everything started with the data: a preprocessing pipeline standardized 7,400 raw images and corrected the imbalance through asymmetric augmentation, producing 11,661 clean, balanced image–mask pairs.

The custom six-stage UNet architecture with skip connections and dropout schedule — Designed deliberately, not by default — 6 stages, 20.1M parameters, sized to the dataset

Approach

The UNet used instance normalization for small-batch stability, a spatial dropout schedule graded by depth, bilinear upsampling to eliminate checkerboard artifacts, and a composite weighted cross-entropy + Dice loss. The autoencoder variant pre-trained to excellent reconstruction, then froze its encoder — and let the results expose whether reconstruction features transfer. The CLIP hybrid fused a frozen ViT-B/16’s global semantics into the UNet bottleneck. SAM2 was trained from scratch, under CPU-only memory constraints, for interactive point- and box-prompt segmentation — then shipped as a working Gradio app.

Dual-pathway architecture fusing frozen CLIP features into the UNet bottleneck — CLIP fusion — global semantics meeting local detail

SAM2 architecture with prompt encoder and memory attention — SAM2 — promptable segmentation, trained from scratch

The autoencoder paradox: near-perfect reconstruction, and not a single cat pixel found. What a network learns depends entirely on what you ask it to do.

Three-model prediction comparison on test samples with color-coded error maps — Same inputs, three verdicts — with error maps for every claim

Results

Custom UNet mIoU: 0.69 vs 0.33 baseline
SAM2 Dice — one click: 0.889
SAM2 pixel accuracy: 91.7%
Curated training pairs: 11,661
Perturbation families tested: 8
Architectures compared: 4

The Gradio web app segmenting a pet from a single point prompt — One click segments a pet — the shipped Gradio app

The Gradio web app segmenting from a two-click bounding box prompt — Box prompts for tighter control

Findings

Simplicity, engineered well, beat sophistication: the plain UNet outperformed both foundation-model fusion and self-supervised pre-training. CLIP fusion underperformed for identifiable reasons, yet uniquely succeeded on breed edge cases the UNet misread. And a systematic robustness study across eight corruption families mapped the exact boundary between graceful degradation and collapse.

In collaboration with

University of Edinburgh

Next case study

06 / 11