Fine-tune times

What can a P5200 train?

Every open-weight model a 16 GB Quadro P5200 can hold for a QLoRA fine-tune, and how long the run takes at your dataset size. Set your numbers; the table recomputes.

Checked 24 October 2026 · HP Z VR Backpack G1 · QLoRA: 4-bit base + LoRA adapters · throughput estimated, not measured

Defaults reproduce the reference run: 300 × 400 × 3 = 360K tokens. Batch sizes assume 2K-token sequences.

Dataset360K tokexamples × tokens × epochs
Gemma E4B run16 minat mid estimate
E4B passes/hr3,375400-token examples
Card bandwidth230.4 GB/s16 GB GDDR5 · sm_61
Effective ~115 GB/s½×FP32 dequant penalty
Model▲ Country▲ Released▲ QLoRA GB▲ Fits 16 GB?▲ Est. tok/s▲ Your run▲ Batch @ 2K▲
Gemma 4 E2BMatFormer, 2B effective · Apache 2.0 USAGoogle Mar 2026 4.3official 4-bit table ×1.5 Fits 300–900 10 min 3–4
SmolLM3 3B3.1B dense · Apache 2.0 USAHugging Face Jul 2025 5.4Q4_K_M 1.9 GB + overhead Fits 200–750 13 min 3–4
Phi-4-mini3.8B dense · MIT USAMicrosoft Feb 2025 5.9Q4_K_M ~2.4 GB + overhead Fits 155–630 15 min 2–4
Qwen3.5 4B4B dense, vision · Apache 2.0 ChinaAlibaba Mar 2026 6.0Q4_K_M ~2.5 GB + overhead Fits 150–600 16 min 2–4
Gemma 4 E4BMatFormer, 4B effective · Apache 2.0 USAGoogle Mar 2026 6.8official 4-bit table ×1.5 Fits 150–600 16 min 3–4
Llama 3.1 8B8.0B dense · Llama Community USAMeta Jul 2024 8.4Q4_K_M 4.9 GB + overhead Fits 75–300 32 min 1–2
Qwen3 8B8.2B dense · Apache 2.0 ChinaAlibaba Apr 2025 8.5Q4_K_M 5.0 GB + overhead Fits 75–300 32 min 1–2
Gemma 4 12B12B Unified, encoder-free · Apache 2.0 USAGoogle Jun 2026 10.1official 4-bit table ×1.5 Tight · batch 1 50–200 48 min 1
Qwen3 14B14.8B dense · Apache 2.0 ChinaAlibaba Apr 2025 12.5Q4_K_M 9.0 GB + overhead Tight · batch 1 45–170 56 min 1
gpt-oss-20b21B MoE, 4 of 32 experts · Apache 2.0 USAOpenAI Aug 2025 15.1Q4_K_M 11.6 GB + overhead Boundary · batch 1 70–280 34 min 1 · 1K ctx
Mistral Small 3 24B24B dense · Apache 2.0 FranceMistral AI Jan 2025 18.2Q4_K_M ~14.7 GB + overhead Offload ~3 GB 15–40 3.6 h 1 · spill
Gemma 4 26B-A4B26B MoE, 4B active · Apache 2.0 USAGoogle Mar 2026 21.6official 4-bit table ×1.5 Rent a 24 GB card 10–40 4.0 h —
run minutes = (examples × tokens per example × epochs) ÷ training tok/s ÷ 60

Estimates, not measurements

tok/s figures are derived, never measured on this card: a 4B-class baseline of 150–600 tok/s from the card’s bandwidth and FLOPs, scaled inversely with parameter count. A 10-iteration smoke test on the real machine gives you the true number — then multiply by 3,600.

Why Pascal trains slowly

The P5200 (GP104, sm_61) has no tensor cores, so QLoRA matmuls dequantize 4-bit weights to FP32 — roughly a 2× compute penalty. Effective bandwidth ~115 GB/s. A 16 GB M6 Mac mini runs the same E4B job ~1.3–1.7× faster; the card’s edge is CUDA and dedicated VRAM.

The memory math

Train footprint = 4-bit weights + ~3.5 GB (dequant buffers, activations, adapters, optimizer). Gemma rows use Google’s official 4-bit table ×1.5 — per-layer embeddings (E-series) and all-expert MoE loads make naive per-parameter math underestimate them.

Toolchain: pin the wheels

Default PyPI torch wheels (cu128+) carry no sm_61 kernels — first kernel launch fails with “no kernel image available.” The Unsloth installer (unsloth.ai/install.sh) detects pre-Turing cards and pins cu126 wheels, which still do. Fixed upstream September 2026.

LoRA teaches style, not facts

Fine-tuning changes behavior, format and voice; it does not inject knowledge. If the model needs to know documents or records, that’s retrieval (RAG) on top. Data quality beats quantity: 300 to a few thousand clean, consistently formatted examples outperform larger noisy sets.

What the 40 GB RAM buys

The whole dataset in memory, and a safety valve that keeps 12–14B runs from crashing mid-train. It adds no speed: spilled layers crawl over the MXM PCIe link, which is why the 24B row drops to 15–40 tok/s.

Thermals

The Z VR Backpack is a worn-chassis design pulling air through bottom and side vents, sustained ~150 W. Hard surface, clean fans, and don’t schedule multi-hour jobs on it.

Rent, don’t suffer

Anything past 14B belongs on a rented 24 GB card: single-digit dollars for a one-shot run, finished in ~20 minutes. Ollama Cloud serves its catalog only — it cannot host custom fine-tuned weights.

Sources & method