Vizuara AI Labs · preference-aligned (DPO v2)

Gemma 2B DPO

Gemma 2 2B (legal instruction model) aligned with QLoRA-DPO on 4,000 preference pairs.

2.6B

parameters

4,000

pref pairs

QLoRA

method

β=0.1

dpo

Validation metrics along the Gemma 2 2B lineage

Each perplexity is measured on that stage's own validation set, so read the trend as 'how well the model fits its own stage's data', not as one curve on one dataset. DPO and RLAIF optimize preferences rather than likelihood, so they log preference margin and reward instead of perplexity. Click a stage to open that model.

Base

n/a

Google's weights, not trained by us

→

QA SFT

ppl 4.26

QA val

→

Instruct

ppl 1.90

instruction val

→

DPO

QLoRA-DPO

preference-trained, no ppl

RLAIF

loss 0.63→0.27

best-of-4 SFT, no ppl

→

RAFT on DPO

ppl 1.22

RAFT val

RAFT on RLAIF

ppl 1.25

RAFT val

instruction or question

optional: text to work on (attached as TEXT)

ready

The response will appear here.

What this is DPO v2

The Gemma 2 2B instruction model aligned with DPO (direct preference optimization, beta=0.1); the frozen reference is the instruction model, on 4,000 AI-feedback preference pairs spanning closed-book QA and instruction-following failure modes (wrong figures, invented citations, broken format constraints, ignored instructions), prompts held out of every SFT set. Lineage: base → QA SFT → instruction SFT → DPO v2.

Served scale-to-zero on Modal, so the first request may take ~20–60s while the model wakes.

HuggingFace → SLM Arena 21 → 125M base → 500M base →