Research
A mixture-of-experts model, trained from zero.
Not a fine-tune of an existing checkpoint — randomly-initialized weights, trained on raw token data, on a single consumer GPU. This is the first time we're talking about it publicly.
Architecture
A DeepSeek-style fine-grained mixture-of-experts, sized deliberately small: many narrow experts rather than a few large ones, with a shared pair of experts always active alongside the routed six.
Total parameters
1,031.0M
~1.03B
Active parameters / token
394.0M
~394M
Hidden size
1,024
Layers
20
first 2 dense, rest MoE
Attention
16Q / 4KV
GQA, head dim 64
Context length
2,048
tokens
Routed experts
24
6 active per token
Shared experts
2
always active
Expert FFN size
640
fine-grained segmentation
Router
top-6 of 24
0.01-weighted load-balancing loss
Vocabulary
32,000
tokens
Training stack
Every choice here exists to fit a ~1B-parameter MoE onto 6GB of VRAM — no multi-GPU cluster, no cloud training credits.
- FP16 mixed precision (autocast + GradScaler)
- 8-bit AdamW (bitsandbytes) to keep optimizer state small
- Full gradient checkpointing
- Gradient accumulation
- Single NVIDIA GTX 1660 Ti, 6GB VRAM — a Windows desktop, not a cluster
Data
100,003,832 tokens of packed training data, processed in steps of 4,096 tokens each (batch size × gradient accumulation × sequence length), for a total budget of 24,414 optimizer steps.
- Train loss
- 4.7417–5.2442
- Learning rate
- 5.68e-05
- Throughput
- ~48 tok/s
- GPU
- 100% · 54W of 120W · 59°C
Live. At this pace, roughly 6d 19h remain (Mon, Sep 7, 02:15 AM). It has already survived two full power/network outages and resumed cleanly from checkpoint both times.
What we've learned
A few honest findings from this cycle, across both the fine-tunes and the pretrain.
Synthetic data teaches form, not capability
Narrow synthetic distillation reliably fixes formatting and behavior — a title-length spec, suppressed reasoning leakage, correct tool timing — but doesn't reliably add knowledge the base model lacked. Real, human-annotated data is what moved raw accuracy: Prism Safety's BeaverTails recall went from 13% to 79.5% only after mixing in real labeled data.
Base model choice can matter more than fine-tuning
An untuned Qwen3-1.7B outperformed multiple fine-tuned Gemma-based safety classifiers on held-out benchmarks. Picking the right foundation is sometimes a bigger lever than the training run itself.
Held-out evals are non-negotiable
Early evaluation passes that sampled from the same template bank used for training scored close to 100% — because the model had simply memorized the templates. Every eval we publish uses hand-authored or independently-sourced held-out data with verified zero overlap against training.
Small models plus consumer hardware go further than expected
Every fine-tune we've shipped — and the from-scratch 1B-parameter MoE pretrain — runs on Apple Silicon or a single 6GB gaming GPU. No cluster required.
What's next
- Finishing the from-scratch pretrain (~6d 19h remaining at current throughput) and publishing an honest evaluation once it completes
- Continued iteration on Prism Creative 2 and the Prism Caption line
- More of the same: small, well-evaluated, single-purpose models, built in the open, on hardware anyone can actually buy