August 30, 2026 · 8 min read
State of Vertex AGI — August 30, 2026
A progress report and first public look at our from-scratch pretrain, written 5:13 AM EDT, August 30, 2026.
The reveal: we're pretraining a model from scratch
Alongside the fine-tuned Prism and Amethyst families, we've been quietly running a from-scratch pretraining run — not a fine-tune, not a distillation, a real model trained from randomly-initialized weights on raw token data. This is the first time we're talking about it publicly.
Architecture
A DeepSeek-style fine-grained mixture-of-experts, sized deliberately small to train on a single consumer GPU.
Total parameters
1,031.0M
Active parameters / token
394.0M
Hidden size
1,024
Layers
20
Attention
16Q / 4KV
Context length
2,048
Routed experts
24
Shared experts
2
Expert FFN size
640
Router
top-6 of 24
Vocabulary
32,000
The training stack: FP16 mixed precision (autocast + GradScaler); 8-bit AdamW (bitsandbytes) to keep optimizer state small; Full gradient checkpointing; Gradient accumulation; Single NVIDIA GTX 1660 Ti, 6GB VRAM — a Windows desktop, not a cluster. All specifically tuned to fit a ~1B-parameter MoE onto a single NVIDIA GTX 1660 Ti with 6GB of VRAM. No multi-GPU cluster, no cloud training credits — this is running on a Windows gaming desktop under someone's desk.
Data
100,003,832 tokens of packed training data, processed in steps of 4,096 tokens each, for a total training budget of 24,414 optimizer steps.
Progress as of August 30, 2026, 5:13 AM EDT
step 16,440 / 24,414 (67.3% complete) train loss: ~4.80–4.93 learning rate: 7.39e-05 (cosine decay from 3e-4, past warmup) throughput: ~48 tokens/sec GPU: 100% utilization, ~52W of a 120W budget, 51°C
At the current pace, the run has roughly 10 days of wall-clock time remaining to complete all 24,414 steps. It has already survived two full power/network outages and resumed cleanly from checkpoint both times — resiliency was a first-class design goal from the start, not an afterthought.
This is genuinely one of the more unusual training setups we've run: a real MoE architecture, trained from zero, entirely unattended, on hardware you could buy at a big-box electronics store.
The model family so far
While the from-scratch run has been training in the background, we've shipped a growing family of small, purpose-built fine-tunes — the philosophy throughout has been small, cheap, and good at exactly one thing, distilled from frontier teacher models via LoRA rather than trained from scratch.
Prism — Single-purpose specialists
| Model | Base | Result |
|---|---|---|
| Prism Safety 1 Micro | Qwen3-1.7B | 78.4% accuracy, 86.5% recall — vs. 84.4% / 90.8% for a 5x larger frontier guard model |
| Prism Caption 1 Micro | Gemma-3-1B | 0/20 bad-format outputs (base: 3/20) |
| Prism Caption 1.5 Micro | Qwen3-0.6B | 0/24 format issues, 23/24 within spec |
| Prism Caption 2 Micro | LFM2-700M | 0/275 formatting issues, 273/275 relevant across three eval rounds at increasing scale (24 → 145 → 275 held-out topics) |
| Prism Creative 1 Mini | Qwen3-4B | Coherent, in-genre output with zero leaked planning text or repetition loops |
| Prism Creative 1.5 Mini | Qwen3-4B | 3.2x lower repetition than base; consistent constraint-following on tightening and rewrite tasks |
| Prism Roleplay 1 Small | Qwen3-8B | 100% suppression of leaked reasoning text (base: 100% leaked); 3x lower repetition |
| Prism Roleplay 1.5 Small | Qwen3-8B | 83% blind-judge win rate vs. base (10W–2L–4T across 16 varied scenes) |
| Prism Creative 2 Mini | Qwen3-4B | In data generation now — distilled from a rotating multi-teacher mix |
Amethyst — General-purpose chat
| Model | Base | Result |
|---|---|---|
| Amethyst 1 Mini | Gemma-3-4B | First-generation validation of the distillation pipeline |
| Amethyst 1 Small | Llama-3.1-8B | Same training data, larger base model |
| Amethyst 1.5 Mini | Gemma-3-4B | 16/16 tool-call correctness on held-out prompts (base: 5/16) |
Copal — Agentic tool use
| Model | Base | Result |
|---|---|---|
| Copal 1 Mini | Gemma-3-4B | 11/12 appropriate tool use on held-out tasks (base: 8/12), zero parse errors |
Aquamarine — Code specialist
| Model | Base | Result |
|---|---|---|
| Aquamarine | Qwen3-4B-Instruct-2507 | In active data generation and training |
All models ship in both MLX (4-bit) and GGUF (Q4_K_M) formats in a single repo, so they run equally well on Apple Silicon or via llama.cpp-based runtimes (LM Studio, Ollama, and similar).
What we've learned along the way
A few honest findings from this cycle worth sharing:
Synthetic data teaches form, not capability
Narrow synthetic distillation reliably fixes formatting and behavior — a title-length spec, suppressed reasoning leakage, correct tool timing — but doesn't reliably add knowledge the base model lacked. Real, human-annotated data is what moved raw accuracy: Prism Safety's BeaverTails recall went from 13% to 79.5% only after mixing in real labeled data.
Base model choice can matter more than fine-tuning
An untuned Qwen3-1.7B outperformed multiple fine-tuned Gemma-based safety classifiers on held-out benchmarks. Picking the right foundation is sometimes a bigger lever than the training run itself.
Held-out evals are non-negotiable
Early evaluation passes that sampled from the same template bank used for training scored close to 100% — because the model had simply memorized the templates. Every eval we publish uses hand-authored or independently-sourced held-out data with verified zero overlap against training.
Small models plus consumer hardware go further than expected
Every fine-tune we've shipped — and the from-scratch 1B-parameter MoE pretrain — runs on Apple Silicon or a single 6GB gaming GPU. No cluster required.
What's next
- Finishing the from-scratch pretrain (~10 days remaining at current throughput) and publishing an honest evaluation once it completes
- Continued iteration on Prism Creative 2 and the Prism Caption line
- More of the same: small, well-evaluated, single-purpose models, built in the open, on hardware anyone can actually buy
— The Vertex AGI team