DreamerV3 — Mastering Diverse Domains through World Models

Abstract

Published as arXiv:2301.04104v2, 17 April 2024.

Developing a general algorithm that learns to solve tasks across a wide range of applications has been a fundamental challenge in artificial intelligence. Configuring current reinforcement learning algorithms for new application domains requires significant human expertise and experimentation. We present DreamerV3, a general algorithm that outperforms specialized methods across over 150 diverse tasks, with a single configuration. Dreamer learns a model of the environment and improves its behavior by imagining future scenarios. Robustness techniques based on normalization, balancing, and transformations enable stable learning across domains. Applied out of the box, Dreamer is the first algorithm to collect diamonds in Minecraft from scratch without human data or curricula.

— abridged from the authors' abstract

01 — Core idea

Learning by imagining

Most agents learn by trial and error in the real environment, which is slow and expensive. DreamerV3 instead builds a world model — a compressed simulator learned from its own experience — and trains its decision-making inside it.

In plain English

Think of a chess player who has played a hundred games. They no longer move pieces to test an idea — they picture the board and play the next ten moves in their head. DreamerV3 builds that mental board for any environment you plug it into, and practising there is millions of times cheaper than practising for real.

The problem it solves

Before this work, reinforcement learning was a collection of specialists — one algorithm for Atari, another for robot control, another for sparse-reward exploration — each needing an expert to re-tune dozens of knobs per problem. That tuning cost is the real bottleneck: it puts the field out of reach of anyone who is not already an RL researcher with a compute budget.

Before

Tune for every domain

Moving from video games to robotics meant weeks of hyperparameter search — a new configuration, written by a specialist, for every task.

Why it broke

Magnitudes differ wildly

Rewards may be 0.001 or 10,000. Images versus joint angles. Dense versus sparse feedback. A loss tuned for one magnitude diverges or stalls on another.

DreamerV3

Normalise everything

A small toolkit of transformations keeps every signal in a predictable range, so one configuration works everywhere — and still beats the specialists.

02 — Architecture

The software stack

The whole system is seven layers. The bottom two move data around, the middle three are the neural networks that do the thinking, and the top two are what let one setting work on every problem. Click a layer to open it.

If you remember one thing: the intelligence lives in L3, the agent's mental picture of the world — and the ability to drop that same agent into any new problem lives in L6.

Aside · a different way to build layer L3

How JEPA works

DreamerV3's world model learns by redrawing what it sees. That works, but it spends effort on detail nobody could predict — the exact pattern of leaves on a tree, the flicker of static. JEPA, the Joint-Embedding Predictive Architecture, takes the other road: it never redraws anything. It turns both the present and the future into short codes, and only asks whether its guessed code matches the real one.

The everyday version

Watch a friend put a glass near the edge of a table. You can predict that it will fall and break. What you cannot predict — and do not try to — is the exact shape of every shard. DreamerV3 tries to draw the shards. JEPA only predicts “broken glass on the floor”, and that turns out to be both easier to learn and more useful.

Why it matters here

The ablation later on this page shows that DreamerV3's strength comes mostly from the redrawing loss, not from chasing reward. JEPA asks the obvious follow-up: if the useful signal is “predict what comes next”, can we keep that and throw away the expensive pixel-painting? V-JEPA 2 was trained on over a million hours of internet video this way, then given actions so a robot could plan with it.

Question	Redraw the picture (DreamerV3)	Predict the code (JEPA)
What does it predict?	The next picture, pixel by pixel.	Only the short code that summarises the next picture.
What does it do with unpredictable detail?	Tries to draw it anyway, and wastes capacity doing so.	Drops it. If it cannot be known, it need not be in the code.
How does it avoid cheating?	It cannot cheat — the picture is the answer key.	It could output the same code for everything, so it needs a safeguard: a slowly updated copy of the encoder for the target side.
Where you have seen it	DreamerV3, and most world models up to about 2023.	I-JEPA for images, V-JEPA for video, and V-JEPA 2, which adds actions so a robot can plan.

Neither approach is simply better. Redrawing gives a picture you can actually look at, which is how the video predictions in this paper were checked. Predicting codes scales to messy real-world video, where drawing every pixel is hopeless. Both are world models; they differ only in what they consider worth predicting.

03 — Data flow

One turn of the training loop

The agent keeps playing while all three networks learn at the same time. Here is one full turn of that cycle.

1

Step 1

Act

The agent picks a move. The world answers with a new picture and a score. Both go straight into the store.

2

Step 2

Replay

Pull out a batch of past experience — sixteen stretches of sixty-four moments each.

3

Step 3

Fit the RSSM

Train the mental picture: redraw what it saw, predict the score, and get better at guessing what comes next.

4

Step 4

Imagine

From every moment in the batch, daydream fifteen steps ahead. Thousands of futures, and not one move played for real.

5

Step 5

Improve

The critic learns how good those imagined futures were; the actor learns to steer towards the good ones. Then start again.

and repeat — about 1.5 million learning steps over 200 million moves on Atari

Why this split is the clever part

Real experience is scarce — a robot arm moves at the speed of physics. Imagined experience is nearly free. So DreamerV3 spends real interactions on learning what the world is like and imagined ones on learning what to do in it. That division is why it needs an order of magnitude less data than the algorithms it beats.

04 — Components

Three neural networks, three jobs

Each one has a single job. Read the shaded box first — that is the idea in everyday words. The equations beside it are the same thing written precisely.

Recurrent State-Space Model

Six sub-networks trained end to end: an encoder turns an observation into a stochastic discrete code, a block-diagonal GRU propagates memory, and the remaining heads predict everything the agent needs.

\begin{aligned} \text{Sequence model:} \quad & h_t = f_\phi(h_{t-1},\, z_{t-1},\, a_{t-1}) \\[2pt] \text{Encoder:} \quad & z_t \sim q_\phi(z_t \mid h_t,\, x_t) \\[2pt] \text{Dynamics predictor:} \quad & \hat z_t \sim p_\phi(\hat z_t \mid h_t) \\[2pt] \text{Reward predictor:} \quad & \hat r_t \sim p_\phi(\hat r_t \mid h_t,\, z_t) \\[2pt] \text{Continue predictor:} \quad & \hat c_t \sim p_\phi(\hat c_t \mid h_t,\, z_t) \\[2pt] \text{Decoder:} \quad & \hat x_t \sim p_\phi(\hat x_t \mid h_t,\, z_t) \end{aligned}

Objective — β_pred=1, β_dyn=1, β_rep=0.1

\mathcal{L}(\phi) = \mathbb{E}_{q_\phi}\!\left[\sum_{t=1}^{T}\big(\beta_{\mathrm{pred}}\mathcal{L}_{\mathrm{pred}} + \beta_{\mathrm{dyn}}\mathcal{L}_{\mathrm{dyn}} + \beta_{\mathrm{rep}}\mathcal{L}_{\mathrm{rep}}\big)\right]

\begin{aligned} \mathcal{L}_{\mathrm{dyn}} &= \max\big(1,\; \mathrm{KL}\!\left[\,\mathrm{sg}(q_\phi(z_t \mid h_t, x_t))\;\|\;p_\phi(z_t \mid h_t)\,\right]\big) \\[4pt] \mathcal{L}_{\mathrm{rep}} &= \max\big(1,\; \mathrm{KL}\!\left[\,q_\phi(z_t \mid h_t, x_t)\;\|\;\mathrm{sg}(p_\phi(z_t \mid h_t))\,\right]\big) \end{aligned}

Why discrete codes?

The latent is a vector of categorical variables — a row of dials with fixed notches. Discreteness lets the dynamics predictor be factorised, which makes imagination fast, and stops the model smuggling in continuous detail it cannot actually predict.

In plain English

The world model is a compression engine plus a fortune teller. It squeezes a 64×64 image into a few hundred bits, then guesses what those bits will look like one step later — before seeing the next frame. Guess well, and you have understood the rules of the world.

The model state s = {h, z}: a deterministic recurrent path carrying memory, and a stochastic discrete latent carrying what the current observation added.

The free-bits trick

Earlier Dreamers needed a different regulariser strength for a detailed 2D game than for a cluttered 3D world. Clipping both KL losses at 1 nat switches them off once small enough, so learning refocuses on reconstruction. One setting now covers both extremes.

It really does predict video

Given five context frames and an action sequence, the model renders 45 future frames it has never seen. Mazes stay geometrically consistent and a quadruped robot's legs move plausibly.

05 — The central contribution

The robustness toolkit

Six small choices. Together they are the entire reason one setting works on a robot arm, a 1980s arcade game and an open-world sandbox without anything being adjusted.

Transformation

symlog / symexp

A squashing function. It leaves small numbers almost untouched but shrinks huge ones, so a score of 10,000 becomes about 9. Unlike an ordinary logarithm it also handles negative numbers, which matters because rewards can be negative.

\mathrm{symlog}(x)=\mathrm{sign}(x)\ln(|x|+1)

Transformation

symexp twohot loss

Instead of guessing a number directly, the network picks between buckets — and the buckets get wider as the values get bigger. The size of the correction no longer depends on how big the answer is, so a reward of 10,000 cannot blow up training.

\hat y=\mathrm{softmax}(f(x))^{\!\top} B

Balancing

KL balancing + free bits

The model is pushed to make its predictions and its observations agree — but only up to a point. Once they are close enough, that pressure switches off and the model spends its effort on seeing clearly instead. This is what lets one setting suit both a simple 2D game and a cluttered 3D world.

\max\big(1,\ \mathrm{KL}[\,q\,\|\,p\,]\big)

Normalisation

Percentile return normalisation

Scores are rescaled so they sit in a familiar range, using the middle 90% of recent results and ignoring lucky outliers. Crucially they are only ever scaled down, never up — so when rewards are rare the agent keeps exploring rather than chasing noise.

S=\mathrm{EMA}\big(\mathrm{Per}(R,95)-\mathrm{Per}(R,5)\big)

Regularisation

1% unimix categoricals

Every choice the network makes keeps a 1% chance of anything. Nothing is ever ruled out completely, which stops the maths hitting infinity and removes the training spikes the authors saw in earlier runs.

p = 0.99\,p_\theta + 0.01\,\mathcal{U}

Optimisation

AGC + LaProp

No part of the network may change by more than 30% of its own current size in a single step. Because the limit is relative, it never needs adjusting when anything else changes.

\|g\| \le 0.3\,\|W\|

Figure — symlog versus identity versus log

Near the origin symlog behaves like the identity, so small targets are untouched. Far from it the function grows logarithmically, so a target of 10,000 becomes 9.21 and its squared-error gradient stays sane. Unlike log, which is undefined for x ≤ 0, symlog is defined and symmetric everywhere — essential because rewards can be negative.

A question for the room. One game awards +1 per coin. Another awards +10,000 per coin. Should the agent play differently? Plainly not — it is the same game. Yet almost every earlier algorithm would. Everything in this toolkit exists to make that answer true in code.

06 — The algorithm

One iteration, written out

Twenty-one lines covering all three networks. Every robustness technique from the previous section appears in exactly one of them.

DreamerV3 · one iteration

line 1

The agent never stops playing. Collection and training run concurrently, paced by the replay ratio.

what is active

The fixed configuration

identical across all 150+ tasks

Batch shape	16 × 64
Learning rate	4×10⁻⁵
Replay capacity	5×10⁶
Imagination horizon	15
Discount horizon	333
Return lambda	0.95
Dynamics / representation scale	1.0 / 0.1
Free nats	1
Actor entropy	3×10⁻⁴
Unimix	1%
Return-norm range / limit	Per(95)−Per(5) / 1
Critic EMA decay	0.98
Optimiser	LaProp + AGC(0.3)

symlog and its inverse

\mathrm{symlog}(x) = \mathrm{sign}(x)\ln(|x|+1)

\mathrm{symexp}(x) = \mathrm{sign}(x)\big(e^{|x|}-1\big)

A network predicts symlog(y) under a squared loss and reads out symexp(f(x)). Unlike a plain logarithm it is defined for negative values and symmetric about the origin; unlike normalising by running statistics it introduces no non-stationarity.

the twohot encoding

\hat y = \mathrm{softmax}(f(x))^{\!\top} B, \quad B = \mathrm{symexp}\big([-20 \ldots +20]\big)

\mathcal{L}(\theta) = -\,\mathrm{twohot}(y)^{\!\top}\log \mathrm{softmax}(f(x,\theta))

The loss depends only on the probabilities assigned to bins, never on the continuous values attached to them — which is what decouples gradient magnitude from target magnitude.

return normalisation

S = \mathrm{EMA}\big(\mathrm{Per}(R^\lambda, 95) - \mathrm{Per}(R^\lambda, 5),\; 0.99\big)

\mathcal{L}(\theta) = -\sum_t \mathrm{sg}\!\left(\frac{R^\lambda_t - v_\psi(s_t)}{\max(1, S)}\right)\log \pi_\theta(a_t \mid s_t) + \eta\,\mathrm{H}[\pi_\theta]

Normalising returns rather than advantages preserves information about how often rewards occur. The denominator limit means small returns pass through untouched, so noise is never amplified under sparse rewards.

07 — Evidence

Results across eight benchmarks

In every row below, DreamerV3 uses exactly the same settings. Each competitor was built and tuned for that one benchmark. DreamerV3 wins or ties almost everywhere.

Benchmark	Tasks	Budget	Tuned expert baselines	Outcome
Atari	57	200M frames	MuZero, Rainbow, IQN, PPO	Beats MuZero at a fraction of the compute
ProcGen	16	50M frames	PPG, Rainbow, PPO	Matches tuned PPG, beats Rainbow
DMLab	30	100M frames	IMPALA, R2D2+ (at 1B steps)	Exceeds both with 10× less data
Atari 100k	26	400K frames (≈2 h)	IRIS, TWM, SPR, SimPLe	Best of the comparable methods
Proprio Control	18	500K steps	D4PG, DMPO, MPO	New state of the art
Visual Control	20	1M steps	DrQ-v2, CURL	New state of the art
BSuite	23 (468 cfgs)	—	Bootstrapped DQN and others	New SOTA; largest gain in scale robustness
Minecraft Diamond	1	100M steps	IMPALA, Rainbow, PPO, VPT	First from scratch, without human data

Compute, honestly reported

Every agent in the paper trained on a single graphics card. That is what makes these results reproducible in a university lab rather than only at a large company.

Benchmark	Env instances	Replay ratio	GPU-days	Model size
Minecraft	64	32	8.9	200M
ProcGen	16	64	16.1	200M
Atari	16	32	7.7	200M
DMLab	16	32	2.9	200M
Atari 100K	1	128	0.1	200M
BSuite	1	1024	0.5	200M
Proprio Control	16	512	0.3	12M
Visual Control	16	512	0.1	12M

Model size — larger is strictly better

Six sizes, all with identical settings. Bigger models score higher and need less practice to get there — so if you have spare computing power, you can spend it to reduce the amount of real experience required.

Parameters	12M	25M	50M	100M	200M	400M
Hidden size d	256	384	512	768	1024	1536
Recurrent units 8d	1024	3072	4096	6144	8192	12288
Base CNN channels d/16	16	24	32	48	64	96
Codes per latent d/16	16	24	32	48	64	96

08 — The headline result

Diamonds in Minecraft, from scratch

This was a standing challenge in artificial intelligence, with international competitions built around it. DreamerV3 is the first program to do it without being shown how by a human, without a lesson plan, and without any hand-written hints — using its default settings straight out of the box.

Why it is hard

Every world is new. Each episode generates a fresh, infinite, randomly seeded 3D world. Memorisation is impossible.
Twelve items deep. Chop wood → craft a table → craft a wooden pickaxe → mine stone → craft a stone pickaxe → mine iron → smelt it → craft an iron pickaxe → find and mine a diamond. Skip a step and the chain breaks.
Almost no feedback. Up to thirty minutes of game time per attempt, with encouragement arriving only at those twelve milestones and nothing in between.
It only sees the screen. The agent looks at the picture and presses keys, exactly as a person would. It cannot query the game for answers.

For reference, an experienced human player needs about twenty minutes.

How the comparison went

DreamerV3

reaches diamond — every seed

IMPALA

stalls at the iron pickaxe

Rainbow

stalls at the iron pickaxe

PPO

stalls earlier

The compute comparison that matters

The previous best, VPT, reached diamonds by watching recordings of people playing, and it needed 720 graphics cards for nine days. DreamerV3 worked it out on its own using one card for nine days — and every one of its ten training runs succeeded.

Wood → Crafting table

Early, comparatively dense rewards. Most algorithms reach here.

Stone pickaxe → Furnace

Requires remembering a multi-step recipe across thousands of frames.

Iron ingot → Iron pickaxe

Where every competitor gets stuck. Smelting is a genuinely non-obvious detour.

Diamond

Requires deep mining with the right tool, far from any prior reward. Only DreamerV3 arrives — and reliably, not once by luck.

09 — What actually matters

Ablations: remove one piece at a time

What happens if you take each piece out? Averaged over fourteen tasks, every one of them matters — but each is essential on some tasks and makes no difference on others. That is exactly what you would expect from a toolkit built for generality.

Ablation A

Removing robustness techniques

Ranked by how far performance drops when each is removed:

1
KL balancing + free bits — The largest single effect. Without it the mental picture is either too crude or buried in irrelevant detail, depending on the game.
2
Return normalisation — Replace it with the usual approach and the agent stops exploring whenever rewards are rare.
3
symexp twohot — Replace it with a conventional loss and both reward and value learning slow down.
4
Observation symlog — Matters most when the numbers coming in from the environment are large.

Remove all of them together and performance collapses to near zero.

Ablation B

Which learning signal carries the agent?

The authors switched off the learning that comes from chasing reward, then switched off the learning that comes from simply redrawing what the agent sees, and compared what survived.

The surprising finding

Almost all of DreamerV3's ability comes from learning to redraw what it sees — not from chasing reward. Most earlier algorithms are the other way round.

Why this matters: redrawing what you see needs no rewards, no labels and no actions — the same kind of signal you get from ordinary video. The paper points at this as the door to training world models on internet video, which is the direction the field took next.

Scaling

Model size, 12M → 400M

Bigger is steadily better, with no settings changed. And larger models are not just stronger — they reach a given score after less practice.

Scaling

Replay ratio, 1 → 64

Thinking harder about each piece of experience reliably improves results. Together with model size, that gives two simple dials for trading computing power against real practice.

10 — Ablations

Pull one piece out at a time

Four axes: the five robustness techniques, the two learning signals, model size, and replay ratio. Select one.

Figure 6a · mean over 14 diverse tasks, normalised return

Removing the robustness techniques

Which of the five actually carries the method?

Dreamer, complete

≈100

no KL balance & free bits

≈58

no return normalisation

≈71

no symexp twohot (Huber instead)

≈78

no observation symlog

≈88

without all of themcollapse

≈6

Return, % of full Dreamer · higher is better

what it shows

All of them contribute, and the ordering is stable: the KL objective of the world model matters most, then return normalisation, then symexp twohot regression. Each is critical on some subset of tasks and irrelevant on others — which is exactly what a toolkit built for generality should look like. Remove all five and performance collapses to near zero.

how to read these numbers

The paper states the ordering — KL objective first, then return normalisation, then symexp twohot — but plots training curves rather than tabulating final scores.

The bars above are approximate readings of that figure, shown with a ≈. The ordering is what the paper asserts; treat the magnitudes as indicative.

The two that change how you read the paper. First, performance rests on the reconstruction loss rather than on reward prediction — the opposite of most reinforcement learning, and the reason the authors point at pretraining on unlabelled video as the next step. Second, model size and replay ratio both improve results monotonically without any hyperparameter being touched, so compute converts directly into data-efficiency.

11 — For the discussion

Five things to take away

01

Generality beats specialisation

A single configuration outperformed algorithms hand-tuned for each benchmark. The lesson travels beyond reinforcement learning: robustness engineering can be worth more than domain-specific cleverness.

02

Imagination is cheap

Split 'learn how the world works' from 'learn what to do in it', and you can spend scarce real experience on the first while practising the second as much as you like.

03

The hidden bug was scale

Most of the fragility in this field traced back to numbers whose size nobody knew in advance. The fixes are unglamorous and the effect is enormous.

04

Watching beats scoring

Learning to redraw what it sees, not chasing reward, is what shapes the agent's understanding. That points straight at world models trained on ordinary video.

05

A milestone, honestly measured

Diamonds in Minecraft from scratch, on one graphics card, with open code that reproduces every number in the paper. Reproducibility is part of the contribution.

?

Questions for discussion

Would a world model trained on internet video transfer to real robots? What happens somewhere genuinely unpredictable, like a market or a room full of people? Is imagining enough, or does the agent eventually need to plan?

12 — Reference

Glossary

Every term used on this page, in one or two plain sentences. No background assumed.

Learning by trial and error. An agent acts, receives a numeric reward, and adjusts to earn more over time — like training an animal with treats, except the animal is a neural network with millions of attempts.

A learned internal simulator: given a situation and a proposed action, it predicts what happens next. Once accurate, the agent rehearses plans internally instead of testing them for real.

A compressed summary. A 64×64 image is ~12,000 numbers; the latent keeps only what matters for acting — like describing a room as 'kitchen, door on the left, kettle on the counter' instead of listing every pixel.

Memory. A recurrent network carries information forward in time, so the agent remembers it already collected wood even when wood is off screen. DreamerV3 uses a block-diagonal GRU to hold many memory units without a quadratic cost increase.

Two cooperating networks. The actor decides what to do; the critic estimates how good a situation is. Whether the outcome beat the critic's expectation is the learning signal.

The return is total future reward. γ = 0.997 makes reward now worth slightly more than later, keeping the sum finite (≈333 steps). A λ-return blends short- and long-range estimates so the agent can value outcomes beyond its 15-step horizon.

A setting chosen by a human before training — learning rate, batch size, exploration strength. The paper's central claim is that these never change across 150+ tasks, which is what makes the method usable by non-specialists.

Feedback that arrives almost never. In Minecraft the agent may act for thousands of steps before anything signals progress. This is the hardest setting in RL, and why exploration strength must be handled so carefully.

A store of past experience the agent re-studies. The replay ratio is how many timesteps it trains on per timestep it plays — the dial trading GPU time for fewer real interactions.

A measure of how different two probability distributions are. Here: the gap between the predicted next latent and what the encoder actually saw. Shrinking it is how the model learns to predict.

A bonus paid for being unpredictable. Without it an agent locks onto the first mediocre strategy that works. It is the mathematical form of curiosity.

A floor under a loss term. Once the KL loss falls below 1 nat (≈1.44 bits) it is clipped and stops producing gradients, so optimisation moves on.

The gradient tells each weight which way to move. An enormous target gives an enormous gradient that can destroy the network in one step. DreamerV3 caps each tensor relative to its own size, so the cap never needs re-tuning.

An environment step is one interaction. Action repeat holds the same action for k frames (4 in Atari), so '200M frames' is 50M decisions — worth checking, since papers quote budgets in different units.

The hyperparameters that never change

Identical across proprioceptive and visual inputs, continuous and discrete actions, 2D and 3D domains.

Setting	Value	Controls
Batch shape	16 × 64	16 sequences of 64 timesteps per update
Learning rate	4×10⁻⁵	Identical for all three networks
Replay capacity	5×10⁶	How much past experience is retained
Imagination horizon	15	How far ahead the agent dreams
Discount horizon	333	How far into the future reward counts
Return lambda	0.95	Blend of short- and long-range value estimates
Dynamics / representation scale	1.0 / 0.1	KL balancing between the two directions
Free nats	1	Floor below which the KL losses switch off
Actor entropy	3×10⁻⁴	Exploration strength
Unimix	1%	Uniform floor on every categorical distribution
Return-norm range / limit	Per(95)−Per(5) / 1	Outlier-robust return scaling
Critic EMA decay	0.98	Stability of the value target
Optimiser	LaProp + AGC(0.3)	Scale-free updates and clipping

No annealing, no prioritised replay, no weight decay, no dropout.

Mastering Diverse Domainsthrough World Models