Can you train a model on Simon Willison's deeply unscientific pelican benchmark?

So I saw this tweet by Ben:

there's a pelican svg rl environment

There wasn't. So I built one. It runs as an HF Space, it scores a drawing in three layers, and by the end of this post you will have two commands: one that evaluates any model against it, and one that trains a model against it on a GPU. All of it with OpenEnv, TRL and HF Jobs.

A bit of background

In case you are unaware: there is a famous prompt, "generate an SVG of a pelican riding a bicycle", that Simon Willison has put to nearly every new model since early 2025.

He calls it his "deeply unscientific benchmark", and his advice is "don't go using pelicans to compare models". I agree with him. But it still makes a fun use case for an environment, no?

The test caught on because a person can judge it in a second. You look, and you know. That is also where it stops: there is no number in a blog post, so nothing can optimise against it. An environment gives you the number.

And in case you want the actual evidence on whether frontier labs are training on this thing, Dylan Castillo ran the rigorous version and found little sign of it.

The environment

Pelican SVG Env is the OpenEnv environment I built for this. The code is at envs/pelican_svg_env, and it is deployed as an HF Space at sergiopaniego/pelican-svg-env so you can talk to it without running anything locally. reset() hands you a task, step() takes a model's raw reply and returns a reward. That is the whole interface, and it is the same interface whether an eval script or a trainer is on the other end.

The catalogue holds 6 animals and 5 vehicles, so 30 tasks in total. The default is Simon's pelican on a bicycle, but there are also an axolotl and an octopus, and a skateboard and a scooter.

How do you score it?

How do you score a drawing of a pelican riding a bicycle? I spent longer on that than on the environment itself. It turns out to be three different problems, and they cost very different amounts. So the free ones run first and do the discarding, and a drawing that gets rejected never reaches a call that costs money.

Layer 1, the gate. Free, pass or fail. The source is inspected before anything looks at the picture, because the shortcuts are only visible there. A model can satisfy "draw a pelican" by embedding a photograph as a data:image URI, or by writing the words as <text> and letting the judge read them back. Both render as pictures and both are rejected on sight.

The gate only asks whether the drawing is an honest attempt, not whether it is a good one, which is why the scribble gets through. Note that both rejected cases render as a picture, and one of them would look perfect to a scorer that only ever saw pixels. That is the reason the source is read first.

Layer 2, the structure score. Free, deterministic, weight 0.35. Seven geometric checks on the shapes actually present: the right number of round things for this vehicle, similar in size, level with each other, far enough apart, something spanning between them, a body above the axle line, and that body in proportion. It knows nothing about pelicans, only what a wheeled vehicle carrying a rider looks like geometrically. It reads the document properly rather than pattern-matching the text, so flattened beziers, nested transforms and CSS units all resolve the way the spec says.

Layer 3, the semantic score, from a vision judge. Two multimodal calls, weight 0.65. The drawing is rendered to PNG and shown to a vision model twice. First cold, with no mention of the task at all, asked only what it is. The blind caption is the part I would keep if I had to throw the rest away, and I will come back to it.

Then a checklist of nine yes-or-no questions generated from the task, and semantic is half that checklist and half what the blind caption recognised. The questions ask about features rather than names, so one of the nine is whether the animal has a throat pouch under its beak, which is the thing that makes it a pelican and not a stork. Another asks whether the animal is really riding the vehicle. If the answer is no, the score drops to a quarter, because a bicycle with nothing on it does not answer the prompt.

Every number in this post used Qwen/Qwen2.5-VL-72B-Instruct as the judge, through HF Inference Providers. Which judge you pick moves the score about as much as which model you are testing, so it is a setting rather than a hardcoded choice and the environment records it on every result.

That makes the reward one line:

reward = 0.35 * structure + 0.65 * semantic     (0 if the gate rejects it)

So structure is the free half that any script could compute, and semantic is the half that needs a model to look at the picture and carries most of the weight. Both columns show up in the results tables below.

Evaluating the frontier models

With a way to score a drawing, the obvious next thing was to do what everybody does with this benchmark and point it at the frontier models. The difference is that scoring is now a function rather than a judgement call, so it can be twenty samples a model instead of one, and what comes back is a number rather than a screenshot.

examples/pelican_svg_eval.py generates completions through HF Inference Providers and sends each one to the environment to be scored. It holds no scoring logic of its own, so the trainer in the next section runs against the same scorer, just configured differently:

python examples/pelican_svg_eval.py \
    --env-url https://sergiopaniego-pelican-svg-env.hf.space \
    --models Qwen/Qwen3.6-27B zai-org/GLM-5.2 \
    --samples 20 \
    --save-svgs ./svgs

One HF token, any model the router serves, no other accounts to open. It also supports closed models through their own vendor APIs if you set those keys, which is how the OpenAI and Anthropic numbers below were produced. Start with the open models: they need nothing but the token you already have.

Use --save-svgs. Every scorer bug I found, I found by rendering something that scored badly and looking at it, never by reading code and never from a table.

Here is the median drawing from each of the seven models. Median by reward, so nothing here is cherry-picked:

Look at the captions, because that is the finding. Seven competent drawings, and the judge shown them cold calls two of them a pelican. The other five are three ducks, a goose and a generic bird.

Twenty samples per model, 139 scored in total. The missing one is explained below:

GPT-5.6 Sol leads this run. The next five sit within 0.04 of each other, which at twenty samples is no separation at all, and open and closed weights are interleaved through the whole middle of the table. On this task there is no open-versus-closed gap to report. This is Simon's "mostly severed" point with numbers on it: the middle of the frontier draws pelicans equally well.

Kimi K3 is the odd row out, and the interesting one. It came second, and it is also the only model that stumbled, twice: one SVG was cut off mid-document and took a zero from the gate, a zero that is inside its mean, and one completion never arrived at all because the model was still thinking at the token limit. That second one is why the table says 19.

Those are the same property seen from both ends. Kimi K3 emitted 4.9 times more reasoning than drawing, 567,000 characters of thinking against 117,000 of SVG, where the next open model is at 0.8 times. Simon measured the same thing on a single sample and it is what his cost complaint is about: 16,658 output tokens of which 13,241 were reasoning, "for a total cost of 25 cents!". His one pelican and our twenty agree. Reasoning is what gets billed, and it turns out to be what runs out too.

Two things to keep in view. The open models were sampled at temperature 0.7 through the router. Both closed vendors refuse a temperature on these models, so their spread is not measured under the same conditions and should not be compared. And structure is saturated: 136 of the 139 samples scored exactly 1.000, and five of the seven models scored 1.000 on every single sample. At frontier level that layer emits one number and never varies. All of the remaining signal is in the judge.

Back to the blind caption

I said the blind caption was the part I would keep, and this is why. Ask a vision model "is this a pelican?" and you get a yes. Ask it cold what the drawing is, and across the 139 samples only 40% come back "pelican". Split by model, each one misses in its own way:

Claude Fable 5 draws the same bird twenty times over and the judge calls it a pelican once. Qwen3.6 draws storks. None of that shows up if you ask a leading question, and it costs nothing extra: it is the same call, with the task removed from the prompt.

The table records what one judge said, not what the drawings are: a pelican and a stork are genuinely hard to tell apart in a crude cartoon. That is why the reward leans on the feature checklist rather than the caption. The judge is a setting you can change without touching code. The checklist and the weights are code, one small file, if you want something stricter.

Every drawing, score, blind caption and render is published as a dataset: sergiopaniego/pelican-svg-drawings. The dataset is the run itself, the 139 canonical-task drawings above plus one drawing for each of the 30 catalogue tasks as a second config, and every row records its provider and its judge. Everything here ran on 29 July 2026 with a 24,000-token output budget.

Training against it

With the eval working, the last natural step was to check whether the environment holds up as a training target. Not to produce a better pelican model, which is not the point and is not going to happen at this size, just to find out whether the reward can be optimised at all. So I set up two small training runs: one with the judge switched off, and one with it on.

examples/pelican_svg_grpo.py runs GRPO through TRL, takes its reward from the same environment, and is meant to be launched on HF Jobs so you do not need a GPU of your own:

hf jobs uv run \
    --flavor a100-large \
    --timeout 4h \
    --detach \
    --secrets HF_TOKEN \
    examples/pelican_svg_grpo.py -- \
    --model Qwen/Qwen3-1.7B \
    --steps 80 \
    --probe-samples 24 \
    --push-to-hub \
    --out <your-user>/pelican-svg-grpo-Qwen3-1.7B \
    --trackio-space <your-user>/pelican-svg-grpo-curves

The environment boots inside the job, the trainer talks to it over the OpenEnv client, metrics stream to a trackio Space while it runs, and the checkpoint plus its sample drawings land on the Hub at the end. Nothing else to wire up. Pass --timeout generously: Jobs stop at a default timeout and 80 steps on a 1.7B took about two and a half hours.

In fact, nothing in this post touched local hardware. The environment lives on a Space, generation went through the Inference Providers router, and training ran on an A100 inside an HF Job, billed by the second.

Both rewards go up. Judge off, 0.190 to 0.496 between the first and last fifth. Judge on, 0.074 to 0.198. Both curves are live here, and both look like success.

The probe is what tells the truth. The script re-scores each checkpoint on 24 fresh samples with the judge on, and compares them against the untrained model, which every run measures the same way before training starts:

The judge-off run learned something real. Its structure lands well clear of all five untrained probes, on samples it never trained on, and its invalid submissions fell from 3 in 24 to 1. The judge-on run learned nothing that transfers: its structure finished inside the untrained range. And neither run produced anything the judge recognises.

The drawings say why. After training a motif repeats: a large blob, two small dark circles below it at the same height, something joining them. Not a pelican, not a bicycle. The model did not learn the task, it learned what the checks measure, and that exact shape scores 1.000 on all seven of them. It is not reward hacking, because the judge was not part of that reward: the policy did exactly what it was paid for. The plainer finding is that structure alone is an insufficient proxy for the task.

Paying for the judge does not fix it, because the problem is not the weight. GRPO learns from reward differences inside a group of completions, and the untrained 1.7B scores non-zero on semantic twice in 24 samples, so most groups give it nothing to learn from. The same judge scores non-zero on nearly every frontier drawing, which is why a larger base model is the obvious next experiment.

Before spending GPU hours, check how often your judged term is non-zero on the base model you are about to train. If it is almost always zero, it will teach nothing.

So the loop works, the structural gain is real and transfers, and teaching the full task needs more model than 1.7B. Both checkpoints and their probe drawings are public: pelican-svg-grpo-Qwen3-1.7B and -judged.

Go and play with it

Everything is public, because a scorer with a bug in it looks exactly like a scorer without one until you check.

Duplicate the Space, point the eval script at your copy, pick one of the other 29 animal-and-vehicle pairs or add your own, change the judge, reweight the layers. The whole thing is 30 tasks of scaffolding around one idea: if you can score it, you can train against it.

Conclusion

Ben posted that there was a pelican SVG RL environment. There was not... but now there is.

Whether the frontier labs run anything like it is not something you can find out from outside, because the environments they train on are the one thing they do not publish. This one is published, pelican included 🦆 (that is the closest thing to a pelican the emoji set has, which after this post feels about right).

And if you build one of your own, swap the pelican for a task you actually care about. The scoring layers will not mind.