GRPO in 100 steps: how to fine-tune a 350M model for structured output without a GPU farm

Anyone putting an LLM into production knows that a good part of the work isn't reasoning: it's getting the model to return valid JSON, with the right fields, in the requested form (inside or outside a code block), so the next service can parse it. And that's exactly where small models tend to fail. A guide published on the Hugging Face blog by Leonie Monigatti, Ben Burtenshaw, and Sergio Paniego shows a cheap way to attack this specific problem: fine-tuning Liquid AI's LFM2.5-350M with GRPO (Group Relative Policy Optimization) via the TRL library, using about 500 samples and 100 training steps, enough to fit on a free Colab or Kaggle GPU.

The central result: on the IFStruct benchmark, the model goes from 22.6% to 29.7% overall pass rate. That looks modest until you look at where the gain concentrates.

The problem IFStruct measures

Most benchmarks dilute "structured output" into broader reasoning or extraction scores. IFStruct (open-source at Liquid4All/ifstruct, with a dataset at LiquidAI/ifstruct-v1.0) does the opposite: it measures only schema conformance. The question is binary and practical: did the model return something parseable, in the requested form, that validates against the expected JSON Schema? If not, it can't be hooked into the downstream system.

The guide's baseline is honest about methodology. The IFStruct blog reports 21.1% for LFM2.5-350M; when serving the model locally via llama.cpp in BF16 GGUF, the authors measured 22.6%, close enough to use as the baseline for the same serving stack. Running the full benchmark means 2000 samples:

uv run ifstruct-eval \
  --model LiquidAI/LFM2.5-350M \
  --base-url http://localhost:8080/v1 \
  --api-key dummy \
  --dataset data/test.jsonl \
  --results-file results/lfm2.5-350m-llamacpp-base.json \
  --n-threads 4 --max-tokens 2048 -v

The baseline diagnosis is revealing: the most common errors are required field missing (7,228 times), wrong item counts, and type mismatch. And there's a clear gap by format: JSON passed only 18%, while YAML already reached 27.2%. "Bare" lists were a disaster: 16.6%.

How GRPO tackles this

GRPO is reinforcement learning without a trained reward model: you define reward functions that score each generation, the model samples several completions per prompt (here, 8 per group), and it's pushed toward the ones that score better relative to the group average. Unlike SFT, you don't teach the right text, you teach the success criterion, which pairs well with a verifiable target like "does this JSON validate against the schema?"

The training uses nvidia/Nemotron-RL-instruction_following-structured_outputs, which already pairs prompt, target JSON Schema, and expected field count. Since the Nemotron distribution differs from IFStruct's, the authors augmented the prompts to close two gaps: 40% receive an instruction to "return inside a fenced code block" (so the model learns to obey the format instruction instead of always spitting out raw JSON), and a disjoint 20% become top-level array tasks, training exactly bare-list and item-count, the two biggest weaknesses of the baseline.

The adaptation is via LoRA, with a detail anyone reproducing it needs to know: LFM2.5 uses a hybrid attention/convolution architecture, so the target_modules aren't the usual ones:

lora_config = LoraConfig(
    r=16, lora_alpha=32, bias="none", task_type="CAUSAL_LM",
    target_modules=[
        "q_proj", "k_proj", "v_proj", "out_proj", "in_proj",
        "w1", "w2", "w3",
    ],
)

This trains ~6M parameters, about 1.66% of the model.

The three rewards (and the weights matter)

The heart of the method is three reward functions, each on a [0, 1] scale:

• json_format_reward: is the output parseable and in the requested form? Full credit (1.0) for the correct form (fenced vs. raw), 0.2 for wrong form but parseable, 0.0 for unparseable.
• field_count_reward: does the object have the expected number of top-level fields? An exact match gives 1.0, and the score decays linearly with the error.
• schema_validation_reward: validates against the row's JSON Schema, counting each violation and conditioning partial credit on coverage of required keys.

The combination is a weighted sum with reward_weights=[1.0, 0.5, 2.0], meaning schema validation weighs twice as much as format and four times as much as field count. This choice of weights is where the intent of the training lives, and it's worth using as a starting point for anyone adapting this to another task.

The training configuration fits in 16 GB:

training_args = GRPOConfig(
    learning_rate=5e-5, max_steps=100, warmup_steps=10,
    num_generations=8,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=8,
    max_completion_length=1024,  # room for nested JSON
    temperature=1.1,             # hotter sampling varies the groups
    beta=0.01,                   # KL penalty against the reference model
    reward_weights=[1.0, 0.5, 2.0],
)

Where the gain shows up

After training, the LoRA adapter is merged back (merge_and_unload()), converted to GGUF BF16 with llama.cpp's convert_hf_to_gguf.py, and IFStruct is run again on the same stack. The comparison:

| IFStruct Group | Base | GRPO | Δ | |---|---|---|---| | Overall | 22.6% | 29.7% | +7.1 | | JSON | 18.0% | 31.9% | +13.9 | | YAML | 27.2% | 27.5% | +0.3 | | Wrapper key | 28.5% | 29.7% | +1.2 | | Bare list | 16.6% | 29.7% | +13.1 |

The interesting point is that the gain lands exactly where the training aimed. JSON goes up almost 14 points, bare list goes up 13, and YAML stays practically flat, because the augmentation focused on fenced JSON and top-level lists, not YAML. That's a sign the RL did what it should, not that it "magically improved everything." For reference, the guide cites Qwen3.5-2B scoring 33.15%: a model about six times larger. The light fine-tuning doesn't surpass that, but it comes close.

What this changes for builders in Brazil

The practical takeaway, and here it's my own inference about what the material suggests, is that reliable structured output doesn't necessarily need a large model that's expensive per token. If your task is well-defined (extracting an invoice, building an itinerary, generating a payload with a fixed schema), a 350M model fine-tuned for that specific form can become a viable option to run on-prem or even at the edge, without depending on a paid API for every call. llama.cpp serving a 350M GGUF fits on a MacBook, and the training fits on a free GPU, which knocks down the cost barrier that usually pushes everyone toward external APIs.

There are honest caveats. First, 29.7% conformance is still low to throw into a critical pipeline without validation and retry; the required field missing error remains dominant even after training. Second, the gain is specific to the trained distribution: the guide itself makes clear that the pipeline doesn't recreate the score of IFStruct's original RL model, it's a demonstration that task-specific fine-tuning moves the needle. Third, for tasks with a highly variable schema or that require real reasoning, the small model probably isn't worth it, and it's better to spend on a larger model or use grammar-constrained decoding to force valid JSON instead of hoping the model learns it.

Where the recipe shines is in the case of high volume of a known format: there, a short GRPO run can turn a tiny model into something predictable enough for production, at an inference cost that an API can hardly match. The full notebook and repositories are linked in the Hugging Face post for anyone who wants to reproduce it.