mistral.rs 0.8’s generate_structured hangs on GGUF, so I parse the JSON myself

2026-09-22T00:00:00+00:00

The setup #</a> </h2> Model: Bonsai-8B, a Qwen3-architecture 8B chat model from prism-ml (Apache-2.0), as a Q4_K_M GGUF of about 5 GB</li> Runtime: mistral.rs 0.8, built with Verse’s llm-metal</code> feature so the model runs on the GPU</li> Machine: an M2 Max with 32 GB, with the Bevy renderer running at the same time</li> </ul> What I saw #</a> </h2> The full recipe schema never completed. Neither did a two-field schema. With the model loaded, plain chat ran at about 25 tokens per second. generate_structured</code> on the same loaded model produced roughly zero tokens in three minutes. No error, no output. A probe test that documents the bug #</a> </h2> To make sure the hang was in the grammar path, and not in model loading or the GPU setup, I wrote a test that runs three phases against the real model: Plain chat with no grammar, to check the model and the device.</li> generate_structured</code> with a two-field schema, capped at 60 seconds. This is the bug.</li> The real recipe path that ships instead.</li> </ol> Phase 2 treats failure as the expected result: #[derive(serde::Deserialize, schemars::JsonSchema)] struct Tiny { name: String, warmth: f32, } let tiny: Result<Tiny, String> = rt.block_on(async { match tokio::time::timeout( std::time::Duration::from_secs(60), recipe_model .model_mut() .generate_structured(TextMessages::new().add_message( TextMessageRole::User, "Describe a desert at dusk as JSON with a `name` (string) and \ `warmth` (number 0..1).", )), ) .await { Ok(result) => result.map_err(|e| e.to_string()), Err(_) => Err("timed out after 60s".into()), } });</code></pre> If a future mistral.rs fixes the hang, the test prints a mistral.rs bump fixed the grammar hang!</code> instead of the timeout. It loads a 5 GB model, so it’s marked #[ignore]</code> and only runs when asked. I rerun it on every mistral.rs upgrade: cargo test --features llm-metal -- --ignored --nocapture llm_generation_probe</code></pre>What ships instead #</a> </h2> 1. Ask for JSON in plain words #</a> </h3> The system prompt lists every field with its type, allowed values, and range, and asks for a single object. An excerpt: You design immersive 3D worlds for a music visualizer. Given a song's analysis, reply with ONE JSON object (no prose, no code fences) describing how to dress the world, with exactly these fields: … biomes (array of {mood: 0-3, layout: "spiral"|"grid"|"rings", density: 0..1, tint: [r,g,b]}), … motion_speed (0.25..2.5), density (0.3..2.0). Modulate WITHIN the given mood …</code></pre> The user message carries only what the analysis measured: Mood: {mood_name} (the primary biome's mood index must be {idx}). Tempo: {bpm} BPM. Mean energy: {energy_band} ({mean_energy:.2}). Sections: {sections}. Reply with only the JSON object.</code></pre>2. Take the first balanced object #</a> </h3> The parser doesn’t rely on the “no code fences” instruction. It finds the first {</code>, tracks nesting depth, ignores braces inside strings (escaped quotes included), and returns the first complete object. If the reply stops partway through an object, it returns None</code> rather than a partial one. This is the whole parser: /// Pull the first balanced JSON object out of an LLM reply — tolerating code /// fences and surrounding prose. Returns the object text (without fences) or /// `None` when no complete `{...}` is present (e.g. the reply was truncated /// mid-object; the caller keeps the rule recipe). fn extract_json_object(text: &str) -> Option<String> { let start = text.find('{')?; let mut depth = 0usize; let mut in_string = false; let mut escaped = false; for (i, c) in text[start..].char_indices() { if in_string { if escaped { escaped = false; } else if c == '\\' { escaped = true; } else if c == '"' { in_string = false; } continue; } match c { '"' => in_string = true, '{' => depth += 1, '}' => { depth -= 1; if depth == 0 { return Some(text[start..start + i + c.len_utf8()].to_string()); } } _ => {} } } None // unbalanced (truncated) — no usable object }</code></pre> The tests cover code fences with prose around them, a }</code> inside a string, a truncated reply, and a model that keeps talking after the object: #[test] fn extract_json_handles_fences_and_prose() { assert_eq!( extract_json_object("Sure! ```json\n{\"a\": 1}\n``` hope that helps"), Some("{\"a\": 1}".to_string()) ); assert_eq!( extract_json_object("{\"outer\": {\"inner\": \"}\"}, \"tail\": 2}"), Some("{\"outer\": {\"inner\": \"}\"}, \"tail\": 2}".to_string()) ); assert_eq!(extract_json_object("no object here"), None); // Truncated mid-object (a blown token budget) → None, not a partial. assert_eq!(extract_json_object("{\"a\": {\"b\": 1"), None); // First object wins when the model rambles after answering. assert_eq!( extract_json_object("{\"a\": 1} and then I said {\"b\": 2}"), Some("{\"a\": 1}".to_string()) ); }</code></pre>3. Keep safety out of the decoder #</a> </h3> A grammar would only have guaranteed the shape of the reply. What keeps the app safe works the same no matter how the JSON is produced: Every recipe struct is #[serde(default)]</code>, so a reply with missing fields still deserializes and the gaps get defaults. </li> WorldRecipe::clamped()</code> runs on every parsed recipe and bounds every number before anything reaches the renderer: pub fn clamped(mut self) -> Self { self.motion_speed = self.motion_speed.clamp(0.25, 2.5); self.density = self.density.clamp(0.3, 2.0); for b in &mut self.biomes { b.mood %= 4; b.density = b.density.clamp(0.0, 1.0); for c in &mut b.tint { c = (c).clamp(0.0, 1.0); } } // …the same for landmarks, atmosphere, choreography, and particles self }</code></pre></li> Each generation has a 180-second timeout, well above the 20 to 80 seconds a recipe actually takes, for when the machine slows down from thermal throttling or swapping. </li> A timeout, an error, or a reply with no usable object all return None</code>, and the world keeps the recipe that the rules derived from the analysis. </li> </ul> So with no model, or a misbehaving one, you get the same world as a build without the LLM feature. Results #</a> </h2> A full recipe takes 20 to 80 seconds per track. It’s cached in a per-track sidecar file keyed by a BLAKE3 hash of the track, so each song is only generated once. In the first live run, all four imported tracks got a recipe. Here’s one the model wrote, trimmed: { "world_name": "Tide Gardens", "biomes": [ { "mood": 2, "layout": "spiral", "density": 0.7, "tint": [1, 1, 1] } ], "landmarks": [ { "kind": "spire", "at": "center", "scale": 1.5, "emissive": 0.7 } ], "atmosphere": { "fog_density": 0.4 }, "section_choreography": [ { "at_role": "chorus", "energy_shift": 0.3, "motion": "active" } ], "particles": { "kind": "spark", "rate": 0.6, "drift": 1.2 }, "motion_speed": 1.5, "density": 0.8 }</code></pre> The weak spot is world_name</code>: the model repeated the mood’s name, Tide Gardens, instead of inventing one. That was a prompt problem, not a parsing problem. The prompt now asks for a name of the model’s own and gives an example: for Tide Gardens, “Kelp Cathedral at Blue Hour”. LocalGPT MD reuses it #</a> </h2> LocalGPT MD</a> turns a Markdown file into a walkable world, and it runs the same model to restyle each section from its prose. It ported the whole approach from Verse: plain instructed JSON, the same balanced-object parser (in RegionRecipe::from_llm_text</code>), serde defaults, clamping, and a fallback to the rule-based draft when anything goes wrong. Two more findings from the same run #</a> </h2> The 1-bit Q1_0 quant doesn’t parse in mistral.rs 0.8, so Verse’s fetch script downloads the Q4_K_M instead.</li> The 5 GB Q4_K_M didn’t fit mistral.rs’s CPU device map next to the renderer, which left about 7.6 GB free. Running it on the GPU fixed that: with unified memory the model fits comfortably, and generation got about an order of magnitude faster.</li> </ul> If you know why the grammar path stalls on GGUF, or a newer mistral.rs has fixed it, I’d like to hear about it: open an issue on Verse</a>.

localgpt.rs - md

mistral.rs 0.8’s generate_structured hangs on GGUF, so I parse the JSON myself