localgpt.rs

· 5 min read · LocalGPT Verse, LocalGPT MD

mistral.rs 0.8’s generate_structured hangs on GGUF, so I parse the JSON myself

Grammar-constrained decoding produced about zero tokens in three minutes while plain chat ran at 25 tokens per second. Here’s what LocalGPT Verse ships instead: plain instructed JSON, a 30-line parser, serde defaults, clamps, and a fallback.

LocalGPT Verse builds a 3D world for every song in your music folder. Signal analysis does most of the work: it measures the tempo, the sections, the energy curve, and a mood, and the mood picks the base world. On top of that there’s an optional LLM tier. A local 8B model reads what the analysis measured and writes one JSON object, a WorldRecipe, that restyles the world within that mood: biomes, landmarks, fog, particles, and what changes at the chorus. The model never hears the song and never sees the scene. Text goes in, JSON comes out.

Getting a JSON object out of a model sounds like a job for grammar-constrained decoding, and mistral.rs has it built in: derive JsonSchema on a type, call generate_structured, and the sampler can only produce tokens that fit the schema. On my setup it never produced anything.

The setup #

  • Model: Bonsai-8B, a Qwen3-architecture 8B chat model from prism-ml (Apache-2.0), as a Q4_K_M GGUF of about 5 GB
  • Runtime: mistral.rs 0.8, built with Verse’s llm-metal feature so the model runs on the GPU
  • Machine: an M2 Max with 32 GB, with the Bevy renderer running at the same time

What I saw #

The full recipe schema never completed. Neither did a two-field schema. With the model loaded, plain chat ran at about 25 tokens per second. generate_structured on the same loaded model produced roughly zero tokens in three minutes. No error, no output.

A probe test that documents the bug #

To make sure the hang was in the grammar path, and not in model loading or the GPU setup, I wrote a test that runs three phases against the real model:

  1. Plain chat with no grammar, to check the model and the device.
  2. generate_structured with a two-field schema, capped at 60 seconds. This is the bug.
  3. The real recipe path that ships instead.

Phase 2 treats failure as the expected result:

#[derive(serde::Deserialize, schemars::JsonSchema)]
struct Tiny {
    name: String,
    warmth: f32,
}
let tiny: Result<Tiny, String> = rt.block_on(async {
    match tokio::time::timeout(
        std::time::Duration::from_secs(60),
        recipe_model
            .model_mut()
            .generate_structured(TextMessages::new().add_message(
                TextMessageRole::User,
                "Describe a desert at dusk as JSON with a `name` (string) and \
             `warmth` (number 0..1).",
            )),
    )
    .await
    {
        Ok(result) => result.map_err(|e| e.to_string()),
        Err(_) => Err("timed out after 60s".into()),
    }
});

If a future mistral.rs fixes the hang, the test prints a mistral.rs bump fixed the grammar hang! instead of the timeout. It loads a 5 GB model, so it’s marked #[ignore] and only runs when asked. I rerun it on every mistral.rs upgrade:

cargo test --features llm-metal -- --ignored --nocapture llm_generation_probe

What ships instead #

1. Ask for JSON in plain words #

The system prompt lists every field with its type, allowed values, and range, and asks for a single object. An excerpt:

You design immersive 3D worlds for a music visualizer. Given a song's
analysis, reply with ONE JSON object (no prose, no code fences)
describing how to dress the world, with exactly these fields: …
biomes (array of {mood: 0-3, layout: "spiral"|"grid"|"rings",
density: 0..1, tint: [r,g,b]}), … motion_speed (0.25..2.5),
density (0.3..2.0). Modulate WITHIN the given mood …

The user message carries only what the analysis measured:

Mood: {mood_name} (the primary biome's mood index must be {idx}).
Tempo: {bpm} BPM. Mean energy: {energy_band} ({mean_energy:.2}). Sections: {sections}.
Reply with only the JSON object.

2. Take the first balanced object #

The parser doesn’t rely on the “no code fences” instruction. It finds the first {, tracks nesting depth, ignores braces inside strings (escaped quotes included), and returns the first complete object. If the reply stops partway through an object, it returns None rather than a partial one. This is the whole parser:

/// Pull the first balanced JSON object out of an LLM reply — tolerating code
/// fences and surrounding prose. Returns the object text (without fences) or
/// `None` when no complete `{...}` is present (e.g. the reply was truncated
/// mid-object; the caller keeps the rule recipe).
fn extract_json_object(text: &str) -> Option<String> {
    let start = text.find('{')?;
    let mut depth = 0usize;
    let mut in_string = false;
    let mut escaped = false;
    for (i, c) in text[start..].char_indices() {
        if in_string {
            if escaped {
                escaped = false;
            } else if c == '\\' {
                escaped = true;
            } else if c == '"' {
                in_string = false;
            }
            continue;
        }
        match c {
            '"' => in_string = true,
            '{' => depth += 1,
            '}' => {
                depth -= 1;
                if depth == 0 {
                    return Some(text[start..start + i + c.len_utf8()].to_string());
                }
            }
            _ => {}
        }
    }
    None // unbalanced (truncated) — no usable object
}

The tests cover code fences with prose around them, a } inside a string, a truncated reply, and a model that keeps talking after the object:

#[test]
fn extract_json_handles_fences_and_prose() {
    assert_eq!(
        extract_json_object("Sure! ```json\n{\"a\": 1}\n``` hope that helps"),
        Some("{\"a\": 1}".to_string())
    );
    assert_eq!(
        extract_json_object("{\"outer\": {\"inner\": \"}\"}, \"tail\": 2}"),
        Some("{\"outer\": {\"inner\": \"}\"}, \"tail\": 2}".to_string())
    );
    assert_eq!(extract_json_object("no object here"), None);
    // Truncated mid-object (a blown token budget) → None, not a partial.
    assert_eq!(extract_json_object("{\"a\": {\"b\": 1"), None);
    // First object wins when the model rambles after answering.
    assert_eq!(
        extract_json_object("{\"a\": 1} and then I said {\"b\": 2}"),
        Some("{\"a\": 1}".to_string())
    );
}

3. Keep safety out of the decoder #

A grammar would only have guaranteed the shape of the reply. What keeps the app safe works the same no matter how the JSON is produced:

  • Every recipe struct is #[serde(default)], so a reply with missing fields still deserializes and the gaps get defaults.

  • WorldRecipe::clamped() runs on every parsed recipe and bounds every number before anything reaches the renderer:

    pub fn clamped(mut self) -> Self {
    self.motion_speed = self.motion_speed.clamp(0.25, 2.5);
    self.density = self.density.clamp(0.3, 2.0);
    for b in &mut self.biomes {
        b.mood %= 4;
        b.density = b.density.clamp(0.0, 1.0);
        for c in &mut b.tint {
            *c = (*c).clamp(0.0, 1.0);
        }
    }
    // …the same for landmarks, atmosphere, choreography, and particles
    self
}

  • Each generation has a 180-second timeout, well above the 20 to 80 seconds a recipe actually takes, for when the machine slows down from thermal throttling or swapping.

  • A timeout, an error, or a reply with no usable object all return None, and the world keeps the recipe that the rules derived from the analysis.

So with no model, or a misbehaving one, you get the same world as a build without the LLM feature.

Results #

A full recipe takes 20 to 80 seconds per track. It’s cached in a per-track sidecar file keyed by a BLAKE3 hash of the track, so each song is only generated once. In the first live run, all four imported tracks got a recipe. Here’s one the model wrote, trimmed:

{
  "world_name": "Tide Gardens",
  "biomes": [
    { "mood": 2, "layout": "spiral", "density": 0.7, "tint": [1, 1, 1] }
  ],
  "landmarks": [
    { "kind": "spire", "at": "center", "scale": 1.5, "emissive": 0.7 }
  ],
  "atmosphere": { "fog_density": 0.4 },
  "section_choreography": [
    { "at_role": "chorus", "energy_shift": 0.3, "motion": "active" }
  ],
  "particles": { "kind": "spark", "rate": 0.6, "drift": 1.2 },
  "motion_speed": 1.5,
  "density": 0.8
}

The weak spot is world_name: the model repeated the mood’s name, Tide Gardens, instead of inventing one. That was a prompt problem, not a parsing problem. The prompt now asks for a name of the model’s own and gives an example: for Tide Gardens, “Kelp Cathedral at Blue Hour”.

LocalGPT MD reuses it #

LocalGPT MD turns a Markdown file into a walkable world, and it runs the same model to restyle each section from its prose. It ported the whole approach from Verse: plain instructed JSON, the same balanced-object parser (in RegionRecipe::from_llm_text), serde defaults, clamping, and a fallback to the rule-based draft when anything goes wrong.

Two more findings from the same run #

  • The 1-bit Q1_0 quant doesn’t parse in mistral.rs 0.8, so Verse’s fetch script downloads the Q4_K_M instead.
  • The 5 GB Q4_K_M didn’t fit mistral.rs’s CPU device map next to the renderer, which left about 7.6 GB free. Running it on the GPU fixed that: with unified memory the model fits comfortably, and generation got about an order of magnitude faster.

If you know why the grammar path stalls on GGUF, or a newer mistral.rs has fixed it, I’d like to hear about it: open an issue on Verse.