llama-cpp-server/docs/superpowers/specs/2026-06-10-gguf-metadata-preset-design.md

GGUF Metadata Preset Generator Design

Date: 2026-06-10

Goal

Refine generate_models_preset.py so generated models-preset.ini entries use richer and more practical llama-server parameters, with all derivation based on GGUF metadata rather than filename heuristics.

User Constraints

• Use GGUF metadata as the only model-behavior input.
• Do not infer MoE or other model classes from filenames.
• Keep mmproj detection from sibling files because it is an explicit companion artifact, not a naming heuristic.
• Add nearby comments that show the minimum value, maximum value, and chosen value for each key parameter.
• Do not create backup files by default.

Scope

In scope:

• Update generate_models_preset.py
• Expand generated preset fields beyond the current minimal set
• Derive MoE-related values from GGUF metadata
• Emit per-parameter range comments for tunable values
• Update or add automated tests for the new generation rules

Out of scope:

• Changing start-llama-router.bat
• Adding filename-based fallbacks
• Reworking router-wide launch flags

Current Problem

The current generator reads some GGUF metadata, but practical preset output is still too thin:

• many fields are hard-coded shared defaults
• MoE behavior is only partially informed by metadata
• some classification still falls back to filename keywords
• generated output does not document legal value ranges near chosen values
• backup behavior is on by default

This makes the output weaker than the known-good single-model startup profile in start-qwen3.6-35b-a3b.bat.

Recommended Design

Metadata-Only Classification

Classify models from metadata and explicit companion files only:

• asr_multimodal
  • sibling mmproj exists
• moe
  • architecture-specific expert_count is present and greater than zero
• dense
  • everything else

There must be no filename keyword fallback for MoE detection or any other preset behavior.

Metadata Inputs

The generator should continue reading GGUF key-value metadata directly from the header and use these keys where available:

• general.architecture
• <architecture>.context_length
• <architecture>.block_count
• <architecture>.expert_count
• <architecture>.expert_used_count
• general.size_label
• general.name

These values are enough to support classification plus bounded parameter generation.

Parameter Generation Rules

Generate a richer field set per model section.

Dense

Dense models should receive:

• ctx-size
• n-gpu-layers
• threads
• batch-size
• ubatch-size
• parallel
• cache-type-k
• cache-type-v
• kv-unified
• kv-offload
• mlock
• mmap
• seed
• flash-attn

Dense values can stay moderately permissive, but must still be derived from metadata-sensitive rules instead of one fixed default map.

ASR / Multimodal

ASR or multimodal models should stay conservative:

• lower parallel
• lower batch-size
• lower ctx-size than large chat defaults
• preserve mmproj

MoE

MoE models should receive:

• all core fields above
• n-cpu-moe

n-cpu-moe must be emitted only for metadata-confirmed MoE models.

Bounded Value Strategy

Each tunable numeric field should have a clear bounded range, with the selected value derived within that range.

Required nearby comments for key fields:

• ctx-size
• n-gpu-layers
• n-cpu-moe when present
• threads
• batch-size
• ubatch-size
• parallel

Comment format should be simple and consistent, for example:

; ctx-size range = 512..131072; chosen = 43008
ctx-size = 43008

For fields where the lower or upper bound is determined by metadata, the comment must use that metadata-derived bound.

Specific Derivation Direction

The generator does not need exact benchmarking logic. It needs stable, explainable rules.

Recommended direction:

• ctx-size
  • maximum from metadata context_length
  • choose a practical default no higher than the metadata maximum
• n-gpu-layers
  • maximum from metadata block_count
  • choose full offload by default when the maximum is known
• n-cpu-moe
  • maximum from metadata block_count
  • chosen value derived from MoE metadata and model size pressure
• threads
  • bounded by local CPU count
• parallel
  • conservative for multimodal and MoE
• batch-size and ubatch-size
  • reduced for conservative profiles, especially MoE and multimodal
• cache-type-k and cache-type-v
  • may prefer q8_0 for long-context or MoE models, otherwise a safer baseline
• mmap
  • default toward the more practical behavior already used in start-qwen3.6-35b-a3b.bat

Backup Policy

Backup generation should no longer be the default behavior.

Recommended implementation:

• set CLI default to no backup
• keep optional explicit --backup support only if trivial to preserve

Architecture

Keep the current single-script architecture, but tighten responsibilities:

1. GGUF metadata reader
2. metadata-only classifier
3. bounded preset value generators by class
4. comment renderer for min/max/chosen annotations
5. INI renderer

Testing

Tests should cover:

• MoE classification depends on metadata expert_count, not filename text
• dense models never receive n-cpu-moe
• multimodal models are detected from sibling mmproj
• range comments are present near the key generated fields
• backup default is disabled
• output remains valid INI text

Risks

• Some GGUF files may omit certain metadata keys; in those cases the script must use documented conservative fallback ranges without reintroducing filename heuristics.
• cache-type-k and cache-type-v are policy choices rather than direct metadata facts, so the rules must stay simple and predictable.

Implementation Notes

The workspace currently lacks normal Git repository behavior for commit operations, so this spec is written locally without a commit step.