# GGUF Metadata Preset Generator Design Date: 2026-06-10 ## Goal Refine `generate_models_preset.py` so generated `models-preset.ini` entries use richer and more practical `llama-server` parameters, with all derivation based on GGUF metadata rather than filename heuristics. ## User Constraints - Use GGUF metadata as the only model-behavior input. - Do not infer MoE or other model classes from filenames. - Keep `mmproj` detection from sibling files because it is an explicit companion artifact, not a naming heuristic. - Add nearby comments that show the minimum value, maximum value, and chosen value for each key parameter. - Do not create backup files by default. ## Scope In scope: - Update `generate_models_preset.py` - Expand generated preset fields beyond the current minimal set - Derive MoE-related values from GGUF metadata - Emit per-parameter range comments for tunable values - Update or add automated tests for the new generation rules Out of scope: - Changing `start-llama-router.bat` - Adding filename-based fallbacks - Reworking router-wide launch flags ## Current Problem The current generator reads some GGUF metadata, but practical preset output is still too thin: - many fields are hard-coded shared defaults - MoE behavior is only partially informed by metadata - some classification still falls back to filename keywords - generated output does not document legal value ranges near chosen values - backup behavior is on by default This makes the output weaker than the known-good single-model startup profile in `start-qwen3.6-35b-a3b.bat`. ## Recommended Design ### Metadata-Only Classification Classify models from metadata and explicit companion files only: - `asr_multimodal` - sibling `mmproj` exists - `moe` - architecture-specific `expert_count` is present and greater than zero - `dense` - everything else There must be no filename keyword fallback for MoE detection or any other preset behavior. ### Metadata Inputs The generator should continue reading GGUF key-value metadata directly from the header and use these keys where available: - `general.architecture` - `.context_length` - `.block_count` - `.expert_count` - `.expert_used_count` - `general.size_label` - `general.name` These values are enough to support classification plus bounded parameter generation. ### Parameter Generation Rules Generate a richer field set per model section. #### Dense Dense models should receive: - `ctx-size` - `n-gpu-layers` - `threads` - `batch-size` - `ubatch-size` - `parallel` - `cache-type-k` - `cache-type-v` - `kv-unified` - `kv-offload` - `mlock` - `mmap` - `seed` - `flash-attn` Dense values can stay moderately permissive, but must still be derived from metadata-sensitive rules instead of one fixed default map. #### ASR / Multimodal ASR or multimodal models should stay conservative: - lower `parallel` - lower `batch-size` - lower `ctx-size` than large chat defaults - preserve `mmproj` #### MoE MoE models should receive: - all core fields above - `n-cpu-moe` `n-cpu-moe` must be emitted only for metadata-confirmed MoE models. ### Bounded Value Strategy Each tunable numeric field should have a clear bounded range, with the selected value derived within that range. Required nearby comments for key fields: - `ctx-size` - `n-gpu-layers` - `n-cpu-moe` when present - `threads` - `batch-size` - `ubatch-size` - `parallel` Comment format should be simple and consistent, for example: ```ini ; ctx-size range = 512..131072; chosen = 43008 ctx-size = 43008 ``` For fields where the lower or upper bound is determined by metadata, the comment must use that metadata-derived bound. ### Specific Derivation Direction The generator does not need exact benchmarking logic. It needs stable, explainable rules. Recommended direction: - `ctx-size` - maximum from metadata `context_length` - choose a practical default no higher than the metadata maximum - `n-gpu-layers` - maximum from metadata `block_count` - choose full offload by default when the maximum is known - `n-cpu-moe` - maximum from metadata `block_count` - chosen value derived from MoE metadata and model size pressure - `threads` - bounded by local CPU count - `parallel` - conservative for multimodal and MoE - `batch-size` and `ubatch-size` - reduced for conservative profiles, especially MoE and multimodal - `cache-type-k` and `cache-type-v` - may prefer `q8_0` for long-context or MoE models, otherwise a safer baseline - `mmap` - default toward the more practical behavior already used in `start-qwen3.6-35b-a3b.bat` ### Backup Policy Backup generation should no longer be the default behavior. Recommended implementation: - set CLI default to no backup - keep optional explicit `--backup` support only if trivial to preserve ## Architecture Keep the current single-script architecture, but tighten responsibilities: 1. GGUF metadata reader 2. metadata-only classifier 3. bounded preset value generators by class 4. comment renderer for min/max/chosen annotations 5. INI renderer ## Testing Tests should cover: - MoE classification depends on metadata `expert_count`, not filename text - dense models never receive `n-cpu-moe` - multimodal models are detected from sibling `mmproj` - range comments are present near the key generated fields - backup default is disabled - output remains valid INI text ## Risks - Some GGUF files may omit certain metadata keys; in those cases the script must use documented conservative fallback ranges without reintroducing filename heuristics. - `cache-type-k` and `cache-type-v` are policy choices rather than direct metadata facts, so the rules must stay simple and predictable. ## Implementation Notes The workspace currently lacks normal Git repository behavior for commit operations, so this spec is written locally without a commit step.