U Model Runtime — Implementation Plan

The goal: U becomes a complete inference runtime for the Safebox Infrastructure, replacing Python-based runners (vLLM, etc.) with a single compiled binary that loads any model format, manages KV caches with copy-on-write forking, and supports adapter hot-swap (LoRA, soft prompts, steering vectors) without retraining.

What exists today

The runtime (11,656 lines of C) has:

• Tensor ops (42), activations (9), NN layers (8), attention, RoPE, KV cache
• Safetensors loader (F16/BF16/F32/F64)
• u_llama_forward() — full transformer loop with SwiGLU, causal mask, top-p sampling
• u_safebox_runner.c — Unix-socket runner speaking the Safebox protocol, compiles to 34KB
• Autograd, Adam optimizer, gradient functions (for training)
• Dataframe + Column ops (for data preprocessing)

The five pieces

1. GGUF loader (~400 lines)

GGUF is llama.cpp's format. It's a flat binary: magic number, metadata key-value pairs (architecture, vocab, hyperparameters), then tensors with quantization type per tensor. The metadata tells you everything safetensors requires a separate config.json for — layer count, head count, vocab size, context length, rope frequency.

What to implement:

// Header:
[4 bytes] magic "GGUF"
[4 bytes] version (3)
[8 bytes] n_tensors
[8 bytes] n_kv_pairs
// KV pairs: each is type-tagged (string, uint32, float32, array, ...)
// Tensors: name + ndim + shape + type + offset into data section

Quantization types that matter:

The dequantize path is simple: for each block, multiply the stored integers by the block's scale factor to get floats. Q4_0 dequant is ~15 lines of C. Q8_0 is ~8 lines. The quantized matmul (keeping weights compressed during the multiply) is harder but optional — dequant-then-matmul works and is what most backends start with.

The metadata parser replaces config.json: read n_layers, n_heads, n_kv_heads (for GQA), d_model, vocab_size, context_length, rope_freq_base directly from the GGUF header. No separate file needed.

Effort: ~250 lines for the parser + metadata, ~150 lines for Q4_0/Q8_0/F16/F32 dequantization. The quantized-matmul optimization (keeping weights compressed during multiply) is a separate ~200-line pass that's worth doing later for performance but not required for correctness.

2. BitNet / ternary quantization (~80 lines)

BitNet 1.58b uses ternary weights: each weight is -1, 0, or +1. This transforms matmul from float multiply-accumulate into integer operations:

// Standard matmul: y[i] += w[i][j] * x[j]     (float mul + add)
// BitNet matmul:   y[i] += sign[i][j] * x[j]   (conditional add/sub)
//                  which is just: if +1, add x[j]; if -1, sub x[j]; if 0, skip

Packed representation: 2 bits per weight (00=0, 01=+1, 10=-1). A [4096, 4096] weight matrix that would be 64MB in F32 becomes 4MB in BitNet.

The kernel:

static inline void u_tensor_matmul_bitnet(
    const uint8_t* packed_w,  // 2 bits per weight, row-major
    const double* x,          // input vector [in_features]
    double* y,                // output vector [out_features]
    int32_t out_f, int32_t in_f) {
    for (int32_t i = 0; i < out_f; i++) {
        double sum = 0.0;
        for (int32_t j = 0; j < in_f; j += 4) {
            uint8_t byte = packed_w[(i * in_f + j) / 4];
            for (int k = 0; k < 4 && j+k < in_f; k++) {
                int val = (byte >> (k * 2)) & 3;  // 00, 01, 10
                if (val == 1) sum += x[j+k];       // +1
                else if (val == 2) sum -= x[j+k];  // -1
                // val == 0: skip
            }
        }
        y[i] = sum;
    }
}

With SIMD (AVX2/NEON), this becomes popcount-based and runs 10-20x faster than float matmul. The scalar version above is the starting point.

Where BitNet models come from: Microsoft's BitNet b1.58 paper, the 1-bit Llama variants on HuggingFace, and increasingly GGUF files with Q2_K quantization that approximate ternary. The loader detects the quantization type and dispatches to the right kernel.

3. Multi-KV-cache with CoW forking (~200 lines)

This is the most interesting piece architecturally. The insight: a loaded LLM's weights are read-only after loading. The KV cache is the only per-session mutable state. This means:

fork() gives you free KV cache copies.

Parent process:
  - Loads model weights (read-only, shared across all children)
  - Pre-computes system prompt KV cache (the "personality" warmup)
  - Listens on Unix socket

On each request:
  - fork()
  - Child inherits:
    - Model weights (CoW, never written → pages stay shared)
    - System prompt KV cache (CoW, will be extended per-conversation)
  - Child processes the user's tokens, extending its own KV cache copy
  - Child responds and exits (or stays alive for multi-turn)
  - Parent never touched → zero-copy model sharing

The memory savings are dramatic. A 4B parameter model at F16 is ~8GB. With fork(), 100 concurrent sessions share those 8GB (read-only pages). Each session's KV cache is ~50-200MB depending on context length. Without fork, 100 sessions would need 100 × 8GB = 800GB.

ZFS CoW for persistent KV caches.

The KV cache can be memory-mapped from a file on ZFS. ZFS clone gives instant copies:

# Save a "system prompt" KV cache:
zfs snapshot pool/kv-caches/system-prompt@base

# Clone per-session (instant, zero-copy):
zfs clone pool/kv-caches/system-prompt@base pool/kv-caches/session-42

# The session's KV cache extends from the snapshot
# Only modified blocks are new — everything else is shared

This is how you'd implement "conversation memory" — save the KV cache state at key points, clone it to branch conversations, roll back to an earlier point.

Implementation in the runner:

typedef struct USessionPool {
    ULlamaWeights* model;        // shared, read-only after load
    UKVCache*      system_kv;    // pre-computed system prompt cache
    int32_t        max_sessions;
    pid_t*         children;     // forked session processes
    int32_t        n_active;
} USessionPool;

// On request:
pid_t child = fork();
if (child == 0) {
    // Child: KV cache is CoW copy of system_kv
    // Extend it with the user's prompt
    // Generate response
    // Exit (or keep alive for multi-turn)
}

What the Infrastructure already has: ZFS is the storage layer. The system component manages ZFS snapshots and clones. The runner just needs to use mmap() for the KV cache file and let ZFS handle the CoW.

4. LoRA adapter hot-swap (~150 lines)

LoRA (Low-Rank Adaptation) adds small matrices to frozen base weights:

W_effective = W_base + (A × B) × scale

Where A is [d_model, rank] and B is [rank, d_model], and rank is typically 4-64. A LoRA adapter for a 4B model is ~10-50MB (vs 8GB for the full model).

How it works at inference time:

// Instead of: output = input @ W^T
// Do:         output = input @ W^T + (input @ B^T @ A^T) × scale

static inline UTensor* u_linear_with_lora(
    const UTensor* input, const UTensor* weight,
    const UTensor* lora_a, const UTensor* lora_b,
    const UTensor* bias, double lora_scale) {
    
    UTensor* base = u_tensor_linear(input, weight, bias);
    if (!lora_a || !lora_b) return base;
    
    // LoRA path: input @ B^T @ A^T × scale
    UTensor* lora_out = u_tensor_matmul(input, u_tensor_transpose(lora_b));
    lora_out = u_tensor_matmul(lora_out, u_tensor_transpose(lora_a));
    lora_out = u_tensor_scale(lora_out, lora_scale);
    
    return u_tensor_add(base, lora_out);
}

Hot-swap: LoRA adapters are just safetensors files. The runner loads a base model once, then loads/unloads LoRA adapters per-request from /srv/safebox/models/<adapter-hash>/. The base weights stay frozen in shared memory (via fork). Different users can have different adapters active simultaneously.

What LoRA adapters exist for:

• Voice/tone: fine-tuned to write in a specific style (formal, casual, technical)
• Domain expertise: medical, legal, financial vocabulary and reasoning
• Language: better performance in a specific language
• Task: better at code, better at summarization, better at chat
• Character: roleplay, persona, brand voice

LoRA adapter format (safetensors):

{
  "base_model_name_or_path": "meta-llama/Llama-3.1-8B",
  "lora_alpha": 16,
  "lora_rank": 8,
  "target_modules": ["q_proj", "k_proj", "v_proj", "o_proj"]
}

Weight names follow the pattern: base_model.model.layers.0.self_attn.q_proj.lora_A.weight and ...lora_B.weight. The loader matches these to the base model's layers.

5. Adaptation without retraining

Several techniques work at inference time with zero training:

a. Prompt caching (ready now)

Pre-compute the KV cache for a long system prompt once. Every subsequent request starts from the cached state instead of re-processing the system prompt. This is what the fork-based pool does — the system prompt KV cache is computed once in the parent, then CoW-cloned per-request.

For a 2000-token system prompt at Llama-8B, this saves ~3 seconds per request.

b. Representation engineering / activation steering (~50 lines)

Add a "steering vector" to the residual stream at specific layers during inference. The vector is pre-computed (offline, once) and shifts the model's behavior without modifying weights:

// During forward pass, after the residual addition at layer L:
// xx = xx + steering_vector * strength

static inline void u_apply_steering(UTensor* hidden, const UTensor* vector,
    double strength, int32_t layer, int32_t target_layer) {
    if (layer != target_layer || !vector) return;
    for (int32_t i = 0; i < hidden->size; i++)
        hidden->data[i] += vector->data[i % vector->size] * strength;
}

Steering vectors can encode: "be more truthful", "be more creative", "refuse harmful requests more strongly", "write in a formal tone". They're extracted by running the model on contrastive pairs (one formal, one casual) and taking the difference in hidden states.

c. Soft prompts / prefix tuning (~30 lines)

Instead of text tokens, prepend learned embedding vectors directly to the input. These are tiny (~1KB per soft prompt) and hot-swappable:

// Prepend soft prompt embeddings before the token embeddings:
static inline UTensor* u_apply_soft_prompt(
    const UTensor* token_embeds,    // [seq_len, d_model]
    const UTensor* soft_prompt) {   // [n_prefix, d_model]
    return u_tensor_cat(soft_prompt, token_embeds);  // [n_prefix + seq_len, d_model]
}

d. LoRA (see item 4) — the standard approach for "make the model sound like X". Pre-trained LoRA adapters for specific voices/styles are available on HuggingFace.

e. KV cache surgery (~80 lines)

Directly manipulate the KV cache to "insert" context the model didn't see. This is experimental but powerful: compute the KV cache for document A, then graft it into the KV cache for a conversation about document A — the model "remembers" the document without re-reading it.

// Graft: copy KV entries from source_kv (document cache) into
// target_kv (conversation cache) at a specific position:
static inline void u_kv_cache_graft(UKVCache* target, const UKVCache* source,
    int32_t layer, int32_t insert_pos, int32_t source_start, int32_t source_len) {
    memcpy(target->k_cache[layer]->data + insert_pos * target->d_model,
           source->k_cache[layer]->data + source_start * source->d_model,
           source_len * target->d_model * sizeof(double));
    // Same for v_cache
}

How it maps to Infrastructure

Infrastructure layer          What U provides
─────────────────────────     ─────────────────────────────────────
/srv/safebox/models/<hash>/   Base model (safetensors or GGUF)
                              LoRA adapters (safetensors, ~10-50MB)
                              Steering vectors (safetensors, ~1MB)
                              Soft prompts (safetensors, ~1KB)

ZFS pool/kv-caches/           System prompt KV cache (snapshot)
                              Per-session KV cache (clone)
                              Document KV caches (reusable)

Runner binary (34KB)          Loads model + adapters
                              fork() per-session (CoW weights)
                              Manages KV cache pool
                              Speaks Safebox protocol on Unix socket
                              Capability: o {} (no network, provable)

.u.meta in binary             Compiler-proven capability surface
                              No network → prompts can't leak
                              No filesystem write → model can't be modified
                              Deterministic → reproducible inference

Implementation order

Steps 1-5 are independent and can be done in parallel. Step 6 requires the Infrastructure's ZFS layer. Step 7 is a performance optimization.

What this gets you

With steps 1-3 done (~630 lines): U can load any model from HuggingFace (safetensors) or llama.cpp (GGUF), run inference with optional LoRA adapters for voice/style customization, and do it in a 34KB binary with provably no network access.

With steps 4-5 added (~410 lines): Multi-user inference where 100 sessions share one copy of the model weights, each with their own KV cache, pre-warmed system prompts, and per-user steering vectors. The whole thing runs inside the Safebox attestation boundary.

With steps 6-7 added (~300 lines): Persistent conversation memory via ZFS snapshots, and 4x memory reduction via quantized matmul. This is the full vision: a single binary that replaces vLLM + Python + CUDA with compiled C that runs anywhere, proves what it can do via .u.meta, and manages model state through the Infrastructure's existing ZFS and attestation layers.

What U does that llama.cpp doesn't

1. Provable capability surface. .u.meta proves no network access. llama.cpp has no such guarantee.
2. Adapter hot-swap. Load/unload LoRA per-request without restarting. llama.cpp requires restart for adapter changes.
3. KV cache forking. fork() for zero-copy multi-session. llama.cpp's server mode copies KV caches.
4. ZFS-integrated persistence. KV cache lives on ZFS; snapshot/clone/rollback are first-class operations. llama.cpp has no persistence model.
5. Compile to WASM. Same code runs in the browser for small models. llama.cpp has a separate wasm build with significant limitations.
6. Safebox protocol native. The binary speaks the Infrastructure's Unix-socket HMAC protocol directly. No adapter layer, no Python wrapper.
7. Autograd. The same runtime that runs inference can also fine-tune with LoRA training. llama.cpp is inference-only.