# AI in U — Inference, Training, and Model Adaptation

U compiles ML pipelines to native binaries (34KB) and WebAssembly. No Python, no pip, no CUDA toolkit. One language for data loading, model inference, training, and deployment.

## What U Can Do Today

### Load any major model format

```
import Safetensors
import GGUF

// HuggingFace models (safetensors):
weights = Safetensors.load("model.safetensors")
ww = Safetensors.get(weights, "model.layers.0.self_attn.q_proj.weight")
// Handles F16, BF16, F32, F64 — auto-converts to double.

// llama.cpp models (GGUF):
model = GGUF.parse(buffer)
ww = GGUF.get(model, "blk.0.attn_q.weight")
// Handles Q4_0, Q8_0, F16, F32, BF16 — auto-dequantizes.
// Metadata (layers, heads, vocab, context length) is in the header.
arch = GGUF.get_str(model, "general.architecture")  // "llama"
```

### Run transformer inference (Llama / Nemotron / Mistral / Qwen)

Every piece of a transformer forward pass is implemented and tested:

```
import NN
import Safetensors
import Tokenizer
import KVCache

f main()
    // Load model:
    sf = Safetensors.load("nemotron-nano-4b.safetensors")
    model = llama_load(sf, 32, 32, 3072, 9216, 131072)

    // Tokenize:
    tok = Tokenizer.load(vocab_text)
    tokens = Tokenizer.encode(tok, "What is the capital of France?")

    // Generate (autoregressive with KV cache):
    output = llama_generate(model, tokens, tokens.length, 128, 0.7, 0.9)
    print(Tokenizer.decode(tok, output))
```

The forward pass: embedding → per-layer (RMSNorm → Q/K/V → RoPE → multi-head attention with causal mask → KV cache → output projection → residual → RMSNorm → SwiGLU FFN → residual) → final norm → LM head → top-p sampling.

### Train models (XOR example — tested, trains to 4/4 accuracy)

```
import NN
import Tensor
import Autograd

f main()
    xx = Tensor.from_data([0,0, 0,1, 1,0, 1,1], [4, 2])
    yy = Tensor.from_data([0, 1, 1, 0], [4, 1])

    w1 = Tensor.he_normal([8, 2])
    b1 = Tensor.zeros([8])
    w2 = Tensor.he_normal([1, 8])
    b2 = Tensor.zeros([1])

    adam = NN.adam_new(4, 0.01, 0.9, 0.999, 1e-8, 0.0)

    [0..2000].on(epoch => (
        hidden = Tensor.relu(NN.linear(xx, w1, b1))
        pred = Tensor.sigmoid(NN.linear(hidden, w2, b2))
        loss = NN.mse_loss(pred, yy)
        // ... backward + adam_step
    ))
    // Result: [0.016, 0.992, 0.993, 0.007] — 4/4 correct
```

### Adapt models without retraining

**LoRA hot-swap** — load a 10-50MB adapter, apply per-request, no model reload:
```
import LoRA
import Safetensors

adapter = Safetensors.load("formal-voice-lora.safetensors")
lora = LoRA.load(adapter, 32, 8, 16.0)
logits = llama_forward_lora(tokens, n_tok, model, kv, lora)
// Switch voice by swapping adapter — same base model
```

**Steering vectors** — shift behavior with a ~1MB vector, adjustable strength:
```
import Steering
steer = Steering.load(sf, "formal_direction", 2.0, 15)
// Adds a direction to hidden states at layer 15
// "Be more formal" = one vector, "Be more creative" = another
```

**Soft prompts** — per-user tuning at ~1KB:
```
soft = SoftPrompt.load(sf, "user_alice_prefix")
combined = SoftPrompt.apply(token_embeddings, soft)
// Prepends 4 learned embedding vectors — invisible to the user
```

**KV cache grafting** — document memory without re-reading:
```
// Process a 50-page document once:
doc_kv = process_document(model, document_tokens)
KVCache.save(doc_kv, "/srv/kv-caches/doc-123.bin")

// Every future question about this document:
conv_kv = KVCache.new(32, 8192, 4096)
KVCache.graft(conv_kv, doc_kv, 0, 0, doc_kv.cur_len)
logits = llama_forward(question_tokens, n, model, conv_kv)
// Model "remembers" the document — no re-reading, no RAG chunking
```

### BitNet — 10x cheaper inference

```
import BitNet

bw = BitNet.quantize(weight)  // float → 2-bit ternary (-1, 0, +1)
result = BitNet.matmul(input, bw)  // add/sub only, no float multiply
// 4B model: 8GB at F16 → 1GB at BitNet. CPU-viable.
```

### Diffusion models — what's ready, what's not

U has the building blocks for Stable Diffusion inference:

| Component | Status | Notes |
|-----------|--------|-------|
| Linear layers | ✅ | Dense projections |
| Conv2d | ✅ | 2D convolution with stride/padding |
| MaxPool2d | ✅ | Pooling |
| LayerNorm / GroupNorm | ✅ | Normalization (GroupNorm = LayerNorm per-group) |
| GELU / SiLU | ✅ | Activations |
| Multi-head attention | ✅ | Self-attention and cross-attention |
| Safetensors loader | ✅ | Loads SDXL / FLUX weights |
| **UNet architecture** | 🔜 | Composing the above into residual blocks + down/up sampling |
| **VAE decoder** | 🔜 | Transposed convolution (learnable upsampling) |
| **DDPM scheduler** | 🔜 | Noise schedule + denoising loop (~100 lines) |
| **CLIP text encoder** | 🔜 | Transformer encoder (same attention, different architecture) |

The individual ops are tested. What's missing is the architecture composition (~400 lines) and transposed convolution (~50 lines). A diffusion model is structurally simpler than a transformer decoder — no KV cache, no autoregressive generation, just repeated UNet passes with decreasing noise.

### Server-side inference vs. client-side WASM

**Server-side** (the normal path): The u-runner binary loads the model, listens on a Unix socket, speaks the Safebox `Protocol.LLM.Local` wire format. The model stays on the server. Users send prompts, get responses. No model transfer.

```
// Server: load model, serve inference
./u-runner model.safetensors /run/safebox/services/u-llm.sock

// Client (Safebots/LLM.js, unchanged):
resp = await safebots.llm({
    model: 'local/nemotron-nano',
    system: systemPrompt,
    userInput: userMessage
})
```

**Client-side WASM** (for small models): U compiles to WebAssembly. A classifier, sentiment model, or embedding model can run in the browser with zero server roundtrip:

```html
<!-- In the browser: -->
<script>
  // Load the WASM-compiled U model:
  const wasm = await WebAssembly.instantiate(modelBytes, shim);
  wasm.exports.main();
  // Runs inference locally — no network, no latency, no server cost
</script>
```

Use WASM for: classification (<10MB models), embeddings, tokenization, sentiment, small transformers.
Use server for: LLMs (>1GB), diffusion, anything that needs a GPU.

### With TensorFlow models

TensorFlow models export to safetensors via the `safetensors` Python package:

```bash
# Convert once (Python):
from safetensors.torch import save_file
# or from TF: convert TF SavedModel → ONNX → safetensors
```

Then load in U:
```
weights = Safetensors.load("converted-model.safetensors")
```

U doesn't run TensorFlow graphs natively (and doesn't need to — safetensors is the universal weight format). For TensorFlow.js models specifically, the recommended path is server-side U inference rather than transferring multi-GB models to browsers. The u-runner serves the same purpose as TF.js serving but without the JavaScript overhead and with the `.u.meta` capability proof.

## U vs. vLLM vs. llama.cpp

| | **U** | **vLLM** | **llama.cpp** |
|-|-------|---------|--------------|
| **Binary size** | 34KB | ~3GB (Python + PyTorch) | ~2MB |
| **Cold start** | <1s | 10-30s | 2-5s |
| **Weight formats** | Safetensors + GGUF + BitNet | Safetensors | GGUF |
| **Quantization** | Q4_0, Q8_0, BitNet (ternary) | AWQ, GPTQ, FP8 | Q2-Q8, IQ, BitNet |
| **GPU** | `+V` → WebGPU (any GPU) | CUDA only (+ROCm) | CUDA, Metal, Vulkan |
| **Multi-user** | fork() CoW — OS-level isolation | Internal batching — shared process | Server mode — shared process |
| **KV cache** | ZFS-backed, persistent, forkable | In-process, volatile | In-process, volatile |
| **Adapters** | LoRA hot-swap per-request | LoRA (restart required) | LoRA (restart required) |
| **Training** | Autograd + Adam built-in | No | No |
| **Capability proof** | `.u.meta` — compiler-proven | None | None |
| **WASM** | Compiles to WASM | No | Yes (limited) |
| **Safebox protocol** | Native (Unix socket, HMAC) | Needs wrapper | Needs wrapper |

### Where U is better

**Provably safe inference.** The binary's `.u.meta` section proves its capability surface. No `o { Network }` = the binary provably cannot exfiltrate prompts. No sandbox to escape — the capability doesn't exist in the compiled code. Neither vLLM nor llama.cpp can make this claim.

**Persistent KV caches.** vLLM and llama.cpp lose all KV caches on restart. U saves them to ZFS. System prompts, document memory, conversation state — all survive reboots. Fork a KV cache for branching conversations. Snapshot for rollback.

**Per-request adapter swap.** Load a different LoRA adapter for each user without restarting the model. vLLM requires a restart for adapter changes. llama.cpp requires restart or pre-loading all adapters.

**Native Safebox integration.** The u-runner speaks the Infrastructure's HMAC-authenticated Unix-socket protocol directly. vLLM and llama.cpp need a Python/Node wrapper to translate.

### Where U is not yet better

**GPU performance.** vLLM's PagedAttention and continuous batching on CUDA are highly optimized. U's `+V` GPU dispatch is spec'd but not yet emitting WGSL/CUDA kernels. CPU inference only for now.

**Quantized matmul.** ✅ Now implemented. U does Q4_0 and Q8_0 matmul in-place — weights stay compressed, no dequantization allocation. A [4096,4096] weight matrix: F32 = 64MB, Q8_0 = 18MB (3.6x savings), Q4_0 = 9MB (7.1x savings). Results match the dequant-then-multiply path exactly.

**Continuous batching.** vLLM batches multiple requests into one GPU kernel call. U uses fork() for isolation instead. On GPU, batching wins. On CPU with many cores, fork() is competitive.

**Model coverage.** vLLM and llama.cpp have been tested against hundreds of model architectures. U has been tested against Llama-class (Llama, Nemotron, Mistral, Qwen). Other architectures (GPT-2, BERT, T5, Mamba) would need architecture-specific forward loops — the ops are all there, the composition differs.

### The honest summary

U replaces vLLM for **CPU inference on attested hardware** where provable safety, persistent KV caches, per-request adapters, and zero-dependency deployment matter more than peak GPU throughput. It replaces llama.cpp for deployments that need the Safebox governance chain, ZFS integration, and `.u.meta` capability proofs. It doesn't yet replace either for maximum-throughput GPU serving.


## Cache Breakpoint Protocol

U's runner implements a cache breakpoint protocol compatible with Anthropic's  API and the Safebots/LLM.js five-block prompt discipline. The protocol gives unlimited breakpoints (Anthropic's API limits to 4).

### How it works

The Safebots/LLM.js wrapper structures every prompt into four blocks:



Each breakpoint saves a KV cache snapshot. On the next call, the runner checks prefix hashes from deepest to shallowest. The deepest match gives the most cached tokens — only the suffix after that match runs through the model.

### Wire format



### How it maps to Anthropic / OpenAI

| Provider | Cache mechanism | U equivalent |
|----------|----------------|-------------|
| Anthropic |  on content blocks, max 4 breakpoints | Unlimited breakpoints, per-bot tenant isolation, LRU eviction |
| OpenAI | Automatic prefix detection (no explicit control) | Explicit prefix hashes — deterministic cache hits, no guessing |
| U runner | KV cache snapshots on ZFS, fork()-shared across sessions | Survives restarts, branchable, graftable |

### Integration with Safebots/LLM.js

The existing  wrapper already hashes the prefix and manages slots. The U runner replaces the slot backend:



The wrapper computes  for each block boundary and sends them to the runner. The runner looks up the deepest matching KV cache snapshot, grafts it into the session, and processes only the suffix. On a typical multi-turn conversation, the system prompt (500-2000 tokens) and pinned context (500-1000 tokens) are cached from the first call — subsequent turns only process the new user message.

## The Runtime — Full Inventory

**12,477 lines of C** in a single header (`u_runtime.h`):

| Category | Count | Functions |
|----------|-------|-----------|
| Tensor ops | 42 | zeros, ones, full, eye, from_data, arange, linspace, rand, randn, xavier, he, add, sub, mul, div, scale, neg, abs, sqrt, clamp, where, sum, mean, max, min, argmax, argmin, dot, norm, matmul, transpose, reshape, cat, stack, relu, sigmoid, tanh, softmax, gelu, swish/silu, leaky_relu, elu, relu6 |
| NN layers | 11 | linear, conv2d, maxpool2d, layer_norm, rms_norm, batch_norm, dropout, embedding, attention, rope, causal_mask |
| Training | 15 | autograd tape, leaf, backward, grad, sgd, sgd_momentum, adam, adam_step, zero_grad, lr_cosine, lr_warmup, lr_step_decay, grad_clip_norm, grad_clip_value, mse_loss, cross_entropy |
| Gradients | 10 | sigmoid, tanh, gelu, softmax, relu, mse, cross_entropy, layer_norm, linear (input+weight+bias), tape (add, mul, matmul, scale) |
| Safetensors | 3 | parse (F16/BF16/F32/F64), get, info |
| GGUF | 6 | parse, get (Q4_0/Q8_0/F16/F32/BF16), get_u32, get_f32, get_str, info |
| BitNet | 2 | quantize (float→ternary), matmul (add/sub only) |
| Transformer | 5 | llama_forward, llama_forward_lora, llama_load, llama_generate, sample_top_p |
| LoRA | 3 | lora_load, linear_lora, forward_lora |
| Steering | 3 | load, apply, compute |
| Soft prompts | 2 | load, apply |
| KV cache | 8 | new, append, advance, get_k, get_v, reset, save, load, graft |
| Session pool | 6 | mmap_open, mmap_close, pool_new, pool_warmup, session_handle, pool_reap |
| Tokenizer | 4 | load, encode, decode, encode_bytes |
| Dataframe | 15 | from_csv_string, head, sort, select, drop, filter_mask, unique, sample, count, groupby_agg, with_column, join, get_column, to_table |
| Column | 12 | sum, mean, min, max, gt, lt, eq, str_len, str_upper, str_lower, str_contains, to_tensor |

## Quick Start for the Safebox Infrastructure

```bash
# Build the runner (one command):
gcc -O2 -o u-runner u_safebox_runner.c -I runtime/ -lm
# Result: 34KB binary

# Deploy alongside existing vLLM runners:
cp u-runner /opt/safebox/runners/
./u-runner /srv/safebox/models/<hash>/model.safetensors \
           /run/safebox/services/u-llm.sock

# The runner speaks the same protocol as vLLM:
# Safebots/LLM.js → Protocol.LLM.Local → u-runner
# No code changes in Safebots or Safebox.
```