# U Language — Plan (v3): The Web Story, WASM, and Closing the Last Language Gaps ## Where things actually stand (measured, not assumed) The original 29-turn plan is complete. On top of it this project added a full crypto stdlib and standard-conforming post-quantum signatures (SPHINCS+, byte-identical to the reference). Release engineering has begun: `u2c` is now pip-installable and `u2c build` compiles a `.u` file to a native binary in one command, both verified in a clean virtualenv from the packaged artifact. **Suite: 1070 tests** (parser 120 · codegen 599 · linter 345 · optimize 6), all green. All 31 stdlib modules parse. A spec-vs-implementation audit (U.html section by section against the runtime, codegen, and linter) shows the core language is deeply real: the modifier system (+R/+M/+N/+V/+A/+E), Q rational arithmetic, MVCC (tested under 8-thread contention), stackful fibers with multiple suspend points, +A fork/join, automatic backpressure, and context-aware templates all have passing tests. The honest remaining gaps are small and named below. --- ## The architecture this plan serves (now fully clarified) U's web story has **two independent halves**, and it's worth stating why: 1. **Frontend — "play with U in the browser."** Other languages translate *into* U (the JVM/CLR move: be liberal in what you accept, strict in what you emit). `JS-U.zip` is the starting point: a client-side playground (CodeMirror + acorn) that translates JavaScript, TypeScript, and Python into U live in the browser. Today it is a rough prototype with real bugs. It needs to become correct and trustworthy — the shop window people first touch. 2. **Backend — "the real compiler, in the browser."** The Python `u2c` is the **reference implementation**: deterministic, readable, the source of truth for what U *means*. Porting it to the web is done by compiling that same Python to **WASM** (via Pyodide/emscripten), so the browser runs the identical reference — not a reimplementation that could drift. This is the deterministic, single-source-of-truth path. **Why Python, recorded as rationale (belongs in implementation.html):** Python was chosen because the strongest LLM/codegen tooling lives there, which matters for a language whose thesis is "written by LLMs, verified by a compiler." The alternative would have been writing the compiler in C — the very language we target — which ships with every OS and would have made the toolchain trivial to distribute. We deliberately traded that distribution simplicity for tooling strength, and the WASM port buys back the distribution: the Python reference runs everywhere the web does, deterministically, without a reimplementation. **Two levels of WASM, not to be confused:** - **U -> C -> WASM** (a *U program* running in the browser) — via Emscripten on the emitted C. Covered in plan-v2 Phase D. - **`u2c` (Python) -> WASM** (the *compiler itself* running in the browser) — via Pyodide. Covered here in Phase W. Together they mean a browser can take U source, compile it with the real reference compiler, and run the result — all client-side, all deterministic. --- ## Discipline (unchanged) Spec CODE BLOCKS are authority. Every milestone verified against a REAL external artifact (published vector, reference implementation, a real language's own runtime, or the Python reference's own output). The feature matrix stays honest: Works / Simplified / Placeholder. No regressions; all suites green at every gate. implementation.html gets the "what + why" for each milestone as it lands. --- ## PHASE F — Harden the frontend playground (JS-U.zip -> trustworthy) ✅ COMPLETE **Status: DONE.** All 64 bundled examples emit U the real `u2c` accepts (JavaScript 22/22, Python 22/22, TypeScript 20/20), verified every turn by the headless harness against the actual reference compiler. The one construct with no U equivalent (Python `*args`/decorator) is labelled in place with a `// [not translatable to U]` comment rather than mistranslated. The playground went from not loading at all → 8 → 13 → 15 → 24 → 29 → 31 → 47 → 57 → 59 → 60 → 62 → 63 → 64 of 64 over the phase, each step a spec-grounded fix measured against the reference. The playground is the first thing anyone sees. Right now the translators are hand-written AST visitors with visible bugs (JS strings calling `.strip()`, `node` vs `n` variable confusion, a `pname` used-before-declared, `while`/`for` lowerings that emit questionable U). "Rough demo" is fine for a prototype; it is not fine for the shop window. Goal: every bundled example translates to U that the *real* `u2c` accepts, and failures are honest. **F1 — Fix the known bugs in the JS visitor.** The undefined-variable bugs (`pname`, `node` where `n` is in scope), the invalid `.strip()` on JS source strings, and the malformed control-flow lowerings. Each bundled JS example must produce output with no `/* NodeType */` fallbacks unless genuinely untranslatable. *Checkable:* every JS example's output, fed to the Python `u2c` linter, produces zero parse errors (or a clearly-marked "unsupported" comment, never a silent mistranslation). **F2 — Same for the Python and TypeScript paths.** The Python translator is a line-based indent parser (fragile); the TS path is regex type-stripping. Tighten both so the bundled examples round-trip to `u2c`-valid U. *Checkable:* every Python and TS example's output passes the `u2c` linter or is explicitly flagged. **F3 — Ground-truth the playground against the reference.** The playground's translations are only trustworthy if the U they emit actually compiles. Add a harness (run in this repo, not the browser) that takes every playground example, runs it through `u2c`, and asserts it type-checks — turning "looks like U" into "is valid U." This is the frontend's version of the reference-diff discipline. *Checkable:* a test iterates the extracted examples through `u2c` and fails on any that don't lint clean; wired into the suite so playground drift is caught. **F4 — Honest "what doesn't translate" surface.** When a construct can't be expressed (metaclasses, `eval`, prototype surgery, arbitrary `for`/`while` that doesn't fit `.on`), emit a clear diagnostic in the console pane naming the limitation — never a plausible-but-wrong translation. This mirrors the matrix's "name the gap" rule. *Checkable:* a curated set of known-untranslatable snippets each produce a named diagnostic, asserted by the harness. **Gate F:** every bundled example lints clean through the real `u2c`; the untranslatable set is named, not mistranslated; JS-U.zip is rebuilt and the harness is in the suite. --- ## PHASE G — Close the last language gaps (spec parity) From the U.html audit, these are the genuine unimplemented or Simplified items. Small list — the language is otherwise real. **G1 — Template metaprogramming (TemplateLang).** ✅ RESOLVED (honest scope). Note: the plan mislabelled this "+D" — in the spec `+D` means *Depends* (ambient mutable state), NOT metaprogramming. The metaprogramming facility is the `TemplateLang` interface (`__check__`/`__validate__`/`__output__`) behind backtick tags (SQL`, HTML`, Regex`, ...). Audit found it is NOT silently absent: tags parse and get distinct types, interpolations are full expressions that are scope- and type-checked (`__check__`), Regex enforces its raw-S injection guard, and SQL/HTML correctly accept an S in value position (parameterized / escaped). The one genuine gap is grammar-aware `__validate__` per tag — e.g. SQL structural-position injection (`SELECT * FROM {{table}}`), which needs the sql module to parse SQL grammar at compile time. The SQL structural-position case is now IMPLEMENTED (Works): a raw S after FROM/JOIN/INTO/UPDATE is a compile error, value-position S stays parameterized, sub- templates compose. Lexical position analysis (clause keyword before the interpolation), not a full SQL parser — catches the common structural slots the spec names. HTML deep grammar validation remains trust-the-module. A near-miss this pass: a blanket "reject all raw S in SQL" broke correct value-position parameterization (2 codegen tests) and was reverted — the existing turn-137 reading (only Regex has no safe escape) is the defensible one. **G2 — `+V` vectorization** ✅ IMPROVED (honest scope). Audit: `f+V` emitted BYTE-IDENTICAL C to a plain function — the modifier was accepted, type-checked, and then erased (zero codegen effect). Fixed the tractable, verifiable part: a `f+V` function now carries `__attribute__((optimize("tree-vectorize","O3")))`, so gcc/clang auto-vectorize element-wise loops in its body into SIMD regardless of the caller's -O (verified: "loop vectorized using 16 byte vectors"; plain functions are NOT tagged; result unchanged, checked by compile-and-run). Exact remaining scope, in the matrix: (1) it is an auto-vectorization HINT, not explicit lane/warp intrinsics or GPU codegen; (2) the inline `grid.on(f+V (px) => …)` handler syntax NOW PARSES and its .on()/forEach loop gets a #pragma GCC ivdep (verified vectorized + correct); the inlined .map()/.filter() lowerings now emit _Pragma("GCC ivdep") too (a #pragma line is invalid inside their statement-expression, so _Pragma is used); remaining: push-based append loops vectorize less readily than pure-store .on() loops; (3) the spec's "+V forces a compile error if not vectorizable" rule is NOW ENFORCED — a +V map/filter/on handler that mutates outer scope or calls an impure function is a compile error (accumulate/reduce forms exempt as the sanctioned reduction); remaining: the purity check is conservative (sound, not maximally permissive), and these are hints not lane/warp intrinsics or GPU. Named, not silent. Guarded by a test. **G3 — Generics: type-erasure fixed** ✅ IMPROVED (honest scope). Audit found worse than "sharp edge": a generic field `item: ElemType` was emitted verbatim as `ElemType item;` — an undefined C type — so the generated C did NOT compile at all (no void* fallback; just broken C). The earlier "works" row only exercised linting and the single-letter-T `Box[I]({...})` construction path. Fixed: a type-parameter field is type-erased to `void*`, a scalar is boxed at construction and unboxed on read, for MULTI-LETTER params (ElemType/Key/Val), and both construction forms (`Box({...})` and `Box[I]({...})`) agree. Verified by COMPILING the emitted C with gcc and RUNNING it — Box[I] round-trips 42, Box[S] round-trips a string. Remaining scope, stated in the matrix: this is type-erasure (one void* representation), not full monomorphization (a distinct specialized struct per instantiation); a generic field of class/collection type uses the erased representation. Scalar + pointer type args work end-to-end. A compile-and-run test now guards it. **G4 — Real publish signatures** ✅ DONE — cosign is now a verified asymmetric signature over the module hash (u2c/pubsign.py, forgery-resistant, honest scope: EdDSA-style, not claimed RFC-8032 Ed25519). Matrix promoted to Works. Turns 27–29 built the provenance chain with placeholder signatures; the crypto stack now has real ECDSA/SPHINCS+. Wire actual signatures into `u publish`/`cosign`. *Checkable:* `u verify` validates a real signature produced by the crypto runtime, not a stand-in. **Gate G:** ✅ SATISFIED. G4 Works (real signatures), G1 named Simplified (template __validate__ scope), G3 improved to Works-with-compile-and-run (type-erasure; monomorph scope named), G2 improved + named Simplified (vectorize hint; lane/warp + enforcement scope named). Every gap is either Works-with-a-test or Simplified-with-exact-scope. No silent gaps remain. --- ## PHASE W — `u2c` in the browser (Python reference -> WASM via Pyodide) The deterministic reference compiler, running client-side. This is what makes the playground's U output *runnable*, not just viewable, and it does so with the identical reference implementation rather than a JS reimplementation that could diverge. **W1 — Pyodide load of `u2c` unmodified.** ✅ DONE — Get the Python `u2c` package running under Pyodide in a page: import it, call `transpile(u_source)`, get C back. No compiler changes should be needed if the package is import-clean (it is; it's pure Python with no C extensions). *Checkable:* a page loads Pyodide + `u2c`, transpiles `examples/hello.u` to C in the browser, and the C is byte-identical to the native `u2c` output for the same input (determinism across the port — the whole point). **W2 — U -> C -> WASM -> run, fully in-browser.** 🔶 PARTIAL — emitted C proven WASM-ready (test-enforced); in-browser emcc run pending a toolchain. Combine W1 with plan-v2 Phase D: the C that `u2c` emits is compiled to WASM (a prebuilt Emscripten pipeline or clang-in-wasm) and executed, so the playground can show program *output*, not just translated source. *Checkable:* the browser runs `hello.u` end-to-end (U -> C via Pyodide-`u2c` -> WASM -> stdout) and prints "Hello from U!"; a SPHINCS+ keygen in-browser produces root `2f94dbe8...`, byte-identical to native — a perfect determinism oracle. **W3 — Wire the playground to the real backend.** ✅ DONE — validation loop proven in WASM; it caught that the frontend was measured on parse, not full compile (Any is not a real type). Corrected: true validity 40/64 against the full type-checker. Replace/augment the JS-side "U output" pane so that, alongside the frontend translation (Phase F), the Python-reference `u2c` validates and compiles it. Now the shop window is powered by the source of truth. *Checkable:* toggling "compile & run" in the playground executes the U through the Pyodide `u2c` path and shows real output/diagnostics. **Gate W:** the reference compiler runs in the browser producing byte-identical C to native; a U program compiles and runs client-side; the playground is backed by the real compiler. Documented honestly (load time, size, which C->WASM path). --- ## PHASE R — Release 1.0 (ship it, with its own provenance) Carried/expanded from plan-v2 Phase E. **R1 — Editor + docs finish** (plan-v2 A3–A5): JSON diagnostics mode, a TextMate grammar, and a QUICKSTART whose every code block compiles. **R2 — Dogfood the provenance chain:** sign U's OWN release with `u.lock` + Merkle root + M-of-N publish + transparency log (now with real signatures from G4). The compiler that verifies supply chains ships with a verifiable one. **R3 — Final honest reconciliation:** matrix <-> implementation.html <-> stl.html <-> paper, so no claim outruns evidence. Tag 1.0. *Checkable:* clean-machine install -> build -> run -> `u verify` all pass; every matrix claim has a test or a named-gap label. --- ## Explicitly NOT in this plan (named, not hidden) - **Kyber / Dilithium (ML-KEM / ML-DSA):** Placeholder; lattice schemes, a separate build. Addable later via the proven pull-the-reference-and-diff method. - **Native LLVM/WASM backend & GPU (+G):** Phase-2-later. WASM here is via Emscripten (programs) and Pyodide (compiler), not a native backend. - **Constant-time / side-channel-hardened crypto:** reference-correct today, not hardened; its own effort. - **Full LSP:** the JSON diagnostics mode (R1) is the seed; a complete language server is future work. --- ## Order & how much is left **F (frontend) -> G (language gaps) -> W (compiler-in-browser) -> R (ship).** - **F** makes the thing people touch trustworthy — highest visible leverage, moderate size (fixing real bugs + a validation harness). - **G** is small: four named items, most of which are "promote or precisely scope," not net-new subsystems (G1 `+D` is the one real design question). - **W** is the flagship: the reference compiler in the browser. W1 is small if the package is import-clean; W2 depends on the C->WASM path from plan-v2 D. - **R** is packaging + dogfooding provenance. **Bottom line on "how much is left":** the *language and its compiler are done and proven*. What remains is (1) making the browser playground actually trustworthy (F), (2) closing four small, named gaps to spec parity (G), (3) putting the real reference compiler in the browser so people can compile and run U client-side, deterministically (W), and (4) shipping 1.0 with U's own provenance (R). The core is behind us; this plan is about reach — getting U into people's hands on the web. ## Consolidation audit (matrix vs. compiler) A clean-room audit verified every Works row compiles/runs and every Simplified/ Placeholder scope statement is accurate. Findings: - FIXED (real bug): explicit generic construction `Box[I]({...})` emitted a non- pointer local (`Box x = Box__new(...)`) that didn't compile — the `[]`-BinOp callee bypassed the pointer-decl heuristic. Now both `Box({...})` and `Box[I]({...})` emit `Box*` and run. Pinned by test. - DOCUMENTED (limitation, not a silent gap): a subclass arg to a U-level base- typed parameter (`describe(sh: Shape)` with a `Shape.Circle`) is rejected. Accepting it at the linter is unsound today — class params emit as value types and method calls are static, not vtable dispatch — so C wouldn't compile or dispatch. Reverted the tempting linter change; named the boundary in the matrix + a pinning test. The vtable mechanism works at the C level (inheritance test); U-level base-parameter polymorphism does not yet. - VERIFIED accurate: CLI (--version, -o, console entry point), real signatures (forgery-resistant), SQL structural check, and the Simplified/Placeholder rows (constant-time caveat; Kyber/Dilithium unimplemented). ## Spec-vs-impl audit + constant-time crypto Audited U.html feature surface against the compiler (compile-AND-run). Findings: - FIXED (2 real bugs, same pointer-decl family): annotated `d2: Base = d1.c()` copy and `Box[I]({...})` explicit generic construction both emitted non-pointer locals that didn't compile. Both now run; pinned by tests. - GAP (largest): +F Function Domain Modifier — a full spec section (8 subsections) essentially unimplemented (parser rejects `fn: +F`; 0 tests). Now an explicit Placeholder row in the matrix, not silently absent. - Everything else verified: async/streams/MVCC/templates/db/crypto/generics run. - CONSTANT-TIME CRYPTO: added ctcrypto.ct_equal (position-independent timing, measured <3x first-vs-last ratio vs ~160x for naive early-exit) + ct_select; wired into pubsign.verify's final point check; added branchless u_ct_equal to the C runtime (genuinely constant-time on target). Scope: comparison leak closed; field/scalar arithmetic not yet constant-time; CPython has an interpreter floor — full CT belongs in the C runtime. Matrix row updated. ## +F core forms + +V real SIMD by default - +F Function Domain Modifier: core forms now compile AND run — bare `T +F`, typed `T +F(S)`, nullary `T +F()`. Emits real C function pointers (c_type + c_decl helper), resolves function-name args to mangled symbols, excludes +F functions from inlining. Fixed a pre-existing class-return bug (functions returning a class now pointer-typed) + extended the pointer-decl heuristic to class-returning calls. Matrix: Placeholder -> Simplified. NOT done: +F +R/+A/+N oracle combinations and [T +F] collections. (Type note: `(I,I)->S` == `S +F(I,I)` — same type; only +F spelling parses in param position today.) - +V real SIMD: CORRECTED the "hints only" claim. The runtime already had real vendor-agnostic SIMD (GCC vector extensions, vector_size(32)) — but behind an off-by-default U_VEC_ENABLE, so emitted calls never LINKED and old tests only checked the call string. Restructured: SIMD tier always compiled (map/sum run by default, verified pmuludq/paddd in asm, compile-LINK-run tests), thread fork-join tier stays opt-in with a serial fallback. Fixed a latent include- guard bug (vec section was outside #endif U_RUNTIME_H, double-defined on re-include). Matrix: Simplified -> Works. WASM: same vector types lower to WASM SIMD 128 via emcc -msimd128. GPU: OpenCL kernel-STRING builder only, no dispatch backend — real vendor-agnostic GPU (SPIR-V/WGSL) named out of scope. ## +R(GPU) — WebGPU/WGSL compute kernels (real, execution-verified) Built the GPU backend's core, verified by RUNNING (not just header-checking): - u2c/codegen/wgsl.py: emits WGSL compute shaders for the element-wise class — map (arithmetic over the element) and atomic i32 sum reduction, storage/global memory only (portable, no vendor extensions or workgroup tiling). - lower_handler_to_wgsl: lowers a real U handler AST → WGSL expression, enforcing lane-representability (rejects outer capture + impure calls → stays on CPU). - Discovery: wgpu-py (0.32) has a working WebGPU device via llvmpipe software backend, so emitted WGSL is VALIDATED AND EXECUTED end-to-end — px*3+1 → correct, f32 map → correct, atomic sum → 55. Same compile-AND-run bar as the rest. - emit_host_map_c: host C over the standardized webgpu.h (fetched real header), compiles clean — but NOT executed (no emcc/Dawn), dispatch body partly elided. Honest tier-2 (compiles-against-real-header) vs tier-1 (runs-on-device). - tests/test_wgsl.py: 6 tests, skip cleanly without wgpu/cc. Matrix: +R(GPU) Simplified with exact two-tier scope. NOT done: workgroup-memory tiling (matmul), full host dispatch executed E2E, wiring +R(GPU) into the .map() codegen route (still CPU fallback there). Model note: host+R → device+R(GPU) upload = the spec's explicit residence copy; async readback maps onto +A (designed for, not yet lowered). ## +R(GPU) remainder — wired through compiler + full host dispatch - transpile_gpu() entry point: a +R(GPU) .map() now routes through the WGSL path (WGSL kernel + host-C dispatch), collected as separate artifacts. Default transpile() unchanged (CPU fallback, compiles with plain gcc). - FULL LOOP VERIFIED: U source -> compiler -> emitted WGSL -> executed on real WebGPU device -> correct (map(px*3+1) → [4..31]). tests/test_wgsl.py now 8 tests incl. the compiler-wiring + device-execution path. - Host dispatch COMPLETED (no longer elided): shader module, storage buffers, bind group layout+group, pipeline, compute pass, dispatch, copy-to-readback, submit — all 14 wgpu* calls verified present in the real webgpu.h; compiles clean. Only the final mapAsync readback callback elided (event-loop-dependent). - +A: the async readback is DESIGNED to lower onto +A (GPU dispatch = pending value) but NOT yet lowered through the fiber scheduler — named honestly. - Regression caught+fixed: my edit dropped a set_types_used init line (24 tests went red, restored, back to green). The suite caught it. NOT done: workgroup tiling (matmul); host executed E2E natively/WASM (only WGSL half runs here); +A scheduler lowering. ## +R(GPU) — host now RUNS natively + workgroup tiling (tier-1 across the board) Breakthrough: wgpu-py bundles libwgpu_native-release.so + matching webgpu.h/wgpu.h, so the emitted host C can be LINKED and RUN natively — not just compiled. - emit_host_map_runnable_c(): complete host incl. async mapAsync readback driven by wgpuDevicePoll. Compiles+links+RUNS against libwgpu_native → correct output ([4..31]). Host path: tier-2 (compiles) → tier-1 (RUNS). All 14 wgpu* calls verified in the real header; hand-checked the async callback-info API shape. - emit_tiled_reduce_sum_shader(): workgroup-memory (var) tree reduction — the DATA-REUSE tier (matmul/stencils technique), previously scoped OUT. Emitted + RUNS on device (128 elems → partials → 8256 correct). - tests/test_wgsl.py now 10 tests: WGSL execution, native host execution, tiled reduction execution, compiler wiring, lowering safety. Skip cleanly without wgpu/cc. Matrix: +R(GPU) Simplified → Works. Honest remaining: async readback RUNS (polled) but not lowered through U's fiber scheduler as a T +A pending value (host blocks on poll vs suspends a coroutine); browser/WASM execution unverified (no emcc; API identical via emdawnwebgpu); full tiled MATMUL (vs tiled reduction) not emitted. Named as next steps. ## +A fully lowered onto the fiber scheduler (GPU async, runs) The big remaining piece — GPU readback as a first-class T +A pending value, not a blocking poll loop: - u2c/runtime/u_gpu_async.h: a GPU dispatch (u_gpu_map_dispatch) returns a PENDING FIBER FRAME whose prefix matches UGenericFrame (header/parent/ resume_point/status). u_gpu_map_step (signature = UFiberRunFn) polls the device once per scheduler step, returns U_FIBER_SUSPENDED until mapAsync lands, then U_FIBER_DONE. The scheduler drives it cooperatively. - VERIFIED BY EXECUTION against real wgpu-native: frame is SUSPENDED right after dispatch (non-blocking); a concurrent fiber interleaves BEFORE the GPU completes; result correct. The readback suspends a coroutine, not a thread. - emit_gpu_async_glue_c(): compiler emits per-kernel WGSL constants + a dispatch wrapper returning the pending frame. Compiles against runtime + webgpu.h. - tests/test_wgsl.py: 11 tests (added the +A fiber-integration run + its driver fixture gpu_async_driver.c). Matrix: +R(GPU) stays Works, +A caveat closed. Honest remaining: browser/WASM execution unverified (no emcc; native runs); tiled MATMUL (vs tiled reduction) not emitted; auto-await INSERTION at the U source call site reuses fetch()'s fork/await path but is exercised via runtime glue, not a full end-to-end U-source async GPU program. ## Math standard library (real, libm-backed, tested) Built the missing math stdlib — the "boring essential" gap flagged in the readiness review: - Registered as builtins in linter (double/N domain): sqrt, cbrt, sin/cos/tan, asin/acos/atan, sinh/cosh/tanh, exp, ln, log2, log10, floor, ceil, round, trunc, fabs, pow, atan2, hypot, fmod, fmin, fmax, copysign, sign, + constants pi()/e()/tau(). - Codegen emits DIRECT libm calls (portable: libm native + Emscripten → WASM unchanged). ln→log, sign→(x>0)-(x<0), constants→literals. - User function of same name OVERRIDES builtin (decl wins over decl=None sig); fixed the registration + codegen guard, tested. - 12-case compile+run test. codegen 613→614. Matrix: new Works row (79 total). Remaining: Math. namespace-qualified spelling (Math.sqrt) not wired — bare names work; needs namespace resolver to treat Math as built-in (small follow-up). ## Sockets (TCP/UDP + epoll reactor) and SQLite bindings — both RUN - u2c/runtime/u_net.h (353 lines): TCP listen/connect/accept/read/write, UDP bind/sendto/recv. All non-blocking, all returning PENDING FRAMES (+A shape, UGenericFrame prefix) driven by the fiber scheduler. THE NEW MACHINERY: an epoll REACTOR. A bare ready-queue can't drive a socket read (the frame would spin — nothing says when the fd is readable), so readiness wakeups now come from epoll and the process sleeps instead of busy-polling. PROVEN: accept with no client suspends 5/5 steps; epoll reports 0 ready before connect, 1 after; frame completes on wake. Also verified: full TCP echo + UDP datagram through the scheduler. Scope: NATIVE (POSIX epoll). Browsers forbid raw TCP -> WASM backend is WebSockets/fetch (one interface, two backends, like GPU). Not done: TLS, DNS (IP literals only), async connect/write frames, kqueue/IOCP, wire-protocol clients (MySQL/Postgres/HTTP) that sit ON this layer. - u2c/runtime/u_sqlite.h: complete bindings vs real libsqlite3 3.45 — open/exec/ prepare/bind/step/columns/finalize, transactions, last_insert_id, changes. Full CRUD RUNS correctly (3 inserts, bound SELECT -> 2 rows, rollback restores). All values bound via sqlite3_bind_* (never concatenated) — runtime half of the linter's static SQL-injection guarantee. Works on both targets (WASM via sql.js pattern). Not done: U-level ORM sugar, BLOB binding. - tests/test_io.py: 3 tests (TCP/UDP echo, reactor suspend/wake, sqlite CRUD), skip cleanly without POSIX/libsqlite3. ## Multiplexing (stream_select) + Scheduler (clock/timers/cron) - u_net_select(): PHP stream_select equivalent, but fd-VALUE-AGNOSTIC by design. select(2)'s fd_set bitsets can't see fd >= FD_SETSIZE(1024) and WASI RANDOMIZES fds across [3,2^31) -> bitset select silently drops sockets; 0..max_fd loops become billions of iterations (this is exactly what bit the PHP-on-WASM port). So: explicit fds[] in, explicit ready[] out. epoll backend natively; same surface maps to poll()/wasi:io/poll on WASI. - u_sched.h: MONOTONIC clock for deadlines (NTP/DST-immune) vs REALTIME for timestamps/cron — separated by construction. Timers: after/every/cron + cancel + fire counts. Cron: 5 fields, * numbers lists ranges steps, Sun=0|7. 13 matcher cases verified. KEY INTEGRATION: timers and IO share ONE wait — u_sched_next_timeout_ms() feeds the epoll timeout, so the loop sleeps until whichever comes first. PROVEN: 6 timer wakeups while a listener idled, and the socket still seen readable through the same select. - tests/test_io.py now 4 tests (added scheduler driver). WASI/WasmEdge research (2026): the 2022 WasmEdge-extension route is now LEGACY — WASI Preview 2 stabilized 2025 and standardizes wasi-sockets (TCP/UDP) + wasi-http, portable across Wasmtime/WasmEdge/WAMR/wazero. WASIp3 (in dev) adds native async. Note: WASI P2 is SYNCHRONOUS with no in-WASM cooperative concurrency — which is exactly what U's fiber+reactor model already provides. Target strategy: keep the one-interface/two-backend split (native epoll | WASI poll), NOT a WasmEdge-specific binary. ## Website (site/) — deployable, responsive, metadata-complete Built site/ as a self-contained deployable web presence: - index.html — hero + SIGNATURE element: an interactive modifier strip. Chips are magnet poles (+ red / - blue); clicking swaps in REAL u2c output. - learn.html — 10-lesson tour (functions -> GPU kernel) with sticky TOC + scrollspy. - playground.html — real compiler output per example, live U highlighting on your own code, tab-indent handling, and SHARE VIA URL HASH (works today, no server). Honest notice: compiling arbitrary code in-browser awaits the u2c WASM build. - snippets.html — 9 copyable snippets, each with copy / open-in-playground / copy-share-link. - u_language.html, implementation.html, FEATURE_MATRIX.html, web-implementation.html brought in with injected metadata + a shared site navbar. Design: palette derives from a horseshoe magnet (red + pole / steel-blue - pole) which IS U's modifier syntax — pole colors carry meaning wherever a modifier appears, including in the syntax highlighter. Type: Space Grotesk / IBM Plex Sans / JetBrains Mono. Assets: ORIGINAL U-magnet logo drawn with PIL (not the flaticon icon — that carries attribution obligations); favicon .ico + pngs, apple-touch-icon, and a 1200x630 OG card. Every page has title + description + keywords + OG/Twitter tags + favicons. Verified: 0 broken links, metadata complete on all 8 pages, HTML structurally valid, highlighter unit-tested under node (pole colors, escaping/no-injection, share round-trip). Examples are generated BY THE COMPILER (assets/examples.js) — including a real WGSL kernel — so nothing on the site is hand-faked output. ## WASM build PROVEN + networking library designed in U WASM: installed clang-18 (wasm32 backend) + wasi-sysroot-24 + compiler-rt builtins. U -> C -> wasm32-wasi -> RAN IN NODE with correct results (scale3(7)=22, hypotSq(3,4)=25). So a WASM build is doable here, not hypothetical. Two real findings for the WASM target: (1) U's runtime uses setjmp -> needs `-mllvm -wasm-enable-sjlj` (WASM EH). (2) U emits `static` functions, so NOTHING is exported to the wasm table. A real --target=wasm needs the compiler to emit export_name attributes; I used hand-written shims to prove execution. Networking library designed IN U (stdlib/net.u) following spec §10.5 exactly: adapters implement z f __check__/__validate__/__output__/__schema__ + f __execute__ -> R+A; capabilities ride the `o` line with SQUARE BRACKETS selecting the connection: `o Database[Database.Connection.Postgres] { get, insert }`. One `sql` tag, engine chosen by connection — MySQL writes ?, Postgres writes $1, U source never sees the difference. Errors extend the spec's NetworkError tree. Reads/writes speak [B], the spec's protocol boundary type. net.u PARSES with the real parser. COMPILER BUG FOUND BY WRITING THE DESIGN (and fixed): the `!` error-return parser consumed only ONE identifier, so `NetworkError.Protocol.SSL` left `.SSL` dangling ("'SSL' is not defined") — blocking the spec's own error taxonomy. Now consumes the full dotted name (both `!` and `x:` forms). Test added. net.u lint errors 25 -> 15; the remaining 15 are Type[T]/Schema/L — spec compile-time reflection types not yet implemented, i.e. real unimplemented features, not design errors. ## stdlib: time, env, filesystem, async HTTP — all RUN - u_time.h: MONOTONIC vs REALTIME separated by construction (conflating them is the classic timer bug). Calendar components UTC/local, ISO-8601 format+parse with verified round-trip, strftime formatting, duration humanising. clock_gettime is POSIX + wasi-libc -> same source native and WASM. - u_env.h: env get/has/or/set/get_int, args, exit, cwd, pid, hostname. One interface, three backends: native full; WASI host-supplied env/argv; browser has NO process env -> degrades (empty/-1) instead of failing. - u_fs.h: read_all/write_all/append (size-capped + NUL-terminated), stat, mkdir/mkdirs/remove/rename, list, basename/extension/join. WASI note: host must PRE-OPEN dirs (capability-based, no ambient disk authority). - u_http.h: async HTTP/1.1 client built ON u_net.h, NOT libcurl — which is the proof the socket layer is a sufficient transport. State machine (connect/send/ read) driven by the fiber scheduler; suspends to the epoll reactor between reads. VERIFIED vs a real loopback server: GET 200 + body, POST echo, 404, with 17 suspensions and a concurrent fiber interleaving 3 steps (genuinely async, not a fast path). Scope: no TLS/chunked/redirects/keep-alive/cookies. - tests/test_io.py now 6 tests. Matrix +4 Works rows. Include-order note: u_time/u_sched need _DEFAULT_SOURCE before any system header (feature-test macros must precede them); compile with -D_DEFAULT_SOURCE under strict -std=c11. ## Compile-time reflection — DESIGN (stdlib/reflect.u) CENTRAL CLAIM: `z` is a DOMAIN MODIFIER for time-of-existence, the same shape as +R (residence) and +A (readiness): +A value exists LATER (pending at runtime) z value exists EARLIER (compile time; erased before runtime) So reflection is NOT a bolted-on macro language — it is U's existing modifier idea on one more axis. Type/Field/Method/Diagnostic/Position are ordinary `d` classes walked with ordinary .map/.filter. No macro DSL, no second evaluator. (Contrast: C++ templates and Rust macro_rules! are second languages; Zig comptime is the good precedent.) Staging rules mirror residence rules: z code reads z values + Types; runtime code CANNOT read a z value (same error class as touching a +R(GPU) buffer from host code without the copy); a z f may RETURN a Type, which becomes the call site's static type; z f is pure + must terminate under a step budget, with __schema__ the one sanctioned compile-time I/O. VERIFIED the design is implementable: extracted real Type/Field/Method values from live linter state for a real class hierarchy — fields, declaration order, modifiers, inheritance, params and return types all already present (class_fields, class_field_order, class_table, class_type_params_of, functions). Reflection EXPOSES the symbol table; it computes nothing new. RECONCILED two conflicting spec drafts (found by writing the design): - `z f` vs `fz f` -> keep `z f` - `L` vs `[Diagnostic]` -> keep `[Diagnostic]` (says what it is) - `Type` vs `Type[T]` -> just `Type`; a z f returning Type sets the call site's static type, so no second form OPEN (named, not hidden): (a) union types — the spec's HTML example returns `Templates.HTML | S | I | N`; cheapest fix is `alternatives: [Type]` on Type rather than surface-syntax unions; (b) evaluation order — a z f returning a Type makes type-checking and compile-time eval mutually recursive, so cycles must be detected not hung on; (c) how much of U runs at compile time — smallest useful subset first (literals, arithmetic, comparison, field access, collection pipeline over reflected data). reflect.u PARSES with the real parser. ## Generics + union types: verified, and two real bugs fixed "Make sure it all works" -> ran the whole [T] / union surface through parse -> lint -> codegen instead of assuming. GENERICS: fully working, all 8 positions reach codegen — d Box[T] declaration, multi params (Pair[Key, Val]), param type Box[I], generic field read, explicit construction Box[I]({...}), generic return type, [Box[I]] list-of-generic, nested Box[Box[I]], and [[I]] nested lists. Nothing to fix. (Note: U requires MULTI-LETTER type param names except the conventional T — Pair[K,V] is rejected by design. Reflection reads params off class_table[name].type_params.) UNION TYPES: two genuine bugs, both fixed. (1) Unions were first-class in PARAMETER position but not RETURN position, so `f pick() -> S | I` rejected its own `r => 1` ("expects S|I, got I"). check_type_compat now tries each alternative, which preserves every existing rule (aliases, generics, numeric widening, nullability) instead of re-implementing comparison. Non-members are rejected with a message that names the alternatives. (2) An alternative's OWN modifiers were never consumed by the parser, so `Cat +R | Dog +R` left `+R` in the token stream and corrupted the parse of everything after it — surfacing as a bogus "Return outside function". Member mods are now consumed and merged onto the union (base 'Cat|Dog', mods ['+R']). `S | I +N` works too. Tests added for both; codegen 615 -> 617. ## CORRECTION to the reflection design (I had this wrong) L is U's LOGICAL/BOOLEAN type (-> bool); B is BYTE (-> uint8_t, the protocol boundary type). I had previously dismissed the spec's `L` as leftover shorthand and recommended [Diagnostic] on that basis. The reasoning was wrong even though the recommendation stands: the two spec drafts are genuinely DIFFERENT designs, not different spellings — adapter draft: __check__ -> L (boolean: is this slot valid?) TemplateLang draft: __check__ -> Type (what type is REQUIRED here) __validate__ -> [Diagnostic] (what is wrong, with messages) Type + [Diagnostic] is still the right choice, because a boolean cannot say what was expected or what went wrong — but it is a real design decision, not cleanup. Also fixed my own misuse: reflect.u and net.u had boolean fields typed B (Byte); now L. Both still parse. ## Compile-time reflection Phase 1 — the model (u2c/reflect.py) Implements what stdlib/reflect.u designs: RType / RField / RMethod built from existing linter state. Nothing new is computed — every fact comes from class_table, class_fields, class_field_order, class_field_defaults, functions, type_aliases. ROBUSTNESS, the two things that break naive reflection: UNIONS — modifiers after a union bind to EVERY alternative (Greg's rule), and splitting respects nesting: Box[A|B]|C is TWO alternatives. A union inside a list keeps the list's mods on the LIST. GENERICS— declaration PARAMS (T in d Box[T]) vs use-site ARGS (I in Box[I]) are separate facts, both captured; nesting survives Box[Box[I]]. Also: inheritance (parent fields first, declaration order preserved), lists, maps, nullability, param types, and declared-vs-synthesised defaults (class_field_defaults holds an entry for EVERY field; the synthesised one is None, so hasDefault means a real declared default). TWO ALIASING BUGS found by testing, both now regression-tested: 1. Generic instantiation MUTATED the cached class — Box[I] appended type_args onto the shared Box, so repeated lookups accumulated and Box[I] eventually contained itself: infinite recursion. 2. The cache returned the modifiers of whoever looked a class up FIRST, so a later mods-free Cat came back still wearing +R. Fix: cache holds a CANONICAL modifier-free type; every lookup returns a copy. CLI: `u2c --reflect file.u` dumps the whole model as JSON (useful now for editors, doc generation, adapter authors — no compile-time evaluation needed). SCOPE: Phase 1 is the MODEL only. Not yet callable from U source. Phase 2 is z f compile-time EVALUATION; Phase 3 is z f returning a Type that feeds back into type-checking. The adapter hooks need Phase 3. ## stdlib wave: TLS, DNS, regex, encoding, random, compression, cookies All built and VERIFIED BY RUNNING (tests/test_io.py now 8 tests, all pass). - u_tls.h (OpenSSL 3.0): TLS 1.3 handshake vs a real HTTPS server (TLS_AES_256_GCM_SHA384), encrypted GET -> 200 + body. Non-blocking handshake reports WANT_READ/WANT_WRITE so a fiber suspends to the reactor. Verification AND hostname checking ON by default (OpenSSL does NOT check hostname for you). TEST ASSERTS THE PROPERTY THAT MATTERS: verify-on REJECTS a self-signed cert. TLS 1.2 floor. Scope: client only, no resumption/client certs; on loopback the handshake never returned WANT_* so the suspend path is implemented but unexercised. - u_dns.h: getaddrinfo; returns ALL addresses (happy-eyeballs/failover), v4+v6, /etc/hosts, IP-literal detection. - u_encoding.h: base64 (+URL-safe), hex, percent, UTF-8. Rejects malformed input rather than guessing — junk base64, odd-length hex, overlong UTF-8, surrogates. - u_random.h: CSPRNG (u_rand_secure_*) vs seedable xoshiro256** (u_rng_*), kept separate by name because picking the wrong one is the classic security bug. Rejection sampling avoids modulo bias. CSPRNG failure never downgrades silently. UUIDv4 always from CSPRNG. - u_regex.h: POSIX ERE via libc regex.h (also in wasi-libc, so both targets). search/groups/test/count/replace-all; zero-width matches advance so no spin. Named honestly as ERE not PCRE. - u_compress.h: gzip/zlib/raw deflate + crc32 (verified vs 0xCBF43926). - u_cookie.h: full jar. Security properties tested, not just features: Secure never over http; host-only doesn't leak to subdomains; and the suffix attack is blocked (example.com cookie NOT sent to evilexample.com). NOT DONE this wave (named): structured logging, testing framework, subprocess spawn; HTTP/2, HTTP/3, FTP; MySQL/Postgres wire clients (still blocked on reflection Phase 3 for the adapter hooks). ## Reflection Phase 2 + Phase 3 — z f evaluation and type feedback PHASE 2 (evaluation): extended u2c/ct_eval.py (which already folded scalar z f calls) with the reflection half: - reflection builtins typeNamed / typeOf / allTypes / fieldsOf, evaluated against u2c/reflect.py - member access on reflected values (.kind, .name, .fields, .type.name, .isClass, .isUnion, .typeParams, ...) - collection pipelines over reflected data: .len .map .filter .join .first .last .contains, with LAMBDAS - string ops, guard chains (U has no ternary: `cond ? value` lines with `!` as default), list literals - the existing 100k step budget covers all of it, so a runaway compile-time loop is a compile error, not a hung build Registered Type/Field/Method/Diagnostic/Position as builtin classes so `-> Type` type-checks, and the builtins so z f bodies lint clean. PHASE 3 (type feedback): a z f declared `-> Type` PRODUCES a type rather than having one. Hooked into call resolution: evaluate the hook and record the returned RType as the expression's static type. Mismatches report against the EVALUATED type ("'val' expects Point, got Row"). Guard chains can pick a different type per branch. Re-entrant evaluation on the same node is reported as a compile-time type CYCLE rather than hanging. BUG FOUND+FIXED en route: _to_literal emitted typ='string'/'bool' but the parser uses 'S'/'L'. It was dead code until Phase 2 made string folding reachable, at which point a folded string emitted as (null). Now matches the parser. Tests: 2 new cases; codegen 618 -> 620. STILL BLOCKING THE ADAPTERS: __schema__ (compile-time I/O to load an external schema) is not implemented, so MySQL/Postgres adapters can evaluate hooks but cannot yet check real column names. NOT DONE THIS TURN (carried): TLS session resumption + client certs + exercising the WANT_* suspend path; structured logging; testing framework; subprocess spawn; HTTP/2, HTTP/3, FTP; the MySQL/Postgres wire clients themselves. ## Sprint close: __schema__, logging, subprocess, test framework, mTLS __schema__ (the last adapter blocker) — compile-time I/O in ct_eval: schemaFrom(path, table) -> Type, schemaHas(path, table, col) -> L. Deliberately narrow because it is the ONE hole in z-purity: read-only, JSON only (no code executed), memoised per path so evaluation stays deterministic, and a missing file/table is a COMPILE ERROR — never a silent empty schema, which would make every column check pass. Verified: a typo'd column name is caught at compile time. u_log.h — structured logging. Typed fields, JSON correctly typed (strings quoted, numbers/bools not) and escaped; text sink for humans. stderr, so stdout stays program output. u_proc.h — fork/exec with explicit argv. NO command-line entry point on purpose (system() = shell = injection). Captures stdout/stderr/exit code; missing program -> 127. Native only (no process model on WASI or browser). u_test.h — small runner + assertions that report WHAT differed. Dogfooded by the logging and subprocess tests. TLS — mutual TLS now works against a CERT_REQUIRED server behind a real CA: client cert accepted, absence rejected, mismatched cert/key caught at load. REAL BUG FOUND BY TESTING: client certs were set on the SSL_CTX, but SSL_new has already copied the context — the cert was never sent. Under TLS 1.3 the client cert goes out AFTER the client thinks the handshake finished, so this presented as a successful handshake then zero bytes read. Now set on the SSL. Session tickets: the NewSessionTicket callback fires and the session is accepted by a new connection, but an ACTUALLY RESUMED handshake is NOT demonstrated — OpenSSL reports the session non-resumable against the test server and SSL_session_reused() stays 0. Test asserts only what is proven. tests/test_io.py now 10 tests, all passing. STILL NOT DONE (unchanged, explicitly out of this sprint): HTTP/2, HTTP/3, FTP; the MySQL/Postgres wire clients (now genuinely unblocked — transport, TLS and __schema__ all exist); demonstrated TLS resumption; exercising the WANT_* path. ## FINAL BLOCKER CLEARED: a real PostgreSQL wire client u_pgsql.h — PostgreSQL v3 wire protocol over U's OWN sockets (u_net.h) and U's OWN crypto. No libpq. SCRAM-SHA-256 auth is assembled from SHA-256 + HMAC + PBKDF2, all of which the runtime already had — that is the only reason a real authenticating client was buildable at all. VERIFIED against a live PostgreSQL 16: SCRAM handshake; SELECT with named columns and ordered rows; INSERT with command tag; SQL NULL distinguished from empty string; a server error surfaced with the connection still usable after. Test seeds its own fixture (the first version asserted row counts left by a previous run — not a test). Scope: simple query protocol only. No prepared statements / binary params, no pooling, no TLS-wrapped connection (both halves exist, not yet joined), no LISTEN/NOTIFY or COPY. Also fixed: u_hex_encode collided between u_encoding.h and u_runtime.h with DIFFERENT signatures (mine bounded, the runtime's not). Renamed mine to u_hex_encode_n rather than silently shadowing either. TEMPLATE FINDING (answering the completeness question): template TAGS are NOT wired — sql"...", html"...", fetch"..." all fail with "'sql' is not defined". Plain {{ }} interpolation works. The adapter hook machinery now exists (reflection 1-3 + __schema__) but the TAG SYNTAX is not connected to it. ## CORRECTION: template tags ARE wired (I tested the wrong syntax) Greg was right. The tag syntax is UPPERCASE + BACKTICKS: HTML`...`, SQL`...`, CSS`...`. I had tested html"..." / sql"..." (wrong case AND wrong delimiter) and wrongly concluded the tag system was unwired. Retested properly: - HTML`...` -> type HTML, CSS`...` -> type CSS (tags produce their OWN types, so an HTML value nests into an HTML slot without re-escaping) - SQL value slot: parameterised, clean - SQL structural slot: correctly REFUSED ("cannot interpolate a raw S into a structural position") So templates were ~80% done, not 0%. ADDED the one spec-mandated rule that was missing: an interpolation inside an on* attribute (onclick/onmouseover/onerror) is now a COMPILE ERROR, because that value is parsed as JavaScript and HTML escaping cannot make it safe. The spec's TemplateLang example raises CompileError for exactly this. href/class/content slots remain allowed. Helper _html_attr_before finds the unclosed attribute opener before a slot, the same grammatical approach as the SQL structural rule. REMAINING template gaps (named): per-attribute TYPE rules (href should demand a URL type, style a CSS type) are not enforced; a nested template inside a single-brace expression slot ({xs.map(ii => HTML`...`)}) does not resolve. ## MySQL: NOT DELIVERED, and why MariaDB 10.11 installs and starts, answers one query, then exits repeatedly in this sandbox. Without a server that stays up I cannot verify a MySQL wire client, and shipping an unverified one would be exactly the failure this project has avoided everywhere else. The Postgres client (verified, SCRAM-SHA-256, live server) is the pattern to follow; MySQL needs mysql_native_password (SHA1-based -- u_sha1 is already in the runtime) plus its packet framing. HTTP/2, HTTP/3, FTP: not started. HTTP/2 needs HPACK + framing + stream multiplexing; HTTP/3 needs QUIC (its own transport, congestion control and loss recovery) -- these are each comparable in size to the whole TLS+socket layer, not increments on it. ## MySQL client delivered — and WHY it failed before u_mysql.h: v10 handshake, mysql_native_password auth built on the runtime's own SHA-1 (SHA1(pw) XOR SHA1(salt + SHA1(SHA1(pw)))), text query protocol, length-encoded integers, column defs, row parsing, NULL handling, typed errors. No libmysqlclient. VERIFIED vs live MariaDB 10.11: auth, SELECT with named and ordered columns, INSERT affected-rows, NULL vs empty string, server error with the connection still usable. ROOT CAUSE of the earlier failure: background daemons do NOT survive between tool calls. mariadbd started fine, answered one query, then was gone by the next call -- it looked like a crash and was not. Starting the server and running the test in the SAME call fixed it immediately. (The Python HTTP/TLS servers only ever worked because they were always started and used within one call.) The suite test now starts its own server, seeds its own fixture, and skips cleanly when mariadbd is unavailable. Scope: text protocol only -- no prepared statements/binary params, no caching_sha2_password (MySQL 8 default; the auth-switch request is DETECTED and reported rather than mishandled), no TLS-wrapped connection, no compression. ## Case sensitivity: kept, plus near-miss hints U stays case-sensitive for classes, functions and methods. Verified current behaviour: wrong-case type uses and wrong-case calls are both rejected, while naming CONVENTIONS are not enforced (d point and f GetName() compile) -- the rule is identity, not style. Added the ergonomic half: an unknown identifier differing from a known class/function/alias only by case now reports "did you mean 'HTML'? names are case-sensitive". Forgiveness without losing the signal, and applied uniformly rather than PHP's split between names and vars. ## STILL NOT DONE: HTTP/2, HTTP/3, FTP Unchanged and deliberately not started. HTTP/2 needs HPACK + framing + stream multiplexing; HTTP/3 needs QUIC (its own transport, congestion control, loss recovery). Each is comparable in size to the entire socket+TLS layer. ## Prepared statements, unix sockets, WebSockets - u_pgsql.h: PREPARED STATEMENTS via the extended query protocol (Parse/Bind/Describe/Execute/Sync). Statement and values travel as SEPARATE wire messages, so a parameter cannot be parsed as SQL. Verified live: prepared INSERT/SELECT, NULL param -> SQL NULL (not the string "NULL"), and the decisive case: "greg'; DROP TABLE pp; --" binds as a VALUE, matches nothing, table survives with all rows. Scope: text-format params; binary format wired but not exercised per-type. - u_net.h: UNIX DOMAIN SOCKETS (u_unix_connect/u_unix_listen). Local DB connections use them by default; filesystem permissions are the access control; nothing exposed to the network. - u_websocket.h: RFC 6455 client. NOT on HTTP/2 -- it is an HTTP/1.1 Upgrade then its own framing over TCP, TLS just wraps it. Small because U already had TCP + HTTP/1.1 + SHA-1 + base64 + secure random. Verified: handshake with the accept hash VERIFIED (skipping it trusts any 101), masked client frames (servers drop unmasked -- classic first bug), extended lengths >125, auto PONG, clean close, and rejection of a non-upgrading server. BUG FOUND: u_net.h's include guard closed at line 387 of a 439-line file, so the multiplexing section appended earlier sat OUTSIDE it and redefined itself when two modules both included the header. Same bug class as u_runtime.h's vec section. Guard moved to EOF; double-inclusion now tested. PROCESS: background daemons do NOT survive between tool calls -- this bit twice (MariaDB, then Postgres). Servers must be started in the same call as the test. NOT DONE: MySQL prepared statements (COM_STMT_PREPARE + binary result rows); HTTP/2, HTTP/3, FTP. TLS/HTTP2 note: HTTP/2 does effectively require TLS, because browsers only do h2 over TLS with ALPN negotiating "h2" (ALPN is already in u_tls.h). DH params are server-side only and TLS 1.3 makes them moot (ECDHE, standard named groups). ## GUI.md — the template model settled, and the GUI consequences Settled the template model through discussion (Greg's design, my earlier framings were wrong twice and corrected): `foo {{bar}}` raw template, slots untyped HTML`foo {{bar}}` tag types the slots -> Html +F(S) HTML`foo {{bar}}`({bar}) call with data -> Html HTML(`foo {{bar}}`) SUGAR for the line above The PARENS ARE THE INVOKE MARKER. HTML(...) yields Html; HTML`...` yields a template. Lifting a template out for reuse is deleting one pair of characters. Slots: {{name}} = a PARAMETER of the template function; {expr} = a U expression captured from lexical scope. THE COMPILER CURRENTLY HAS THIS BACKWARDS. Consequence: slots are ARGUMENTS, NEVER TEXT, so safety stops being a per-tag rule -- no tag can forget, because no tag concatenates. That dissolves the SQL-vs-HTML asymmetry I had been arguing for. Handlebars falls out as a SIBLING of HTML (both Template[Html]), differing only in who calls them and when -- which is exactly Qbix's addTemplate pattern (one artifact, callable server- or client-side). Qbix alignment (checked against qbix.com/LLM.txt): tools stay a MICROFORMAT (data-* on ordinary elements + Q.activate), templates stay Handlebars {{ }}, events stay delegated+namespaced -- and U's on* prohibition ENFORCES the Qbix convention rather than fighting it. No JSX, no vDOM: Qbix does fine-grained tool.rendering() updates and the runtime already has tree diff/merge. U's contribution over current Qbix: move runtime failures to build time -- misspelled tool names, bad option keys, missing template slots. Wrote GUI.md (295 lines), every section marked IMPLEMENTED / PARTIAL / DESIGN. Verified the checkable claims: tool markup lints clean, on* refused, SQL structural refused, HTML|S|I|N union expressible, HTML +F(S) types a template. Implementation order: braces -> tags-as-functions -> per-position __check__ -> SQL prepare/execute lowering -> Handlebars -> tool registry -> webview shells. ## Spec templates section extended; web.html written (GUI.md retired) FOUND: the spec ALREADY stated the model we converged on by discussion -- "Template tags are compile-time functions that return compiled runtime functions... the inline form SQL`...`(args) compiles once and calls immediately; the stored form const q = SQL`...` is a prepared statement." We rediscovered it. What was MISSING and is now added as u_language.html sections 15.1-15.3: 15.1 THE SLOT MAP. A template function takes {S : }, and each tag publishes that union as an alias: d HtmlSlot : HTML | S | I | N | [HTML] d SqlSlot : S | I | N | L | [B] HTML`

{{name}}

` : HTML +F({S : HtmlSlot} +R) SQL`SELECT ... {{id}}` : [Row] +F({S : SqlSlot} +R) +A VERIFIED all four forms lint clean today (alias syntax is `d Foo : Bar` with no body -- `a` is await, not alias). 15.2 The two brace forms, and the consequence: slots are ARGUMENTS NEVER TEXT, so safety stops being a per-tag rule. 15.3 The invoke sugar: parens around the literal mean call-with-scope. Plus an explicit implementation-status note so the spec does not overclaim. web.html (326 lines) replaces GUI.md, in site style, nav-wired into all pages, 0 broken links, structurally valid. New content beyond GUI.md is section 6: - TWO ENGINES, five shells: WebKit (WKWebView, WebKitGTK) and Chromium (WebView2, Android). Both modern, both run the Qbix JS unchanged. - The JS->native channel differs per host; a GENERATED shim normalises them. The WebKit family shares an identical JS call, so there are 3 variants not 5. - EVERY host requires UI-THREAD MARSHALLING (WKWebView main thread, Android UI thread, WebView2 pump, GTK main loop) -> the scheduler needs a post-to-UI queue alongside the reactor. This is the one place the concurrency model has to grow to meet the GUI. - Asset delivery: custom scheme handler beats file:// (origin restrictions break fetch); all four hosts support one. - iOS is the cheapest mobile target (Obj-C is a C superset); Android is the only one where the shell is a subproject, not a shim (JNI + Java UI thread). ## CORRECTION to the template status wording (not drift, and not a swap) Greg asked whether the status note implied compiler DRIFT. It did not, and the wording was imprecise. Re-verified the exact behaviour: {{nm}} -> requires nm IN SCOPE: immediate lexical capture {nm} -> literal text, ONE part, not an expression slot at all HTML(`...`) -> "HTML is not defined": tags are syntax, not function values SQL -> u_str_concat("SELECT ... n = ", nm) So "the brace meanings are reversed" was WRONG. They did not swap: {{ }} does the job { } should do, and { } does nothing. The fix is one CHANGE plus one NEW FEATURE, not a swap. Corrected in both web.html and spec 15.3. NOT DRIFT: nothing regressed. The template tags predate this conversation; what we ADDED here was the on* prohibition, the union-type fixes, reflection Phases 1-3, and __schema__. The brace and SQL-lowering gaps are pre-existing state that testing surfaced. The suite has been green throughout. ## CORRECTION + first implementation step: slots are PARAMETERS CORRECTION I OWED: I invented a "{expr} = captured expression inside a template" rule. It does not exist. Checked the spec as Greg asked: "{foo, bar} is type-directed: target {K:V} or no annotation -> property shorthand ({foo: foo, bar: bar})" "One {} at the end: anonymous destructured map, defaults allowed" "Constructors follow the same rule: fields without defaults are positional, fields with defaults go in the trailing { } bag." So inside a backtick literal there is exactly ONE special form: {{name}}, the slot. A single { is an ordinary character (which is why CSS braces need no escaping). The braces at the CALL site are U's ordinary map literal, and the slot map IS the trailing {} bag -- the universal function/constructor rule, not a template invention. Because the bag destructures into named parameters, the compiler need not materialise a map for a literal call site. Fixed u_language.html 15.2 and web.html accordingly; both had my invented rule. Also corrected the status wording: it was never "brace meanings reversed" -- the real gap is that a {{ }} slot was resolved against LEXICAL SCOPE instead of becoming a parameter. IMPLEMENTED (step 1 of the plan): a {{slot}} is no longer scope-checked. - check_TemplateLiteral collects node.slot_names instead of resolving bare identifiers; non-identifier parts (a nested template) are still walked. - `HTML`

{{nm}}

`` with nm nowhere in scope now lints clean. - All safety rules still fire: on* refused, SQL structural refused, Regex raw-S refused, nested templates still type-checked. - One existing test asserted the OPPOSITE ("interpolations must be scope-checked") -- it encoded the capture semantics, so it was updated to the new model rather than worked around. - New test pins slots-as-parameters. codegen 621 -> 622. GOTCHA worth recording: that loop appears TWICE in linter.py, so my first patch silently hit the wrong copy and the edit was lost; the assert that should have caught it was written against a variant without the trailing newline. Patched by locating the loop INSIDE check_TemplateLiteral by line, then verified the change was on disk before trusting it. STILL TO DO: tags as constructors (HTML(`...`) sugar), the template-function TYPE (HTML +F({S : HtmlSlot} +R)), __check__ per position, SQL prepare/execute lowering, Handlebars. ## JS template tag — values into JavaScript Added the JS tag (14th). Slot union: S | I | N | L | Tree | JSON. The constructor JSON-encodes each slot so the result is a valid RIGHT-HAND SIDE. THE POINT: JSON encoding is NECESSARY BUT NOT SUFFICIENT. Checked the runtime's existing u_json_str -- it is a correct JSON encoder but escapes neither `<` nor U+2028/9, so its output inside BREAK-OUT — JSON does not escape `<`, so a string containing "" terminates the script element and the rest parses as HTML. `< > &` -> \u003c \u003e \u0026. 2. U+2028 / U+2029 — legal in a JSON string, historically ILLEGAL as literal characters in a JS string literal (older engines SyntaxError). Escaped. 3. POSITION — a JSON value is safe on the RHS of an expression; it is NOT safe inside a JS string literal or a JS TEMPLATE literal, where ${...} re-opens interpolation and turns data back into code. Those are a COMPILE ERROR, not an escaping problem: no encoding makes them safe. Linter: _js_open_quote tracks quote state (backslash escapes and // comments included) so an apostrophe inside a double-quoted string does not read as an opener. Also handles single quotes and backticks. Runtime: u_js_string / u_js_int / u_js_num / u_js_bool / u_js_null, plus u_js_from_json which re-escapes an existing JSON document while preserving structure. NaN/Inf have no JSON spelling -> null rather than a syntax error. Buffers are refused (-1), never truncated. TEST BUG worth recording: my first U+2028 test used "a\xe2\x80\xa8b..." — C hex escapes are GREEDY, so "\xa8b" parses as hex a8b and overflows. Split the string literals; the encoder was correct all along. A release guard pins the exact tag set, so adding JS broke it by design — updated. Tests: linter positional rules + a 6-case runtime encoder driver. codegen 621 -> 623, io 14 -> 15. Documented in all three: u_language.html 15.4, web.html 2b, FEATURE_MATRIX row (104 Works). Both edited HTML files re-verified structurally valid. NOT DONE: __validate__ is not called, so the JavaScript itself is not parsed. ## CORRECTION: < > escaping belongs to COMPOSITION, not the JS encoder Greg was right. `<` is unremarkable in JavaScript; the hazard belongs to the HTML PARSER, which scans for ` element regardless of JS string context. So over-escaping it in the JS encoder was wrong -- it makes a .js file ugly for a hazard that does not exist there. Split accordingly: - u_js_string: JSON escaping + U+2028/U+2029 (INTRINSIC to JavaScript) - u_js_for_script: breaks up `` is the same string to a JS parser. Applied by the tag that OWNS the context, which is the same principle the rest of the template system follows. Verified: ordinary `<` survives untouched through both paths; still cannot close the element when composing. ## Invoke sugar + Handlebars implemented - parser: Tag(`...`) recognised as an invoke form (IDENT '(' TEMPLATE ')'), setting node.invoked. - linter: the INVOKED form scope-checks its slot names (the sugar supplies the map from scope, so a missing name is an error); the BARE form leaves them as parameters. That gives the two spellings genuinely distinct semantics rather than being notation. - Handlebars registered as the 15th tag -- a SIBLING of HTML, slots deferred to the browser, needing no U binding. - The tag-set release guard broke twice by design (JS, then Handlebars) and was updated both times. Tests: invoke sugar (bare vs invoked, both tags, safety rules survive) and Handlebars. codegen 623 -> 625. STILL NOT DONE: __check__ per position; SQL lowering to prepare/execute (codegen still emits the capture form for BOTH spellings, so the invoked form does not yet lower to a real call); the HTML tag does not yet apply u_js_for_script automatically; __validate__ is never called. ## Finished: __check__ per position, SQL lowering, HTML script context 1. HTML PER-POSITION SLOT TYPES (the spec's __check__ design): href/src/action -> URL, style -> CSS, and inside position rule now requires a JS value, so the two halves meet. Tests: SQL statement/params (incl. the injection payload staying out of the text) and HTML per-position rules. codegen 625 -> 627. STILL NOT DONE: tags are a fixed built-in set rather than user-definable constructors; __check__ is a built-in table rather than a z f hook; the built statement is not auto-executed (binding a connection to a call site is the remaining step); __validate__ is never called. ## SQL[Conn] — the connection rides in BRACKETS Greg asked: first parameter, or the [connection] syntax we already have? BRACKETS, and the reason is not aesthetic: the connection selects the DIALECT, and the dialect decides the PLACEHOLDER SYNTAX the statement is compiled with (Postgres $1, MySQL ?). That is a COMPILE-TIME decision -- a runtime first argument arrives too late, because by the time you hold a connection value the statement text is already built. Brackets are also already U's compile-time parameter syntax (Box[I]) and already the spec's way of selecting a connection: `o Database[Database.Connection.Replica] { get, find }` (parses and lints clean today). Implemented: - parser: Tag[Conn]`...` and Tag[Conn](`...`), recording node.connection; backtracks cleanly so ordinary indexing (xs[1]) is unaffected. - codegen: _sql_dialect() picks the placeholder spelling, following BOTH links -- a subclass parent AND a type alias -- so `d AppDb : Database.Connection.MySQL` inherits `?` without repeating itself at each call site. - values stay bound parameters in every dialect; the bracket only changes the placeholder spelling. Verified by emission for default / Postgres / MySQL / alias-inherited. Distinction worth keeping straight: the bracket names a connection TYPE (compile-time: dialect, and later the schema for column checking). A runtime connection INSTANCE is still needed to EXECUTE -- that binding is the remaining step before SQL(`...`) actually runs a query. MY TEST BUG: wrote xs[0] to check that indexing still parses -- U lists are 1-INDEXED, and the compiler said so. Fixed to xs[1]. codegen 627 -> 628. ## SQL[dialect] — the engine as a type argument (already wired; now proven) Greg's design: the compile-time input is the DIALECT, not the connection. Placeholder syntax, quoting and type spelling are static properties of the engine; the connection is a runtime concern. So the engine rides in a type argument on the tag. FOUND: this was already wired but dormant. The parser sets node.connection from SQL[X]`...`, and _sql_dialect walks the connection's parent chain -- but nothing had ever supplied the bracket, so the mechanism had never run. VERIFIED by emission: SQL[Database.Connection.MySQL] -> WHERE name = ? AND age > ? SQL[Database.Connection.Postgres] -> WHERE name = $1 AND age > $2 SQL[AppDb] (d AppDb : Database.Connection.MySQL) -> ? (inherited via parent chain) no bracket -> defaults to $n Both spellings parse: SQL[Conn]`...` and SQL[Conn](`...`). Values stay parameters in every dialect. TYPE-SYSTEM finding behind the design: generics attach to the LAST segment, so Database.Query[Conn] parses Database[Conn].Query does NOT ("expected ')', got '.'") The second would be a PATH-DEPENDENT type (Scala's db.Query) -- a much heavier feature. Database.Query[Conn] expresses the same thing with ordinary generics, and a wrong type argument is a genuinely distinct type (Query[Postgres] is rejected where Query[MySQL] is expected), so MySQL-shaped SQL cannot be run against Postgres by accident. OPEN BUG worth fixing before the ORM surface: inheriting a type parameter works through a SIMPLE parent (d Crate[T] : Box[T] is clean) but NOT through a dotted one -- d Database.Rows[Conn] : Database.Query[Conn] fails with "'Conn' is not defined". Every derived query type would otherwise have to hardcode its connection. find/get/insert: nothing to remove -- `o Database { get, find }` parses but binds NOTHING (find(...) is "not defined"), so the sugar ORM was never reachable. ## Config[T]`path` — typed config binding, and the build/runtime layer split THE MECHANISM (Greg's design, made concrete): config/app.json checked in -> the SHAPE ({"db":{"main":{"engine":"mysql"}}}) .local/app.json gitignored -> the VALUES (host, user, password) local.sample/ committed -> the manifest a developer copies from 1. LAYER SPLIT (correctness-critical): load_build_config reads framework -> plugins -> config/app.json and STOPS. load_runtime_config adds .local. A build must not read .local: it is gitignored, so a type inferred from it would differ between a laptop and CI, and compiling against secrets risks baking them in. Greg's point stands separately and is right: a MISSING .local should complain -- but that is a COMPLETENESS check against a manifest, not a type input. Different jobs; only the second sees secrets. 2. Config registered as a template tag. Config[Database.Connection.MySQL]`db.main`: - compile time: the path must exist in the checked-in layers (missing = ERROR, never a silent empty -- an empty config makes every later lookup succeed) - cross-checks the config's own `engine` string against the declared type - produces the BRACKET TYPE, so the dialect is known statically THE TYPE IS THE SOURCE OF TRUTH, THE JSON STRING IS A CROSS-CHECK. Letting the string pick the type would mean editing JSON silently changes which SQL is emitted, with no diff in the source -- action at a distance. 3. A tag bracket naming a VARIABLE now resolves to that variable's declared type at lint time, so SQL[main] works and nothing downstream needs a special case. VERIFIED: MySQL-bound name -> `?`, Postgres-bound name -> `$1`. BUG en route: self.lookup() returns a DICT {'type':..,'const':..,'line':..}, not an object -- my getattr chain fell through silently and the resolution did nothing. Caught by the emitted SQL not changing, not by a failing assert. Release guard broke a third time by design (Config); updated. codegen 629 -> 630. NOT DONE: `u2c config check` + generated local.sample/app.json (required_keys_for works; nothing calls it); runtime registry handing a populated connection to u_pg_connect/u_my_connect. ## Rename, lockfile, config check — and a misdiagnosis corrected RENAME: DbConnection -> Database.Connection (20 in the spec, 7 each in matrix and plan, 4 in code comments). Database.Row / Database.Query already used that namespace in generated code; PHP's Db_Row flattening exists because PHP lacked namespaces, and U has them. MISDIAGNOSIS CORRECTED: I claimed "dotted subclasses inherit nothing". Wrong. d Circle : Shape inherits ✓ d A.Base.Sub : A.Base inherits ✓ d X : Y (NO body) is a type ALIAS -- empty field map is CORRECT d Crate[T] : Box[T] genuinely broken (generic parent only) What actually broke `config check` was my own code reading class_fields[declared] without resolving the ALIAS first, so the manifest came back empty and the .local check reported success on an incomplete config -- a green result meaning "I could not see anything", which is worse than a red one. FIXED (real bug): a GENERIC parent was never attached. The parser did not consume the parent's [T], so the type args fell out as loose ExprStmts and the class inherited nothing -- SILENTLY, because member access is permissive on a class with no recorded fields, so reading an inherited field still passed. Now the parent's type arguments are consumed and recorded (parent_type_args), the parent attaches, and fields merge. Test asserts the INHERITANCE, not that a read passes. Scope: type-argument SUBSTITUTION is not applied -- an inherited `item: T` stays T rather than resolving to the instantiated argument. LOCKFILE (u.lock.json, committed): resolve_bindings / read / write / lockfile_drift. A config change that alters emitted SQL now shows as a diff, with the consequence named. This is what makes config-driven engine selection safe rather than action at a distance -- same principle as committing generated Base classes. u2c config check: resolves bindings against checked-in layers, reports lockfile drift, and lists what .local still owes (fields without defaults). --write-lock records; --strict errors at deploy while a build only warns, so CI can typecheck a repo with no secrets. Verified end to end on a real project. codegen 630 -> 631. NOT DONE: runtime registry handing a populated connection to u_pg_connect / u_my_connect; generated local.sample/app.json; type-argument substitution in inherited field types. ## Database layer at parity with Qbix — pass one Read the uploaded Qbix Db (14,972 lines). The piece worth moving to compile time is Db_Relation::compile(): it finds the ROOT table (the unique foreign_table no edge points out of), throws on >1 root, throws on 0 (a cycle), and BFS-levels the graph -- and the LEVELS ARE THE JOIN ORDER ("the order they should appear in JOIN statements, for the statements to make sense"). Join order is DERIVED FROM THE KEYS, never hand-written. Every input is static, so U does it at build time. IMPLEMENTED u2c/relation.py: root_store / levels / join_order / validate / sql_joins. Qbix's two runtime exceptions are now build errors that NAME the tables ("would produce a cross product", "no table to start the joins from"), plus one Qbix cannot detect at all: the same pair joined by two different keys is ambiguous. Verified users<-posts<-comments levels to [[posts],[comments]] and emits joins in dependency order. IMPLEMENTED stdlib/db.u (parses): Database.Connection(+MySQL/Postgres/SQLite), Database.Range (incl. the prefix trick), Database.Edge/Relation with z f rootStore/levels/validate, Database.Row (engine-AGNOSTIC and non-generic, as Qbix keeps it), Database.Query, Database.Shard. IMPLEMENTED schema from SQL DDL (Greg's idea): parse_sql_ddl reads CREATE TABLE instead of scanning a database. Better build input -- versioned, no server needed so CI can generate without credentials, the same artifact a migration applies, and FOREIGN KEYs give the relation graph FOR FREE. BUG caught writing it: matching the table body with a non-greedy \\(.*?\\) stops at the first ')' -- the one inside varchar(64) -- silently truncating the column list. Body is now extracted by BALANCING parens; the test asserts the LAST column survived. CORRECTION: I had claimed type-argument substitution was missing. It is not -- Box[S].item resolves to S and returning it as I correctly fails, and an INHERITED generic field substitutes too. class_fields stores the DECLARATION form (T); infer_type substitutes at the USE site, which is the right place. Query[T] deliberately deferred to pass two, per Greg. codegen 631 -> 633. NOT DONE: relation checks are Python-callable but not yet z f hooks reachable from U; Database.Query/Row not registered as linter types; index-coverage and shard fan-out checks not wired into query compilation. ## Db parity pass one: relations, DDL, Database.* types Read the uploaded Qbix Db (14,972 lines). The centrepiece is Db_Relation::compile(): it does NOT ask for a join order, it DERIVES one -- find the root table (referenced but never referencing), BFS-level outward, and THE LEVELS ARE THE JOIN ORDER. Two structural mistakes are errors: >1 root (cross product) and 0 roots (cycle). Every input is static, so U does it at COMPILE time. IMPLEMENTED: - u2c/relation.py: root_store / levels / join_order / sql_joins / validate. Errors name the offending stores. validate() also reports non-fatal ambiguity (same pair joined by two different keys; self-reference). Qbix recomputes per query and raises at runtime; here it is computed once. - u2c/schema.py parse_sql_ddl(): schema from checked-in DDL. FOREIGN KEYs give the relation graph for free. Body split on commas at PAREN DEPTH ZERO, not by regex -- a naive split breaks on decimal(10,2) and KEY x (a,b) and silently truncates the column list; the test asserts the LAST column survives. Multi-column index ORDER preserved (leftmost-prefix depends on it). - Database.Row / Query / Range / Relation / Connection / Expression / Row registered as linter types. ENGINE-AGNOSTIC, exactly as Db_Row is. VERIFIED: a class generated by schema pull now lints clean, so schema -> generated class -> linter connects end to end. GOTCHA: a list or map field on a BUILTIN class gets walked by every structural pass -- `indexes: [S]` on Database.Row and `columns: {S:S}` on Row each broke a Tree-mapping test. Both are declared by the generated subclass instead. NOTE: two tests for this work were already in the suite from the interrupted run, which fixed the API (root_store/sql_joins on dict edges) -- I rewrote relation.py to match rather than inventing a second shape. codegen 631 -> 633. NOT DONE (pass two): Query[T] parametrization; type-argument substitution; compile-time index-coverage errors wired to the generated `indexes`; hasOne/ hasMany declarations; Range type-vs-column checking; runtime connection registry. ## Verified on three real engines The joins DERIVED from foreign keys (never hand-written) run correctly on SQLite, PostgreSQL 16 and MariaDB 10.11 -- identical rows on all three, including the LEFT JOIN NULL for a parent with no grandchildren: from posts->users and comments->posts: root = users order = [posts, comments] LEFT JOIN posts ON posts.userId = users.id LEFT JOIN comments ON comments.postId = posts.id all three: [('greg','hello','nice'), ('greg','hello','ok'), ('ada','world',None)] USEFUL FINDING: the DDL is dialect-specific (SQLite rejects MySQL's inline `KEY name (cols)`; Postgres wants it separate) but the RELATION GRAPH derived from it is not, because FOREIGN KEY ... REFERENCES is spelled the same everywhere. That is exactly the portability claim, demonstrated instead of asserted. tests/db_engines_driver.py + a suite test that starts the servers itself and skips cleanly when one is unavailable. io 15 -> 16. ## qbix.html — side-by-side comparison, and an honest gap Wrote qbix.html (228 lines), nav-wired into every page, 0 broken links, structurally valid. ANSWERING "do we still need .get()?": yes, but NOT for convenience. If it all becomes SQL anyway, a structured API looks like ceremony -- it isn't, because STRUCTURE IS THE ONLY THING A COMPILER CAN REASON ABOUT. A SQL string is opaque: you cannot ask it which index it needs, which shard it belongs to, or whether its joins are ordered right. A structured query you can. So the structured path is the CHECKABLE one and SQL`...` is the EXPRESSIVE escape hatch -- and unlike every other system that lands here, U's escape hatch is ALSO parameterised, so taking it costs expressiveness but not safety. VERIFIED THE HONEST GAP before writing the page: Database.Query.where()/.select() -> the type exists, NO methods Database.Row.save() -> NO methods reachable from U find() via o Database { find } -> parses, binds NOTHING u_query_* runtime builder -> exists in C, unreachable from U SQL`...` template -> the ONLY working query path So U has NO general-purpose query builder reachable from U source. There is no equivalent of Db::connect()->select()->where()->fetchDbRows(). The page says so plainly rather than letting the comparison tables flatter it. The comparison covers: connection/dialect, model classes, relations and joins (where U differs most), indexes and sharding, safety, and what is missing. One-line difference: Qbix computes relations, index checks and dialect choices PER QUERY AT RUNTIME and reports mistakes as exceptions or after-the-fact signals; U computes them ONCE AT BUILD TIME and reports mistakes as build errors. Everything else is a consequence. ## The structured query builder (u2c/query.py, ~250 lines) SCOPED BY CHECKABILITY, not SQL coverage. The builder and the SQL template lower to the SAME artifact (statement + params) and neither is less safe -- they are PEERS, not layers. What differs is the INPUT representation and therefore what the compiler can see. So an operation belongs only if structure buys a check: IN: select / where / orderBy / limit (column existence, index coverage) AUTO: joins -- derived from the FK graph, not a method OUT: aggregates, GROUP BY, CTEs, windows -- structure buys nothing there That boundary is why it is ~250 lines and Qbix's Db_Query is 3,719: sharding, caching, chunking and transactions stay runtime concerns. CHECKS THAT BITE (all before any query is sent): - LEFTMOST PREFIX index coverage: KEY (userId,id) serves a filter on userId and does nothing for one on title -> "no index covers that prefix - this will scan the whole table". Qbix's signalMissingIndex only fires AFTER the query. - unknown column in where/select/orderBy; unsupported operator - INSERT missing a NOT NULL column with no default - IN () becomes `1 = 0`, not a syntax error -- dropping the clause would WIDEN the query rather than narrow it - every method returns a NEW query, so the predicate accumulates as a VALUE, which is what makes it checkable at all VERIFIED BY EXECUTION on SQLite, PostgreSQL 16 and MariaDB 10.11: select/where/ orderBy/IN and get() return identical rows. TEST-ISOLATION BUG found: Postgres returned empty because the `posts` table from the RELATION test still existed with a dependent FK, so DROP failed, CREATE never ran, and the query read stale rows. Fixed with DROP ... CASCADE. Same class of mistake as the earlier non-idempotent Postgres test. PORTABILITY FINDING: Postgres FOLDS unquoted identifiers to lowercase -- `userId` is stored as `userid`. It works because both sides fold consistently, but camelCase column names are silently lowercased there and are NOT preserved across engines. codegen 633 -> 634, io 16 -> 17. ## Db layer BOUND to U source — parity with Qbix Db_Row / Db_Query Until now the entire Db layer lived in the COMPILER (query.py, relation.py, schema.py, ~40KB) and no U source could reach it: `.where()` and `.save()` both reported "has no method". Now bound and type-checked. BOUND (Qbix parity, scoped by the checkability rule): Database.Query : select where orderBy limit offset join fetchAll fetchRow execute getSQL Database.Row : save remove retrieve retrieveOrSave set get toArray wasRetrieved wasModified wasInserted modifiedFields getPrimaryKey getTable find A GENERATED model inherits the whole Row lifecycle -- that is the integration that makes the layer usable. Everything reaching the wire is +A, so a query suspends on the same fiber scheduler as every other wait. getSQL is exposed on purpose: a Query and a SQL template are PEERS producing the same statement+params, so you can always see what you built. SCOPE CREEP REFUSED AND TESTED: groupBy, having, onDuplicateKeyUpdate, replace, lock exist in Db_Query and are deliberately NOT bound -- structure buys the compiler nothing for them, so they belong in `SQL`...``, which is already parameterised. Binding them is how Db_Query reached 3,719 lines. codegen 634 -> 635. STILL NOT DONE: codegen does not lower these calls to the runtime builder, so a compiled program cannot execute them end to end; no connection registry; Query[Model] typing (blocked on type-argument substitution); index_warning is not yet emitted as a compiler diagnostic; update/delete; hasOne/hasMany; Range vs column; sharding. ## CORRECTION: groupBy/having/lock/upsert DO belong on the structured path I cut these by applying the checkability rule too quickly. Greg was right that they are used heavily in Qbix apps AND that they help structural analysis. Looking properly, each carries a real compile-time check: groupBy - column existence PLUS the select-list consistency rule (every selected column grouped or aggregated). Postgres enforces it always, MySQL only under ONLY_FULL_GROUP_BY -- so the same query can pass locally and fail in production. Catching it makes queries portable by construction. having - only meaningful over groups; a plain column there is either an error or a silent scan that where() would have indexed. Both refused. lock - FOR UPDATE vs LOCK IN SHARE MODE, and SQLite has NO row-level locking; reported rather than silently emitted. upsert - THE strongest case. Postgres/SQLite REQUIRE a conflict target that MySQL FORBIDS, which is why hand-written upserts get special-cased per engine. The conflict columns are DERIVED from the unique index, so ONE description emits all three forms correctly. Plus a check no string can carry: an upsert against a table with no unique constraint has an unreachable update branch; one where every supplied column is in the key would change nothing. Both refused. replace - refused on Postgres WITH THE REASON (it DELETES the old row, firing triggers and clearing unsupplied columns). VERIFIED BY EXECUTION: insert-then-conflict gives one updated row on SQLite, PostgreSQL 16 and MariaDB 10.11. Also fixed: select() was rejecting COUNT(*) as an unknown column -- aggregates are not columns and must not be checked as such. The line now sits at arbitrary SQL (window functions, CTEs), which the template already carries safely. codegen 635 -> 636, io 17 -> 18. ## getSQL returns S, and row.find removed Greg's reasoning, which is right and worth recording: in JS a template literal interpolates right there; we have functions. Does it make sense to turn strings into template literals AT RUNTIME? No -- that is not a pit of success. Same for the query builder. So getSQL() returns a plain string. The deeper reason: the `SQL` type's guarantee is that it can only be built from a LITERAL -- fixed text, varying slots. A function that minted a SQL value at runtime would hand back one whose text is ARBITRARY, which is precisely the injection vector the type exists to exclude. The guarantee would be worthless. So getSQL is for INSPECTION (logging, EXPLAIN) and executing the query is how you run it. Verified: -> S is clean, -> SQL is refused. Also removed find() from the row INSTANCE -- starting a query is not something an individual record does, and Query(model) already says it. ## HTTP client surface — fluent, protocol-named Bound HTTP / Request / Response / CookieJar so U source can make requests. KEY POINT: nothing new was needed for chaining -- U's +A ALREADY IS the promise, so the scheduler drives the wait and the syntax stays direct. No .then(). HTTP.get/post/put/patch/delete/head(url) -> Response +A (common case) HTTP.request(url).header().bearer().json().timeout().cookies() .followRedirects().send().json() (everything else) Response: ok / text / json (-> Tree) / bytes / header / headers CookieJar: set / header / count / clear (backed by the tested jar) HTTP.encodeURL / decodeURL / encodeQuery `https` is deliberately NOT a separate class -- it is HTTP with a certificate, so it is a URL scheme, not a different API. GOTCHA: registering a `URL` CLASS shadowed the existing URL template TAG and broke two tag tests. URL helpers moved onto HTTP instead -- one URL concept, not two. codegen 636 -> 637. NOT DONE: codegen does not lower these to u_http.h; the client still lacks redirect-following, keep-alive and automatic gzip decoding. ## THE DB LAYER NOW EXECUTES — u_db.h The missing link between "the compiler builds a statement" and "a program runs a query". Everything above it existed (USqlStmt with params apart; pg/mysql wire clients); nothing turned a config NAME into a live connection and pushed a statement through. - registry BY NAME, not by object: `main` resolves to a DIALECT at compile time (which is how placeholders were chosen) and to a SOCKET at run time. Keeping those apart is what lets a build happen on a machine with no credentials. - connections open LAZILY -- a program that never queries never opens a socket - u_db_exec(name, USqlStmt, UDbResult) -> engine-independent rows VERIFIED BY EXECUTION vs live PostgreSQL 16 + MariaDB 10.11: statements built exactly as SQL[main]`...` lowers them are prepared and executed; an unknown connection name is reported, not guessed; and THE DECISIVE CASE -- the payload "x'); DROP TABLE runq; --" is stored as a VALUE and the table survives with both rows, on BOTH engines. Postgres uses the real extended protocol (Parse/Bind/Execute). MySQL substitutes at the DRIVER with proper quoting, because its prepared statements use a binary result protocol that is not implemented -- the statement still travels as text plus a separate parameter list, and only that last hop merges them. io 18 -> 19. STILL NOT DONE: codegen does not yet lower Database.Query/.Row METHOD CALLS to u_db_exec -- the SQL template path executes, the fluent builder does not. And nothing emits u_db_register() from resolved config automatically. ## HPACK, HTTP/2 framing, FTP u_hpack.h (RFC 7541) -- the hard part of HTTP/2. Headers are INDICES into a table both peers maintain identically; a single eviction disagreement desynchronises the connection PERMANENTLY and SILENTLY (HTTP/2 has no resynchronisation point). So it is verified against THE RFC'S OWN VECTORS: C.1 1337 in a 5-bit prefix -> 1f9a0a C.2.1 literal-with-indexing byte-for-byte C.3.1 decodes a real request 8286 8441 0f77... -> :method GET, :scheme http, :path /, :authority www.example.com with the dynamic table ending at exactly 57 bytes Static table (61), dynamic table with FIFO eviction and the RFC's size rule (name+value+32), prefix integers, length-prefixed strings, all four field representations, size updates. Malformed input REFUSED not guessed: truncated continuations, overlong integers that would wrap, out-of-range indices. SCOPE: Huffman strings are DETECTED and reported (-2), not decoded. The H bit is optional to emit so this interoperates, but a Huffman-coding peer is unreadable. u_http2.h (RFC 7540) -- 9-byte frame header, the exact 24-byte preface (deliberately invalid HTTP/1.1), SETTINGS pairs. THE RESERVED HIGH BIT OF THE STREAM ID IS MASKED, not trusted -- a classic interop bug, and the test sets it to prove it is ignored. SCOPE: framing + HPACK only. No stream state machine, no flow control, no push, nothing drives it over TLS+ALPN h2 yet. u_ftp.h (RFC 959) -- control + passive data channels. MULTILINE REPLIES ARE CONSUMED WHOLE: `220-` means more lines follow, and reading only the first is the classic bug that makes every later reply arrive off by one. TYPE I on connect (ASCII mangles binary). PASV port is p1*256+p2. VERIFIED against a real pyftpdlib server: login through a multiline banner, passive LIST, RETR exact bytes, STOR round-trip byte-identical, MKD/CWD/DELE, and a bad password refused. NON-IDEMPOTENT TEST caught again: the driver creates sub/ and uploaded.txt, so a second run saw 129 bytes instead of 69. Fixture now resets. Third time this class of bug has appeared (Postgres tables twice, now FTP files). io 19 -> 22. NOT DONE: HTTP/2 client/server on a real connection; FTP server; Huffman decode. ## webview.html — the native shells, written up before building 281 lines, nav-wired into every page, 0 broken links, structurally valid. Marked STATUS: DESIGNED, NOT BUILT at the top -- no shell exists yet; the page is the plan, written down so the platform differences are known rather than discovered. KEY POINTS: - TWO ENGINES, not five platforms. WebKit powers WKWebView AND WebKitGTK; Chromium powers WebView2 AND Android. Both modern, both run the Qbix JS runtime unchanged. - THREE BRIDGE VARIANTS, not five -- the WebKit family shares an identical JS call (window.webkit.messageHandlers.u.postMessage). A GENERATED shim normalises all of them to U.post(). Messages are JSON both ways, because Android's addJavascriptInterface only passes primitives and strings reliably. - eval() takes a JS value, not a string, so anything crossing into the page is already encoded and the and U+2028 hazards are handled before it leaves U. - EVERY host needs UI-THREAD MARSHALLING (WKWebView main thread; Android UI thread incl. evaluateJavascript; WebView2 COM apartment + pump; GTK main loop). So the scheduler needs a post-to-UI queue alongside the epoll reactor. This is THE ONE PLACE the concurrency model has to grow to meet the GUI. - ASSETS: file:// is simplest and WRONG -- its origin restrictions break fetch and XHR, which is exactly what a Qbix front end uses. Custom scheme handler instead; all four hosts support one, giving a real origin so relative URLs, fetch and cookies behave as on a server. - QBIX COMPATIBILITY: the front end does not change. Q.activate, Q.Tool.define, Q.Template.render, delegated events, Q.Tool.clear all unchanged -- and U's on* prohibition ENFORCES the delegated-event convention. THE INTERESTING PART: Q.req can be routed to the BRIDGE instead of HTTP, so the request never leaves the process, U serves it directly, and the tool's code is untouched. Same front end, no network, no local server. BUILD ORDER: Linux (plain C, proves the bridge + UI queue) -> macOS (Obj-C shim) -> iOS (reuses the shim verbatim) -> Windows (COM) -> Android LAST, because it is the only target where the shell is a subproject rather than a shim (JNI + Java UI thread). ## Playground: real compiler in the browser, translators, sharing, responsive CORRECTION FIRST: I twice said things did not exist that were sitting in outputs (JS/Python->U; FTP and HTTP/2 headers). Checked this time before building, and found js-to-u.html (49KB, acorn+CodeMirror, JS/TS/Python->U), a u2c wheel for Pyodide, u2c-web.html and u2c-run.html. Reused rather than rebuilt. Committed the lesson to memory. BUILT: - site/playground.html + assets/playground.js: source pane (U / JavaScript / TypeScript / Python) -> U pane -> emitted C pane. Real u2c under Pyodide, loaded LAZILY on first Compile (~10MB; most visitors read before compiling, so paying it on page load would make the page feel broken for everyone). - assets/translate.js: the translators, extracted from the prototype. - u2c/browser.py: one JSON entry point (compile_source / compile_json). Never raises -- a compiler crash is reported as a diagnostic, because a blank screen is the worst answer for someone learning the language. Line numbers are pulled out of diagnostics so the UI can point at a line. - Sharing: source travels in the URL fragment (base64url, language prefixed), navigator.share where available, clipboard fallback, and the fragment is restored on load so a shared link opens exactly what was sent. - Responsive: 3 columns -> single column with Source/U/C tabs under 1000px, tighter type under 560px, viewport-fit=cover for notched phones. THREE REAL BUGS CAUGHT BEFORE SHIPPING (all by checking, not by assuming): 1. The shipped wheel was STALE -- no browser.py, and none of this session's work (query/relation/reflect). The playground would have failed on import. Rebuilt from source; verified by installing into a clean venv and running the exact call the page makes, for both the success and error paths. 2. translate.js threw at load without CodeMirror, which the new page does not use. Guarded, so it now loads headless and in an editor-less page. 3. convertJS/convertPy return {code, messages}, NOT a string -- the pane would have shown "[object Object]". Fixed, and the messages are surfaced as translation notes. VERIFIED: wheel installs clean and compiles U->C (34,856 bytes) with correct errors; both translators emit correct U headless in node; both JS files parse; 0 broken links; structure valid. NOT DONE: running the compiled program in the browser still needs the C->WASM toolchain (~50MB clang+sysroot) -- the page compiles to C, it does not execute. ## -M propagation — transitive const through parameter references DESIGN DECISION: class fields are +M by default (the owner mutates them); local variables and parameters are -M by default (borrowers read only). -M on a binding PROPAGATES INWARD — nothing reachable through it can be modified, even fields declared +M in their class. The class's own methods (t/self) are exempt — they are the owner. This is C++ transitive const applied uniformly through U's modifier algebra. Four mutation points checked: member assignment, index assignment, list mutator methods (.push/.pop/etc), and << patch. All reject through a -M reference. Adding +M to the parameter allows mutation. IMPLEMENTED: - linter.py: _is_ref_immutable() — recursive walk from mutation point to root binding; t exempt by name. - Wired into check_Assignment (member+index), check_Call (list mutators), check_Patch (<< operator). - 9 new linter tests: param field write rejected/ok, param list push rejected/ok, param index write rejected/ok, param nested field rejected, self field write ok, self list push ok. linter 346 -> 355. DOCUMENTED: - u_language.html: §03 (variables), §05 (modifier defaults), §22 (defaults table) — position-dependent rule + propagation paragraph. - U-primer.md: ±M row updated, propagation section added. - implementation.html: design rationale section added. - FEATURE_MATRIX.html: new Works row (127 Works total). - plan.md: this entry. ## Database.Query codegen — builder chain lowered to u_dbq_* C runtime IMPLEMENTED: - u_db.h: UDbQuery struct + u_dbq_new, u_dbq_select, u_dbq_where, u_dbq_order_by, u_dbq_limit, u_dbq_offset, u_dbq_group_by, u_dbq_having, u_dbq_lock, u_dbq_build (assembles dialect-aware SQL), u_dbq_get_sql, u_dbq_execute, u_dbq_fetch_all, u_dbq_fetch_row, u_dbq_free. - exprs.py: intercept Database.Query method calls by receiver type, emit u_dbq_* calls. Database.Query({store: "t"}) → u_dbq_new("main","t"). Database.Row methods emit u_dbrow_* calls. - 1 new codegen test: verifies transpiled C contains u_dbq_new, u_dbq_select, u_dbq_where, u_dbq_order_by, u_dbq_limit, u_dbq_get_sql. - codegen 635/638 → 636/639. ## HTTP client codegen — verbs + builder chain lowered to u_httpc_* IMPLEMENTED: - u_http.h: URL parser (UHttpUrl), high-level client API (u_httpc_get/post/put/patch/delete/head), request builder (UHttpReq + u_httpc_request/req_method/req_header/req_body/ req_bearer/req_timeout/req_send), response accessors (u_httpc_resp_ok/status/text/header), URL encode/decode. - exprs.py: intercept HTTP.verb() calls BEFORE the generic static method handler (which would emit a mangled name), emit u_httpc_* calls. Request builder chain and Response accessors handled by receiver type. Response.status field access → u_httpc_resp_status. - 1 new codegen test (4 sub-assertions): HTTP.get → u_httpc_get, HTTP.post → u_httpc_post, request builder chain → u_httpc_request /req_method/req_header/req_bearer/req_send, Response.status → u_httpc_resp_status. - codegen 636/639 (same 3 pre-existing failures). ## Fixed the 3 pre-existing codegen failures — now 639/639 1. 1-based list indexing write test: the test function took [I] as a parameter (default -M) and tried to write arr[1]=99 — correctly rejected by -M propagation. Fixed by adding +M to the parameter, which is the right U: you declare mutation intent. 2. +G static field (Log.calls = ...): Log is a class name declared is_const=True (you can't reassign the class), and _is_ref_immutable was treating it as a -M reference, blocking the +G field write. Fixed: class names (type='class:X') are now exempt from propagation — they're namespaces for +G fields, not references. 3. z f __load__() FIRES before main: was caused by the same +G issue — __load__'s body wrote Ct.count, which went through Log-style class access. Same fix resolves both. All 1,124 tests now pass: parser 120, linter 355, codegen 639, optimize 6, WGSL 4+7skip. ## +G static fields default to -M (global constants) DESIGN: +G fields are global namespaced identifiers, not instance fields owned by `t`. They belong in the same category as locals and parameters: -M by default. A bare `PI: N +G = 3.14` is a global constant; mutable global state requires explicit `+G +M`. RULE: +M defaults to things owned by an instance (t). Everything else -M. - Instance field: +M (owner mutates) - +G static field: -M (global constant unless +M) - Local variable: -M - Parameter: -M IMPLEMENTED: - default_const(): +G in mods → True (const). Checked before the non-scalar fallthrough, so +G collections and classes are -M too. - class: prefix stripped in field-level check so class_fields lookup succeeds for class-name member access (Cfg.tags). - 4 new linter tests: +G no +M rejected, +G +M ok, +G scalar no +M rejected, +G constant read ok. Linter 355 → 359. DOCUMENTED: u_language.html §03 (variables), §05 (info-box), §22 (table). ## Element +M hole-punch in -M propagation DESIGN: explicit +M on an element type in a parameter's type annotation BREAKS the -M propagation chain at that level. [I] as param → container -M, elements -M (full freeze) [I +M] as param → container -M (no push), elements +M (arr[1]=x ok) [[I +M]] as param → outer -M, inner -M, but I is +M (arr[1][2]=x ok) The +M must appear in the BINDING's own type — field declarations inside the class body (which are +M by default) do NOT break propagation. IMPLEMENTED: - _element_has_explicit_m(): checks if the base string of a container type (e.g. '[I+M]') contains +M in the inner type. - Index assignment check: after _is_ref_immutable fires, checks for element +M hole-punch before emitting the error. - _is_ref_immutable BinOp[] case: checks element +M at each nesting level during the recursive walk, so nested access chains (arr[i].x) also respect the hole-punch. - 4 new linter tests: element +M write ok, element +M push rejected, no-M write rejected, nested element +M ok. Linter 359 → 363. - Codegen 639/639 (no regressions). ## +G +M is MVCC-managed — << is the only write path DESIGN: +G +M state (global mutable) is MVCC-managed by default. - Direct assignment (ClassName.field = x) is a compile error. - All writes go through << (e.g. Counter << { count: Counter.count + 1 }). - << on a class name patches +G +M static fields. - For scalars: currently emits a plain C store to the global (correct for single-threaded); will become CAS when concurrency runtime lands. - For objects: uses the existing MVCC version-swap machinery. TRANSACTION BLOCKS (<< ( ... )): - Multiple << operations grouped into one atomic commit. - Optimistic: buffer patches, check all versions at commit, retry on conflict. - Nesting is flat — inner << ( ) is subsumed by outer. - Not yet implemented in parser/codegen; design is settled. IMPLEMENTED: - Linter: check_Assignment rejects direct = to +G +M fields with error pointing to <<. check_Patch handles class-name targets, validates only +G +M fields are patched. - Codegen (funcs.py): stmt_Patch detects class-name targets (class:X), emits plain assignment to C global (ClassName_field = val). - Updated +G static field and __load__ codegen tests to use <<. - 1 new linter test: +G +M direct assign rejected. - Linter 363 → 364, codegen 639/639 (zero failures). ## << ( ... ) transaction blocks — parser + linter IMPLEMENTED: - ast_nodes.py: TransactionBlock node (body: list of stmts). - parser.py: << followed by ( triggers transaction parse; collects statements until closing ), handles nesting. - linter.py: check_TransactionBlock enforces -E -D (pure, deterministic) on all calls, bans `a` (async), tracks in_transaction depth for flat nesting. - Fixed truncated check_Return (accidentally cut during editing). - 2 new linter tests: single-patch and multi-patch transaction blocks. - Linter 364 → 366, all other suites unchanged. SPEC: u_language.html — Transaction blocks section added with full restrictions list and code example. Cross-object transactions row updated from "Not in v1" to << ( ) description. IMPLEMENTATION.HTML: Added rationale sections for ±M defaults (why instance +M, why +G -M, why propagation, why class names exempt, why element +M breaks chain), +G +M MVCC enforcement, transaction blocks (why -E -D, why no async, why flat nesting, why eager c, rollback semantics).