# Dataframe — Columnar Data in U

U's Dataframe is not a library — it is a compile-time template tag that builds typed, columnar query plans. The compiler sees the entire pipeline and optimizes it: predicate pushdown, projection pruning, stage fusion, and GPU kernel emission. The same `.on()` / `.filter()` / `.map()` / `.reduce()` chain that works on lists works on Dataframes, with the modifier system controlling execution strategy.

## Construction

Dataframes are built from columns, CSV, JSON, or the backtick template:

```
// From columns:
df = Dataframe.from_columns({
    "name": ["Alice", "Bob", "Carol", "Dave"],
    "age": [30, 25, 35, 28],
    "salary": [70000, 55000, 90000, 62000]
})

// From CSV (async):
df = Dataframe.from_csv("data.csv") +A

// From JSON rows:
df = Dataframe.from_json([
    {"name": "Alice", "age": 30},
    {"name": "Bob", "age": 25}
])

// Backtick template with interpolation:
min_age: I = 25
cols = ["name", "salary"]
df2 = Dataframe`select {{cols}} from {{df}} where age > {{min_age}}`
```

The backtick form produces a `Formats.Dataframe` — a typed query plan, not a string. The compiler's `__validate__` dunder checks column names, types, and predicates at compile time. Interpolated variables (`{{cols}}`, `{{min_age}}`) are type-checked against the schema.

## Schema

Every Dataframe carries a compile-time schema. The compiler knows column names and types:

```
d UserRow
    name: S
    age: I
    salary: I
    active: B = true

df = Dataframe[UserRow].from_csv("users.csv")
// Compiler knows: df has columns name:S, age:I, salary:I, active:B
// df["email"] → compile error: 'email' is not a column in UserRow
```

When no schema type is given, the Dataframe infers column types from the data (first row for CSV, all rows for JSON).

## Pipeline Operations

Dataframes use the same `.on()` primitives as lists. Every operation returns a new Dataframe (immutable by default):

### Selection

```
df.select(["name", "salary"])       // keep only these columns
df.drop(["active"])                  // remove columns
df.head(10)                          // first 10 rows
df.tail(5)                           // last 5 rows
df.slice(100, 200)                   // rows 100–199
df.sample(50)                        // random 50 rows
```

### Filtering

```
df.filter(row => row["age"] > 30)
df.where("salary", ">", 60000)
df.filter(row => row["name"].starts_with("A") & row["active"])
```

### Transforms

```
df.with_column("bonus", row => row["salary"] * 0.1)
df.rename("salary", "annual_pay")
df.sort("age", desc: true)
df.unique("name")
df.cast("age", "N")                  // I → N
```

### Aggregation

```
df.sum("salary")                     // → I
df.mean("age")                       // → N
df.min("salary")                     // → I
df.max("salary")                     // → I
df.count()                           // → I

// GroupBy:
df.groupby(["active"])
    .agg("salary", "mean")           // mean salary per group
    .agg("age", "max")               // max age per group
    .collect()
```

### Joins

```
orders = Dataframe.from_csv("orders.csv")
users = Dataframe.from_csv("users.csv")

joined = users.join(orders, on: "user_id", how: "inner")
cross = users.cross_join(products)
```

### Column Expressions

Direct column access returns a `Column` — a typed array with element-wise operations:

```
col = df["salary"]                    // Column[I]
col.sum()                             // → I
col.mean()                            // → N
col * 1.1                             // → Column[N], element-wise
col > 60000                           // → Column[B], boolean mask

// Column arithmetic:
df.with_column("tax", df["salary"] * 0.22)
df.with_column("net", df["salary"] - df["salary"] * 0.22)
```

## Modifiers

The modifier system controls how the pipeline executes:

### `+W` — Lazy Evaluation

```
// Nothing executes until .collect():
result = df
    .filter(row => row["age"] > 30)
    .select(["name", "salary"])
    .sort("salary", desc: true)
    .head(10)
    +W                                // lazy: build plan only

// Execute the plan:
result.collect()

// The compiler applies:
// 1. Predicate pushdown: filter before sort
// 2. Projection pruning: only read name, salary, age columns
// 3. Limit pushdown: stop after 10 rows
```

### `+V` — Vectorized / GPU

```
// CPU vectorized (SIMD):
total = df["salary"].sum() +V

// GPU compute shader:
df.with_column("norm", df["value"] / df["value"].max()) +V
// → compiler emits WGSL:
//   @compute @workgroup_size(64)
//   fn main(@builtin(global_invocation_id) id: vec3<u32>) {
//       outp[id.x] = inp[id.x] / max_val;
//   }

// Fused GPU pipeline:
result = df
    .filter(row => row["price"] > 100)
    .with_column("margin", row => row["price"] * 0.15)
    .sum("margin")
    +V
// → one GPU kernel: filter + multiply + reduce
```

### `+R(GPU)` — GPU-Resident Data

```
// Load data to GPU once:
gpu_df = df +R(GPU)

// All operations stay on GPU — no CPU↔GPU transfer:
filtered = gpu_df.filter(row => row["value"] > threshold) +V
transformed = filtered.with_column("scaled", col => col * factor) +V
total = transformed.sum("scaled") +V

// One transfer at the end:
result = total.to_cpu()
```

## Printing

Dataframes print as aligned tables:

```
print(df.head(3))
// ┌──────┬─────┬────────┐
// │ name │ age │ salary │
// ├──────┼─────┼────────┤
// │ Alice│  30 │  70000 │
// │ Bob  │  25 │  55000 │
// │ Carol│  35 │  90000 │
// └──────┴─────┴────────┘
```

The `__string__` dunder on Dataframe produces the table format. `__pack__` produces a Tree (JSON-serializable). `Dataframe.from_tree(tree)` round-trips.

## Tensor Integration

Dataframe columns can be extracted as Tensors for numerical computation:

```
// Column → Tensor:
ages = df["age"].to_tensor()          // Tensor[I], shape [n]
salaries = df["salary"].to_tensor()   // Tensor[I], shape [n]

// Tensor operations:
correlation = ages.dot(salaries) / (ages.norm() * salaries.norm())

// Tensor → Column:
df.with_column("predicted", Tensor.matmul(weights, features).to_column())
```

## Compile-Time Validation

The `__validate__` dunder runs at compile time on `Dataframe` backtick templates:

```
Dataframe`select name, agee from {{users}}`
//                       ^^^^
// Compile error: column 'agee' not in schema UserRow
//   Did you mean: 'age'?

Dataframe`select name from {{users}} where salary > "high"`
//                                                  ^^^^^^
// Compile error: cannot compare salary (I) with "high" (S)
```

This is the same mechanism that makes `SQL` `` `` catch injection and `HTML` `` `` catch XSS — the template tag's `__validate__` sees the full expression and the schema at compile time.

## Implementation Notes

**Storage:** Columns are contiguous typed arrays (`[I]`, `[N]`, `[S]`, `[B]`) with 64-byte aligned allocation for SIMD. The Dataframe struct holds a `{S: Column}` map plus schema metadata.

**Memory:** Dataframes are `+R` (heap, refcounted) by default. Column data is shared between Dataframes that select subsets (zero-copy slicing via offset + length).

**Serialization:** `__pack__` produces `{"columns": {"name": [...], "age": [...]}, "schema": {...}}`. CSV and JSON I/O use streaming parsers that allocate columns incrementally.

**Pipeline fusion:** When the compiler sees `df.filter(p).map(f).reduce(r)`, it generates a single fused loop instead of three passes. With `+V`, it generates a single GPU kernel. With `+W`, it builds a pull-based iterator that processes one chunk at a time.

**GPU dispatch:** Column operations with `+V` emit WGSL compute shaders via the same `codegen/wgsl.py` emitter used for `[I].map()`. The `+R(GPU)` modifier keeps columns in WebGPU storage buffers between pipeline stages, avoiding CPU↔GPU round-trips.