Worked example

A quant pipeline, start to finish

examples/quant_demo/quant_demo.ibex takes a raw tick file and ends with a fitted model and three CSVs. It exercises the parts of Ibex that only matter together: time-bucketed resampling, an asof join, per-group rolling state, and a regression.

Run it Resample Asof join Model Full script

Shape of the job

Five stages, one script

The pipeline, each table expression uses the previous one.

ticks.csv  ─┐
            ├─► resample 1m by symbol  (OHLC bars + size aggregates)
            ├─► asof-join reference    (sector / vol_regime per symbol)
            ├─► feature engineering    (spread, range, log-mid)
            ├─► fit ridge regression   (close ~ open + range + vol_regime)
            └─► save coefficients + fitted predictions to CSV

The same transform block (resample + asof + features) can reused in a streaming variant. Only the source and sink change, the feature engineering is identical, this removes the train/serve skew that separate batch and live pipelines accumulate.

Input

Two CSVs: a tick stream and a reference table

gen_data.py writes a synthetic session: six symbols, one trading day, roughly 100k ticks. Timestamps are nanoseconds from midnight.

ticks.csv

Raw trades

One row per print: timestamp, symbol, price, size.

ts,symbol,price,size
34200635196792,AAPL,179.6452,452
34200731699373,META,510.4185,127

reference.csv

Per-symbol static data

Sector and an assumed-stationary volatility regime. One row per symbol, stamped at the session open so the asof join has something to match backwards to.

ts,symbol,sector,vol_regime
34200000000000,AAPL,tech,0.001200
34200000000000,MSFT,tech,0.001100
34200000000000,GOOG,tech,0.001400
34200000000000,AMZN,consumer,0.001700
34200000000000,META,tech,0.001900
34200000000000,NVDA,semis,0.002800

Both are promoted to TimeFrames before the join, which is what makes the asof join type-checked. Discpline and sticking to a convention is nice, being sure that bugs are caught is better.

import "csv";

let ticks         = read_csv("examples/quant_demo/data/ticks.csv");
let reference_raw = read_csv("examples/quant_demo/data/reference.csv");

// Promote both to TimeFrames so the asof join is type-checked.
let tick_stream = as_timeframe(ticks, "ts");
let reference   = as_timeframe(reference_raw, "ts");

Stage 1

Resample ticks into 1-minute OHLC bars

resample 1m buckets on the time index; by symbol keeps each instrument's bars separate. The same expression drives the streaming variant, where resample triggers the TimeBucket stream kind that emits one row per closed minute.

let bars = tick_stream[resample 1m, by symbol, select {
    open     = first(price),
    high     = max(price),
    low      = min(price),
    close    = last(price),
    bar_size = sum(size)
}];
bars;

Output

rows: 2340
+-------------------------------+--------+----------+----------+----------+----------+----------+
| ts                            | symbol | open     | high     | low      | close    | bar_size |
+-------------------------------+--------+----------+----------+----------+----------+----------+
| 1970-01-01 09:30:00.000000000 | "AAPL" | 179.6452 | 179.992  | 178.5976 | 178.5976 | 5639     |
| 1970-01-01 09:30:00.000000000 | "META" | 510.4185 | 530.6324 | 510.231  | 526.518  | 6735     |
| 1970-01-01 09:30:00.000000000 | "GOOG" | 175.5003 | 176.5177 | 175.0689 | 175.7573 | 7571     |
| 1970-01-01 09:30:00.000000000 | "AMZN" | 184.7709 | 186.346  | 183.0771 | 183.2741 | 7768     |
| 1970-01-01 09:30:00.000000000 | "MSFT" | 414.808  | 414.9505 | 410.2321 | 411.8708 | 7985     |
| 1970-01-01 09:30:00.000000000 | "NVDA" | 1099.453 | 1139.953 | 1094.948 | 1125.336 | 6716     |
| 1970-01-01 09:31:00.000000000 | "META" | 526.1279 | 530.1362 | 524.7472 | 524.7472 | 5620     |
| 1970-01-01 09:31:00.000000000 | "GOOG" | 175.7237 | 175.9283 | 174.0812 | 175.4403 | 9502     |
| 1970-01-01 09:31:00.000000000 | "MSFT" | 412.0055 | 412.5384 | 409.0638 | 411.1072 | 6221     |
| 1970-01-01 09:31:00.000000000 | "AAPL" | 178.4962 | 178.4962 | 175.8907 | 176.7439 | 6639     |
+-------------------------------+--------+----------+----------+----------+----------+----------+
... (2330 more rows)

2340 bars = 6 symbols × 390 minutes, a full 09:30–16:00 session with no gaps. Bars come out time-major (all six symbols at 09:30, then all six at 09:31), which matters later — see stage 3.

Stage 2

Asof-join the reference table

An asof join matches each bar to the most recent reference row at or before its timestamp, keyed on symbol. Because both sides are TimeFrames, the time column is part of the type and the join does not need to be told which column to order on.

let enriched = bars asof join reference on {ts, symbol};
enriched;

Output

rows: 2340
+-------------------------------+--------+----------+----------+----------+----------+----------+------------+------------+
| ts                            | symbol | open     | high     | low      | close    | bar_size | sector     | vol_regime |
+-------------------------------+--------+----------+----------+----------+----------+----------+------------+------------+
| 1970-01-01 09:30:00.000000000 | "AAPL" | 179.6452 | 179.992  | 178.5976 | 178.5976 | 5639     | "tech"     | 0.0012     |
| 1970-01-01 09:30:00.000000000 | "META" | 510.4185 | 530.6324 | 510.231  | 526.518  | 6735     | "tech"     | 0.0019     |
| 1970-01-01 09:30:00.000000000 | "GOOG" | 175.5003 | 176.5177 | 175.0689 | 175.7573 | 7571     | "tech"     | 0.0014     |
| 1970-01-01 09:30:00.000000000 | "AMZN" | 184.7709 | 186.346  | 183.0771 | 183.2741 | 7768     | "consumer" | 0.0017     |
| 1970-01-01 09:30:00.000000000 | "MSFT" | 414.808  | 414.9505 | 410.2321 | 411.8708 | 7985     | "tech"     | 0.0011     |
| 1970-01-01 09:30:00.000000000 | "NVDA" | 1099.453 | 1139.953 | 1094.948 | 1125.336 | 6716     | "semis"    | 0.0028     |
| 1970-01-01 09:31:00.000000000 | "META" | 526.1279 | 530.1362 | 524.7472 | 524.7472 | 5620     | "tech"     | 0.0019     |
| 1970-01-01 09:31:00.000000000 | "GOOG" | 175.7237 | 175.9283 | 174.0812 | 175.4403 | 9502     | "tech"     | 0.0014     |
| 1970-01-01 09:31:00.000000000 | "MSFT" | 412.0055 | 412.5384 | 409.0638 | 411.1072 | 6221     | "tech"     | 0.0011     |
| 1970-01-01 09:31:00.000000000 | "AAPL" | 178.4962 | 178.4962 | 175.8907 | 176.7439 | 6639     | "tech"     | 0.0012     |
+-------------------------------+--------+----------+----------+----------+----------+----------+------------+------------+
... (2330 more rows)

Row count is unchanged at 2340 — an asof join enriches, it does not fan out. Two columns appear on the right: sector and vol_regime.

Stage 3

Rolling state, then bar-level features

Two update blocks. The first carries a windowed rolling mean partitioned by symbol; the second builds ordinary row-wise features on top of it. Splitting them is what lets the rolling buffer be per-group while the arithmetic stays vectorised.

// Per-symbol rolling features (5-bar window). `update + by symbol + window`
// partitions the rolling buffer per group, so AAPL's mean never includes
// NVDA bars.
let with_rolling = enriched[window 5m, by symbol, update {
    rmean5 = rolling_mean(close)
}];

// Bar-level features built on top of the rolling state.
let features = with_rolling[update {
    spread     = high - low,
    range      = (high - low) / open,
    log_mid    = log((high + low) / 2.0),
    z5         = (close - rmean5) / (rmean5 + 0.0001)
}];

// The label: next bar's close. `by symbol` is load-bearing — see below.
let features = features[by symbol, update {
    next_close = lead(close, 1)
}];

Neither statement is printed, so there is no stdout here. The resulting schema is visible in features.csv, written at the end:

features.csv — header and first two rows

ts,symbol,open,high,low,close,bar_size,sector,vol_regime,rmean5,spread,range,log_mid,z5,next_close
34200000000000,AAPL,179.6452,179.992,178.5976,178.5976,5639,tech,0.0012,178.5976,1.3943999999999903,0.0077619663648123656,5.189031378517357,0,176.7439
34260000000000,AAPL,178.4962,178.4962,175.8907,176.7439,6639,tech,0.0012,177.67075,2.605499999999978,0.014596949402844308,5.17724207360518,-0.0052166689133304744,177.0972

Note the row order changed: the CSV is symbol-major (all 390 AAPL bars, then all 390 META bars, …), while bars printed time-major — the by symbol grouping in the window step reorders the table.

That reordering is exactly why next_close gets its own by symbol block. An unpartitioned lead(close, 1) reads the adjacent row of the whole table, and on a symbol-major table that is a correct same-symbol lookahead everywhere except the five symbol boundaries, where it silently returns the next instrument's first close. It is wrong on 6 rows out of 2340 — small enough to survive review, and precisely the train/serve skew this demo exists to argue against. The leak is symmetric: an unpartitioned lag would push each symbol's last close forward into the next symbol's first row. by symbol stops the lookahead at the group edge and emits null there, and the grouped update preserves row order.

Stage 4

Fit a ridge regression

model { … } takes an R-style formula, a method and its hyperparameters. The label is next bar's close, so the last bar of each symbol, which has no successor to look ahead to, is dropped first.

let labelled = features[filter next_close is not null];
let fit = labelled[model { next_close ~ open + range + log_mid + z5 + vol_regime,
                           method = ridge,
                           lambda = 0.1 }];

let coefficients = model_coef(fit);
coefficients;

Output

rows: 6
+---------------+-----------+
| term          | estimate  |
+---------------+-----------+
| "(intercept)" | -13.81263 |
| "open"        | 0.9929705 |
| "range"       | -8.52681  |
| "log_mid"     | 2.916274  |
| "z5"          | 322.8143  |
| "vol_regime"  | 4.463433  |
+---------------+-----------+

The fit is dominated by open at a coefficient of 0.993 — next bar's close is, unsurprisingly, almost this bar's price. The remaining terms are corrections on a much smaller scale, with z5 carrying a large coefficient because it is a near-zero-centred ratio.

filter next_close is not null removes exactly six rows of 2340 — the last bar of each of the six symbols, where the partitioned lead has no successor within its group. 2334 rows reach the fit, 389 per symbol.

Stage 5

Write the results

Three CSVs: the coefficients, the labelled feature matrix, and the fitted values in row order. write_csv returns the number of rows it wrote, which is why three bare integers close out the script's stdout.

let predictions = model_fitted(fit);

write_csv(coefficients, "examples/quant_demo/data/coefficients.csv");
write_csv(labelled,     "examples/quant_demo/data/features.csv");
write_csv(predictions,  "examples/quant_demo/data/predictions.csv");

Output

6
2334
2334

File	Rows	Contents
`coefficients.csv`	6	One row per model term, full double precision
`features.csv`	2334	The labelled design matrix, 15 columns
`predictions.csv`	2334	A single `fitted` column, aligned row-for-row with `features.csv`

coefficients.csv — written at full precision

term,estimate
(intercept),-13.812625458710277
open,0.992970507483574
range,-8.52681044215418
log_mid,2.9162741433821786
z5,322.8143305389048
vol_regime,4.463433023176217

predictions.csv — header and first three rows

fitted
179.64156929442348
176.72396924860047
175.65092898530014

Predictions are handed back as a row-aligned column rather than spliced into the feature table. Reattaching them per row becomes a one-liner once Ibex grows horizontal concat; today the consumer gets the aligned pair.

Reproducing it

Run the demo

The generator is deterministic, so the numbers on this page are the numbers you will get.

python3 examples/quant_demo/gen_data.py

./build-release/tools/ibex_eval \
    --plugin-path build-release/tools \
    examples/quant_demo/quant_demo.ibex

examples/quant_demo/run.sh wraps both steps, then runs the equivalent Polars + scikit-learn pipeline (quant_demo_polars.py) and prints a term-by-term coefficient diff, so the Ibex fit can be checked against a reference implementation.

./examples/quant_demo/run.sh

The whole thing

quant_demo.ibex

Forty lines of actual code, five easy to read and write stages and no glue.

import "csv";

let ticks         = read_csv("examples/quant_demo/data/ticks.csv");
let reference_raw = read_csv("examples/quant_demo/data/reference.csv");

// Promote both to TimeFrames so the asof join is type-checked.
let tick_stream = as_timeframe(ticks, "ts");
let reference   = as_timeframe(reference_raw, "ts");

// ── 1. Resample ticks to 1-minute OHLC + traded size ────────────────────
let bars = tick_stream[resample 1m, by symbol, select {
    open  = first(price),
    high  = max(price),
    low   = min(price),
    close = last(price),
    bar_size = sum(size)
}];
bars;

// ── 2. Asof-join the per-symbol reference table ─────────────────────────
let enriched = bars asof join reference on {ts, symbol};
enriched;

// ── 3a. Per-symbol rolling features (5-bar window) ──────────────────────
let with_rolling = enriched[window 5m, by symbol, update {
    rmean5 = rolling_mean(close)
}];

// ── 3b. Bar-level features built on top of the rolling state ────────────
let features = with_rolling[update {
    spread     = high - low,
    range      = (high - low) / open,
    log_mid    = log((high + low) / 2.0),
    z5         = (close - rmean5) / (rmean5 + 0.0001)
}];

// ── 3c. The label: next bar's close, per symbol ─────────────────────────
//   `by symbol` is load-bearing. Without it `lead` looks at the next row
//   of the whole table, and at each symbol boundary that row belongs to
//   the *next* symbol.
let features = features[by symbol, update {
    next_close = lead(close, 1)
}];

// ── 4. Train: fit a ridge regression of close on the bar features ───────
let labelled = features[filter next_close is not null];
let fit = labelled[model { next_close ~ open + range + log_mid + z5 + vol_regime,
                            method = ridge,
                            lambda = 0.1 }];

let coefficients = model_coef(fit);
coefficients;

// ── 5. Score: persist features + fitted predictions in row order ────────
let predictions = model_fitted(fit);

write_csv(coefficients, "examples/quant_demo/data/coefficients.csv");
write_csv(labelled,     "examples/quant_demo/data/features.csv");
write_csv(predictions,  "examples/quant_demo/data/predictions.csv");