Ibex language tour and reference

Use Ibex as a DSL inside a real notebook workflow

The ibex_pyarrow bridge evaluates Ibex code and returns a pyarrow.Table. The ibex_ipython extension adds %%ibex, %ibexfile, and %ibexreset, so notebooks can bind pandas or Arrow tables and ordinary Python scalars into Ibex, keep expensive table loads alive across cells, and keep plotting in Python.

Scalar bindings currently support Python int, float, bool, str, datetime.date, and datetime.datetime, with the last two mapped to Ibex Date and Timestamp.

This keeps Ibex focused on joins, aggregations, reshaping, and time-series work, while reusing pandas, matplotlib, and the rest of the notebook ecosystem for presentation.

Build the bridge with cmake --build build-release --parallel --target ibex_pyarrow, then start IPython from the repository root and load the extension.

Load once, reuse across notebook cells

%%ibex keeps table-valued let bindings in a hidden Ibex session. Large CSV or parquet loads do not need to be repeated in every cell.

Use Ibex from pure R and hand results straight to ggplot2

The in-repo ribex package evaluates inline Ibex queries or .ibex files from R. Results come back as a data.frame by default, so the common ggplot2 path is immediate. For lower-level interop, format = "nanoarrow" returns the Arrow-backed result.

Like the Python notebook path, ribex supports a persistent Ibex session, so large CSV or parquet loads can happen once and then be reused across later calls.

For literate reports, ribex can also register a real {ibex} knitr engine inside R Markdown, so Ibex chunks and ordinary R / ggplot2 chunks can share one named session.

R input bindings are available by copy for named data.frame tables and named scalar lists. Scalars currently support R integer, double, logical, character, Date, and POSIXct.

Read delimited files with or without headers

The bundled CSV plugin supports custom delimiters and headerless files. When has_header is false, Ibex synthesizes column names col1, col2, and so on.

SQLite is also available through the optional ADBC plugin, so the same pipeline can start from a SQL query instead of a flat file when you want database-side staging first.

Generate feature families without leaving Ibex

Braced select and update blocks support a compile-time map form. It expands from earlier string-list let bindings into ordinary named fields before lowering, so runtime execution stays unchanged.

Use backtick aliases with ${name} interpolation and get(name) inside the map body to turn compile-time names into column references.

columns(table_expr) returns a one-column metadata table named name, so generated schemas can stay in Ibex instead of being rebuilt in notebook host code.

External functions bring data in

extern fn declares a C++ data-source function as a first-class Ibex binding. The compiler resolves it at link time; the REPL loads the corresponding plugin .so at runtime.

Filter expressions support arithmetic, comparisons, and boolean logic (and, or, not). Multiple clauses chain in reading order.

Output plugins mirror the input API: write_csv, write_json, and write_parquet accept a DataFrame and a path and return the number of rows written.

The dedicated I/O guide covers CSV and Parquet usage in more detail, plus SQLite reads through ADBC and Kafka/Redpanda streaming into WebSocket dashboards for both JSON and Schema-Registry-backed Avro, including headerless files, custom delimiters, schema hints, and end-to-end demos.

Build DataFrames from literals or existing column vectors

The Table { … } constructor builds a DataFrame column-by-column. Each column value is either an inline Series literal [v, v, …] or any Ibex expression that produces a series or table — no file or plugin declaration required.

For table-valued expression columns, the runtime extracts the column from the result table: a single-column result is used directly; a multi-column result is matched by column name. Literal and expression columns may be freely mixed.

All columns must have equal row counts. The result is a first-class DataFrame: chain operations, promote to as_timeframe, or join with another table.

One clause for projection and grouped reduction

select doubles as a projection and an aggregation clause. Add by to group; omit it for a global aggregate.

Aggregate functions: first, last, sum, mean, min, max, count, median, std, ewma. All skip null rows by default. IEEE NaN / Inf are not nulls, so normalize them explicitly before aggregation when needed.

Broadcast group aggregates back to every row

update + by evaluates each expression per group and writes the result back to every row in that group — no row reduction, no separate join step. This is the equivalent of a SQL window function with PARTITION BY and no frame clause.

Use it to attach group statistics (mean, z-score, rank) alongside the original columns without losing any rows.

Add or replace columns without losing any data

update appends named computed columns to every row. All existing columns pass through untouched. A tuple left-hand side unpacks a multi-column result in one assignment. The bare update = expr form merges all columns from a table-returning expression.

Relabel columns without touching the data

rename maps old column names to new ones while keeping every other column intact. Use the single-name shorthand or the braced multi-rename form. Column order is preserved.

Because rename runs before select in canonical order, renamed columns are immediately visible to the rest of the pipeline.

Sort, keep the first or last rows, and deduplicate with explicit intent

order accepts a single column, a multi-key block with asc / desc annotations, or no argument at all to sort by every column in schema order.

head keeps the first n rows in the current order, while tail keeps the last n rows. Both work per group when combined with by.

distinct deduplicates on a single column or a set.

Use previous or next rows directly in predicates

Pure row-preserving functions such as lag and lead can appear directly inside filter. They use the current row order, so add a preceding order block when the input order needs to be explicit. The offset can be an integer literal or scalar Int64 expression.

This keeps shift-style patterns concise without materializing temporary columns first. Date - Date returns an integer day delta, so calendar-adjacent filters can combine lag(date_col, 1) with date arithmetic directly.

Assign dense, first, min/max, or average ranks inside update

rank(col) is the shorthand single-key form. For multi-key ranking, use rank(order { ... }) and spell sort direction per key.

The surrounding by clause partitions ranking by group, which makes dense top-k filters much cleaner than distinct + order + head + semi join rewrites.

Inner, left, right, outer, semi, anti, cross, as-of — and non-equijoin predicates

The as-of join attaches the latest right row at-or-before each left timestamp. It is the standard pattern for enriching tick data with bar data, without look-ahead bias.

Join keys are named with on. When the expression after on is a comparison or boolean expression rather than a column name, it becomes a non-equijoin predicate (theta join) evaluated for every pair of rows. Both tables must be TimeFrames for an asof join.

SQL-style three-valued logic, Arrow-style storage

Every column carries an Arrow-compatible validity bitmap alongside its value buffer. Null propagates through arithmetic and comparisons: null * x = null, null > 5 = null. filter silently drops rows where the predicate is null.

Use is null / is not null to inspect the validity bitmap directly — these predicates always return a valid Bool, never null. Three-valued boolean operators follow SQL rules: true OR null = true, false AND null = false.

Nulls arise from joins with unmatched rows (left join: unmatched right columns; right join: unmatched left columns; outer join: both sides), dcast (missing pivot combinations), and certain aggregate functions (std on fewer than 2 non-null values, ewma on an empty group, skew on fewer than 3, kurtosis on fewer than 4).

Floating-point NaN and Inf values are separate from nulls. Use null_if_nan or null_if_not_finite to convert them into nulls before grouped reductions.

Math and date extraction built in

Ibex ships a small set of built-in scalar functions that apply element-wise to a column. Math: abs, log, sqrt, row-wise pmin, row-wise pmax. Date/time extraction: year, month, day, hour, minute, second.

All scalar functions operate on Numeric or Date | Timestamp columns and produce a new column of the return type. Use them inside update, select, or filter expressions.

Explicit casts and rounding for numeric conversions

Ibex never coerces Int ↔ Float implicitly. Use type-name constructors — Int64(x), Float64(x), etc. — for explicit numeric casts. They work on scalars and on entire columns.

Float → Int casts require the value to be a whole number. Use round(x, mode) first when the value may have a fractional part. Five modes are available: nearest (ties away from zero), bankers (ties to even — statistically unbiased), floor, ceil, trunc.

Type annotations on let bindings and function parameters are optional. When present, they are validated: scalar types must match exactly; schema annotations require all declared columns to be present with the correct types.

Generate random columns in a single vectorized pass

Eight built-in rand_* functions each produce one independent draw per row — no row-by-row loop, no user-space scalar overhead. Each thread has its own mt19937_64 seeded from std::random_device, so parallel queries produce independent streams without locking.

Continuous distributions return Float64: rand_uniform, rand_normal, rand_student_t, rand_gamma, rand_exponential. Discrete distributions return Int64: rand_bernoulli, rand_poisson, rand_int.

Fill, repeat, and cycle columns with named-argument syntax

rep(x) mirrors R’s rep(): it fills a column by repeating a scalar literal or cycling an existing column. Named arguments — times, each, and length_out — control the exact repetition pattern.

Passing a Bool literal produces a first-class Series<Bool> (a boolean mask), useful as a computed flag or validity indicator alongside numeric columns.

Running totals and products in a single O(n) pass

cumsum(col) and cumprod(col) compute prefix sums and products over any Int or Float column. Both return the same type as their input and are valid in select and update blocks — DataFrame or TimeFrame — with or without a window clause.

Typical use: cumulative P&L, compounded returns, or an index-rebased price series.

Clean missing and dirty float values before aggregation

Three built-in functions cover data.table’s nafill semantics in a single O(n) pass:

• fill_null(col, value) — constant fill
• fill_forward(col) — LOCF (last observation carried forward)
• fill_backward(col) — NOCB (next observation carried backward)

All three work on any column type (Int, Float, String, …) in select or update blocks. Unfillable leading (LOCF) or trailing (NOCB) nulls remain null.

For dirty Float64 columns, Ibex also provides is_nan(col), null_if_nan(col), and null_if_not_finite(col). These are the right way to make grouped mean / sum ignore NaN and infinities.

Time-based rolling aggregates in one O(n) pass

as_timeframe validates sort order in O(n) and records the time-index column. window <dur> sets the lookback; rolling functions (rolling_sum, rolling_mean, rolling_count, lag) use a two-pointer scan with no per-row heap allocation.

Duration literals: 1s, 30s, 1m, 5m, 1h, …

Aggregate ticks into equal-width time buckets

resample <dur> floors timestamps into fixed-width intervals and reduces each bucket to one output row. Combine with by for per-symbol bars.

The output TimeFrame carries the bucket start time as its time index, ready for downstream joins or further resampling.

Pivot and unpivot with melt & dcast

melt converts a wide DataFrame to long format: id columns stay fixed while measure columns are unpivoted into variable / value rows. Combine with select to choose specific measure columns.

dcast is the inverse — it pivots a long table back to wide format. Distinct values of the pivot column become new column names. Combine with by for row keys and select for the value column. Missing cells are filled with null.

Extract single values and compile to C++

scalar pulls one typed value out of a single-cell result table. It is available as a binding in subsequent expressions.

ibex_compile transpiles a .ibex file to a self-contained C++ source file. The helper script compiles and links it in one step.

Continuous pipelines with no new syntax

Stream { } wires a source extern, an anonymous transform block, and a sink extern into a continuous event loop. The transform is written in plain Ibex — the same clauses used for batch queries.

When the transform contains resample, the runtime automatically switches to TimeBucket mode: it buffers rows and emits one output batch per closed time bucket. Otherwise it operates PerRow, forwarding each incoming batch immediately.

In TimeBucket mode the bucket is flushed by whichever trigger fires first: a wall-clock check (the configured duration has elapsed since the bucket opened) or a data-timestamp check (an incoming row carries a later bucket's timestamp). The wall-clock trigger gives prompt delivery for real-time feeds; the data-timestamp trigger handles historical replay correctly.

Source timeout contract: the wall-clock check runs only when the source call returns. Sources should use a short internal timeout and return StreamTimeout{} when no data arrives — the runtime fires the flush check and immediately calls the source again. Ibex does not buffer data on behalf of the source; whether messages arriving during this window are preserved depends on the transport. For OS kernel-backed sockets (UDP, TCP) the kernel's receive buffer holds packets safely. For user-space transports use StreamBuffered (see below).

Use import to load a library stub that registers the extern source and sink functions as a plugin.

A ready-made producer queue for in-process sources

For in-process or user-space transports the plugin needs a thread-safe queue between its producer and the Ibex event loop. StreamBuffered provides one out of the box — a lockless SPSC ring buffer paired with a compatible ExternFn — so the plugin author doesn't have to implement their own.

Use make_buffered_source(producer_fn) to let the Ibex query control the capacity: the plugin only supplies the data-production logic; the ring is lazily initialised on the first event-loop call using the capacity argument from the query. A producer thread is started at that point, calls buf.write(table) (blocking on backpressure), and signals completion with buf.close(). The event loop drains the ring, returning StreamTimeout{} when empty so wall-clock bucket flushes still fire on schedule.

Cache-line-separated std::atomic head and tail indices keep producer and consumer on independent cache lines — no false sharing, no mutex on the hot path.

When to use each approach: for UDP/TCP sockets the OS kernel already manages a receive buffer (SO_RCVBUF) independently of the application — no application-level queue is needed. Return StreamTimeout{} directly from recvfrom with a short socket timeout instead. StreamBuffered is for in-process queues, shared-memory channels, or any transport without a kernel-managed receive buffer.

R-style formula syntax for regression, built into the query bracket

The model clause lets you fit OLS, ridge, WLS, or the plugin-backed lightgbm method regressions with a familiar y ~ x1 + x2 formula. Ibex builds the design matrix automatically — numeric columns pass through, categorical columns are dummy-encoded, and interaction terms (:) and crossing (*) expand just like in R.

Results are accessed via S3-style accessor functions: model_coef, model_summary, model_fitted, model_residuals, model_importance, and model_r_squared. The method = parameter is optional and defaults to ols; it also supports built-ins (ols, ridge, wls) plus plugin methods registered by name. The bundled lightgbm plugin links real Microsoft LightGBM and trains gradient-boosted trees — use model_fitted for in-sample predictions, model_importance for per-feature gains, and model_predict(m, newdata) to score new data by reusing the trained model.