Ibex | Getting Started

On this page

A first session, end to end

Install and start the REPL Read CSV data Inspect tables Transform tables Aggregate and sort Join tables Write output Run the full script

Install

Get a REPL and the bundled I/O plugins

The quickest start is a release archive with the ibex executable and plugin shared libraries. Building from source gives you the latest work tree.

Download a release

Unpack the archive, keep the plugin directory next to the executable, and start the REPL with an explicit plugin path.

# from the unpacked release directory
./ibex --plugin-path ./plugins

Releases are published at github.com/bobjansen/Ibex/releases.

Build from source

Use a release build for interactive examples and benchmarks. The bundled CSV and Parquet plugins are built with the normal CMake build.

cmake -B build-release -S . -G Ninja \
  -DCMAKE_C_COMPILER=clang \
  -DCMAKE_CXX_COMPILER=clang++ \
  -DCMAKE_BUILD_TYPE=Release
cmake --build build-release

./build-release/tools/ibex --plugin-path ./build-release/tools

Reading data

Import an I/O plugin, then read data into tables

Ibex keeps file formats in plugins. Import "csv" for read_csv / write_csv, or "parquet" for read_parquet / write_parquet. The examples below use CSV so the input data is visible, but the table pipeline is the same after loading.

Imports

Load the readers and writers you need

Imports are explicit, so a script shows which external data formats it depends on. Once imported, plugin functions are called like any other Ibex function.

import "csv";
import "parquet";

Input data

Build two tables, then write them as CSV

A Table literal builds a small table directly from column arrays. Here it also gives us convenient input files for the rest of the guide. The cars table is inspired by R's mtcars dataset.

import "csv";

let sample_cars = Table {
    car = ["Mazda RX4", "Hornet 4 Drive", "Duster 360", "Merc 240D",
           "Fiat 128", "Toyota Corolla", "Camaro Z28", "Volvo 142E"],
    maker_id = [1, 2, 2, 3, 4, 5, 6, 7],
    mpg = [21.0, 21.4, 14.3, 24.4, 32.4, 33.9, 13.3, 21.4],
    cyl = [6, 6, 8, 4, 4, 4, 8, 4],
    hp = [110, 110, 245, 62, 66, 65, 245, 109],
    wt = [2.620, 3.215, 3.570, 3.190, 2.200, 1.835, 3.840, 2.780],
    am = [1, 0, 0, 0, 1, 1, 0, 1]
};

let sample_makers = Table {
    maker_id = [1, 2, 3, 4, 5, 6, 7],
    maker = ["Mazda", "AMC", "Mercedes", "Fiat", "Toyota", "Chevrolet", "Volvo"]
};

write_csv(sample_cars, "cars.csv");
write_csv(sample_makers, "makers.csv");

Load

Import CSV support and read both files

import "csv" loads the bundled CSV plugin. The let bindings name the tables so you can inspect and reuse them in later expressions.

import "csv";

let cars = read_csv("cars.csv");
let makers = read_csv("makers.csv");

Nulls

Tell the reader how missing values are written

CSV files often use empty fields, NA, or null. Pass a comma-separated null specification when those tokens should become typed nulls instead of strings.

let cars_with_nulls = read_csv(
    "cars.csv",
    nulls = "<empty>,NA,null"
);

Inspect

Look at the table before transforming it

The REPL is the quickest way to learn what a source produced. Use command lines for metadata, and evaluate table expressions directly to see rows.

REPL

Check schema, rows, and named values

These commands do not change the program. They help you verify that paths, plugins, inferred types, and schema hints are doing what you expect.

ibex> :tables
tables: cars

ibex> :schema cars
columns:
  car: String
  maker_id: Int64
  mpg: Float64
  cyl: Int64
  hp: Int64
  wt: Float64
  am: Int64

ibex> :head cars 5
rows: 8
+------------------+----------+------+-----+-----+-------+----+
| car              | maker_id | mpg  | cyl | hp  | wt    | am |
+------------------+----------+------+-----+-----+-------+----+
| "Mazda RX4"      | 1        | 21   | 6   | 110 | 2.62  | 1  |
| "Hornet 4 Drive" | 2        | 21.4 | 6   | 110 | 3.215 | 0  |
| "Duster 360"     | 2        | 14.3 | 8   | 245 | 3.57  | 0  |
| "Merc 240D"      | 3        | 24.4 | 4   | 62  | 3.19  | 0  |
| "Fiat 128"       | 4        | 32.4 | 4   | 66  | 2.2   | 1  |
+------------------+----------+------+-----+-----+-------+----+
... (3 more rows)

ibex> :describe cars
columns:
  car: String
  maker_id: Int64
  mpg: Float64
  cyl: Int64
  hp: Int64
  wt: Float64
  am: Int64
rows: 8
+------------------+----------+------+-----+-----+-------+----+
| car              | maker_id | mpg  | cyl | hp  | wt    | am |
+------------------+----------+------+-----+-----+-------+----+
| "Mazda RX4"      | 1        | 21   | 6   | 110 | 2.62  | 1  |
| "Hornet 4 Drive" | 2        | 21.4 | 6   | 110 | 3.215 | 0  |
| "Duster 360"     | 2        | 14.3 | 8   | 245 | 3.57  | 0  |
| "Merc 240D"      | 3        | 24.4 | 4   | 62  | 3.19  | 0  |
| "Fiat 128"       | 4        | 32.4 | 4   | 66  | 2.2   | 1  |
| "Toyota Corolla" | 5        | 33.9 | 4   | 65  | 1.835 | 1  |
| "Camaro Z28"     | 6        | 13.3 | 8   | 245 | 3.84  | 0  |
| "Volvo 142E"     | 7        | 21.4 | 4   | 109 | 2.78  | 1  |
+------------------+----------+------+-----+-----+-------+----+

Expressions

Evaluate a small projection

A bare expression prints in the REPL and in script mode. This is a useful pattern while building a longer pipeline one clause at a time.

ibex> cars[
    select { car, mpg, cyl, hp, wt },
    head 3
];
rows: 3
+------------------+------+-----+-----+-------+
| car              | mpg  | cyl | hp  | wt    |
+------------------+------+-----+-----+-------+
| "Mazda RX4"      | 21   | 6   | 110 | 2.62  |
| "Hornet 4 Drive" | 21.4 | 6   | 110 | 3.215 |
| "Duster 360"     | 14.3 | 8   | 245 | 3.57  |
+------------------+------+-----+-----+-------+

Expressions and contexts

Use bracket clauses to transform rows and columns

An Ibex table expression is table[clauses]. Inside the brackets, select chooses output columns, update adds or replaces columns while keeping the rest, and filter keeps rows.

Select

Project columns and compute new ones

select creates a new table with only the fields listed in braces. Bare names pass columns through. Named expressions such as hp_per_klb = ... create computed fields.

let power_table = cars[
    select {
        car,
        mpg,
        cyl,
        maker_id,
        hp_per_klb = hp / wt
    }
];

Update

Add columns without dropping existing data

update is the Ibex version of "with columns": it keeps the original table and appends or replaces named fields.

let enriched = cars[
    update {
        hp_per_klb = hp / wt,
        mpg_per_klb = mpg / wt,
        is_manual = am == 1
    }
];

Filter

Keep rows that match a predicate

Clauses run in reading order. This example computes car metrics first, then keeps efficient manual-transmission cars, then selects the columns to display.

ibex> enriched[
    filter mpg >= 22.0 && is_manual,
    select { car, mpg, cyl, hp_per_klb, mpg_per_klb }
];
rows: 2
+------------------+------+-----+------------+-------------+
| car              | mpg  | cyl | hp_per_klb | mpg_per_klb |
+------------------+------+-----+------------+-------------+
| "Fiat 128"       | 32.4 | 4   | 30         | 14.72727    |
| "Toyota Corolla" | 33.9 | 4   | 35.42234   | 18.47411    |
+------------------+------+-----+------------+-------------+

Aggregation

Group, sort, and keep the rows you need

Pair select with by to aggregate. Add order, head, or tail when you want leaderboard-style results.

Group

Aggregate with select + by

When select appears with by, aggregate functions such as mean, min, max, and count run once per group.

let summary = enriched[
    select {
        cars = count(),
        avg_mpg = mean(mpg),
        min_mpg = min(mpg),
        max_mpg = max(mpg),
        avg_hp_per_klb = mean(hp_per_klb)
    },
    by cyl,
    order { avg_mpg desc }
];

Top N

Sort rows and keep the first few

order accepts one or more keys. Use desc for largest-first ordering, then head to keep the highest ranked rows.

let top_efficiency = enriched[
    select { car, mpg, cyl, hp, wt, mpg_per_klb },
    order { mpg_per_klb desc },
    head 3
];
ibex> top_efficiency;
rows: 3
+------------------+------+-----+-----+-------+-------------+
| car              | mpg  | cyl | hp  | wt    | mpg_per_klb |
+------------------+------+-----+-----+-------+-------------+
| "Toyota Corolla" | 33.9 | 4   | 65  | 1.835 | 18.47411    |
| "Fiat 128"       | 32.4 | 4   | 66  | 2.2   | 14.72727    |
| "Mazda RX4"      | 21   | 6   | 110 | 2.62  | 8.015267    |
+------------------+------+-----+-----+-------+-------------+

Combining data

Join tables on a key

Joins are table expressions too. Shape the left and right tables, join on a key, then keep or compute the columns you want.

Join

Attach maker names

This is an inner join: rows without a matching key on either side are dropped. Ibex also supports left, right, outer, semi, anti, and cross joins when you need a different matching rule.

let cars_with_makers = (enriched join makers on maker_id)[
    select {
        car,
        maker,
        mpg,
        cyl,
        hp_per_klb,
        mpg_per_klb
    },
    order { maker asc, car asc }
];
ibex> cars_with_makers[head 5];
rows: 5
+------------------+-------------+------+-----+------------+-------------+
| car              | maker       | mpg  | cyl | hp_per_klb | mpg_per_klb |
+------------------+-------------+------+-----+------------+-------------+
| "Duster 360"     | "AMC"       | 14.3 | 8   | 68.62745   | 4.005602    |
| "Hornet 4 Drive" | "AMC"       | 21.4 | 6   | 34.21462   | 6.656299    |
| "Camaro Z28"     | "Chevrolet" | 13.3 | 8   | 63.80208   | 3.463542    |
| "Fiat 128"       | "Fiat"      | 32.4 | 4   | 30         | 14.72727    |
| "Mazda RX4"      | "Mazda"     | 21   | 6   | 41.98473   | 8.015267    |
+------------------+-------------+------+-----+------------+-------------+

Output

Persist the result

CSV output

write_csv writes a header row and returns the number of data rows written. Call it directly when you want to see the count in the REPL; bind it with let in scripts when you only care about creating the file.

write_csv(summary, "summary.csv");

Parquet output

Use Parquet when you want typed columnar storage and faster reloads than CSV.

import "parquet";

write_parquet(cars_with_makers, "cars_with_makers.parquet");

Full script

Put it together in one file

Save this as getting-started.ibex next to the two CSV files and load it from the REPL with :load getting-started.ibex, or pass the file path to the ibex executable.

import "csv";
import "parquet";

let cars = read_csv("cars.csv");
let makers = read_csv("makers.csv");

let enriched = cars[
    update {
        hp_per_klb = hp / wt,
        mpg_per_klb = mpg / wt,
        is_manual = am == 1
    }
];

let summary = enriched[
    select {
        cars = count(),
        avg_mpg = mean(mpg),
        min_mpg = min(mpg),
        max_mpg = max(mpg),
        avg_hp_per_klb = mean(hp_per_klb)
    },
    by cyl,
    order { avg_mpg desc }
];

let top_efficiency = enriched[
    select { car, mpg, cyl, hp, wt, mpg_per_klb },
    order { mpg_per_klb desc },
    head 3
];

let cars_with_makers = (enriched join makers on maker_id)[
    select { car, maker, mpg, cyl, hp_per_klb, mpg_per_klb },
    order { maker asc, car asc }
];

let summary_rows = write_csv(summary, "summary.csv");
let car_rows = write_parquet(cars_with_makers, "cars_with_makers.parquet");

summary;
top_efficiency;
cars_with_makers;

Core syntax

The pieces you will use first

Ibex programs are built from a small set of table expressions. These are the forms used throughout this guide and in the rest of the documentation.

`let x = ...`	Name an immutable table or scalar value.
`table[ ... ]`	Apply a bracket pipeline to a table.
`filter predicate`	Keep matching rows.
`select { fields }`	Choose output columns, or aggregate when paired with `by`.
`update { fields }`	Add or replace columns while keeping the rest of the table.
`by key`	Group rows for aggregation or grouped update.
`order { key desc }`	Sort rows by one or more keys.
`a join b on key`	Combine two tables using matching key values.

Where next

Your first Ibex table pipeline