Operation support matrix

We provide Ibis’s operation support matrix as a Streamlit app that shows supported operations for each backend. Ibis defines a common API for analytics and data transformation code that is transpiled to native code for each backend. This code is often, but not always, SQL – see the backends concept page for details. Due to differences in SQL dialects and support for different operations in different backends, support for the full breadth of the Ibis API varies.

You can use this page to see which operations are supported on each backend.

Tip

Backends with low coverage are good places to start contributing!

Each backend implements operations differently, but this is usually very similar to other backends. If you want to start contributing to ibis, it’s a good idea to start by adding missing operations to backends that have low operation coverage.

Note

This app is built using streamlit.

You can develop the app locally by editing docs/backends/app/backend_info_app.py and opening a PR with your changes.

Test your changes locally by running

$ streamlit run docs/backends/app/backend_info_app.py

The changes will show up in the dev docs when your PR is merged!

Raw Data

You can also download data from the above tables in CSV format.

The code used to generate the linked CSV file is below.

from __future__ import annotations

from pathlib import Path

import pandas as pd

import ibis
import ibis.expr.operations as ops


def get_backends(exclude=()):
    entry_points = sorted(ep.name for ep in ibis.util.backend_entry_points())
    return [
        (backend, getattr(ibis, backend))
        for backend in entry_points
        if backend not in exclude
    ]


def get_leaf_classes(op):
    for child_class in op.__subclasses__():
        if not child_class.__subclasses__():
            yield child_class
        else:
            yield from get_leaf_classes(child_class)


def main():
    internal_ops = {
        # Never translates into anything
        ops.UnresolvedExistsSubquery,
        ops.UnresolvedNotExistsSubquery,
        ops.ScalarParameter,
    }

    public_ops = frozenset(get_leaf_classes(ops.Value)) - internal_ops
    support = {"operation": [f"{op.__module__}.{op.__name__}" for op in public_ops]}
    support.update(
        (name, list(map(backend.has_operation, public_ops)))
        # exclude flink until https://github.com/apache/flink/pull/23141 is
        # merged and released we also need to roll it into poetry
        for name, backend in get_backends(exclude=("flink",))
    )

    df = pd.DataFrame(support).set_index("operation").sort_index()

    with Path(ibis.__file__).parents[1].joinpath(
        "docs", "backends", "raw_support_matrix.csv"
    ).open(mode="w") as f:
        df.to_csv(f, index_label="FullOperation")


if __name__ == "__main__":
    main()
Back to top