Contribute to the Ibis codebase

Workflow

Getting started

First, set up a development environment.

If you’re developing on Mac M1 or M2, see docs for setting up Colima.

Taking issues

If you find an issue you want to work on, write a comment with the text /take on the issue. GitHub will then assign the issue to you.

Running the test suite

To run tests that do not require a backend:

pytest -m core

Backend test suites

You may be able to skip this section

If you haven’t made changes to the core of ibis (e.g., ibis/expr) or any specific backends (ibis/backends) this material isn’t necessary to follow to make a pull request. You can jump directly to the Writing the commit section

First, we need to download example data to run the tests successfully:

just download-data

To run the tests for a specific backend (e.g. sqlite):

pytest -m sqlite

Running the test suite efficiently when making large changes to a backend

If you are adding a new backend, or dealing with a major refactor, some pytest tricks can help speed you along through finding and fixing various test failures.

Run the tests for your backend tests in parallel

pytest -m duckdb -n auto --dist loadgroup --snapshot-update

Then run only the failed tests using stepwise and don’t randomize the test order:

pytest -m duckdb --randomly-dont-reorganize --lf --sw

pytest will stop after a test failure, then you can fix the failing test, then re-run the same stepwise command and it will pick up where it left off.

Setting up non-trivial backends

MacOS users on arm64 CPUs must use a different setup

If you are working with an arm64 architecture (Mac M1/M2) you can setup Docker with Colima. Refer to set up colima. Or you can download Docker desktop.

These client-server backends need to be started before testing them. They can be started with docker compose directly, or using the just tool.

ClickHouse: just up clickhouse
Exasol: just up exasol (no arm64 support)
Flink: just up flink
Impala: just up impala
SQL Server: just up mssql
MySQL: just up mysql
Oracle: just up oracle
PostgreSQL: just up postgres
RisingWave: just up risingwave
Trino: just up trino
Druid: just up druid

and then run the test suite for the backend you just started. For example, if you ran just up postgres:

pytest -m postgres

Test the backend locally

If anything seems amiss with a backend, you can of course test it locally:

export PGPASSWORD=postgres
psql -t -A -h localhost -U postgres -d ibis_testing -c "select 'success'"

Adding appropriate tests

If you pull request involves a new feature, you should add appropriate tests to cover all ordinary and edge cases.

Pytest markers can be used to assert that a test should fail or raise a specific error. We use a number of pytest markers in ibis:

pytest.mark.notimpl: We can implement/fix/workaround this on the ibis side, but haven’t yet.
pytest.mark.notyet: This requires the backend to implement/fix something. We can’t/won’t do it on the ibis side.
pytest.mark.never: The backend will never support this / pass this test. We shouldn’t have any hope of trying to fix this. A common example here is a test running on sqlite that relies on strong typing.

Refrain from using a generic marker like pytest.mark.xfail.

Writing the commit

Ibis follows the Conventional Commits structure. In brief, the commit summary should look like:

fix(types): make all floats doubles

The type (e.g. fix) can be:

fix: A bug fix. Correlates with PATCH in SemVer
feat: A new feature. Correlates with MINOR in SemVer
docs: Documentation only changes
style: Changes that do not affect the meaning of the code (white-space, formatting, missing semi-colons, etc) ` If the commit fixes a Github issue, add something like this to the bottom of the description:
```
fixes #4242
```

Submit a pull request

Ibis follows the standard GitHub pull request process. The team will review the PR and merge when it’s ready.

Colima setup for Mac M1/M2 users

Colima is a container runtime that supports Mac M1 and M2 CPUs.

If you are working with an M1 or M2 CPU architecture, you will need to run Docker to be able to test some backends supported by Ibis. With Colima¹, you can run Docker on newer Macs without using Docker Desktop².

Get started

Uninstall Docker Desktop

If you have Docker desktop installed, follow ³ or ⁴, both of them have a section on how to uninstall Docker Desktop.

Install Docker client with Homebrew

Make sure your Homebrew installation is up to date.

$ brew install docker

Check installation by running

$ docker --version

Notice we haven’t installed any Docker Engine yet, and only the Docker client information is displayed.

Install Colima

$ brew install colima

Install Docker Compose plugin (optional)

If you want to be able to run docker compose or run just up, follow these steps.

The official Docker documentation suggests running the following commands in a separate terminal:

Replace the link on the curl step with the version you desired, find them here https://github.com/docker/compose/releases/

$ DOCKER_CONFIG=${DOCKER_CONFIG:-$HOME/.docker}
$ mkdir -p $DOCKER_CONFIG/cli-plugins
$ curl -SL https://github.com/docker/compose/releases/download/v2.24.6/docker-compose-darwin-aarch64 -o $DOCKER_CONFIG/cli-plugins/docker-compose

Then add execution permission to the downloaded binary:

$ chmod +x $DOCKER_CONFIG/cli-plugins/docker-compose

Check it worked:

$ docker compose version

Start Colima

$ colima start

You can now run docker version and you will see the engine.

By default this launches containers with 2 CPUs, 2GB of RAM, and 60GB of disk space.

You can modify this as well as the architecture by passing command line arguments (--cpu, --memory, --disk, and --arch) to colima start.

$ colima status
$ colima list

In your Ibis clone

$ just up postgres

Once the just command finishes, you can run the tests by doing:

$ pytest -m postgres

Once you are done, you can stop the container by doing:

$ just down postgres

If you are done for the day, and want to avoid the Colima instance eating your resources, you will want to stop it.

$ colima stop

If you upgraded colima, you need to delete the existing instance. If you want to modify the allocation, you need to delete the existing instance too (unless you are going only up).

$ colima delete

`x86_64` or `amd64` based containers

While starting the containers based on x86_64 / amd64, the architecture flag needs to be set in two places: 1. Add platform: linux/amd64 for the service in compose.yaml. 2. Set the --arch flag while starting the VM colima start --arch x86_64

For instance, this step is necessary for the oracle service in compose.yaml. Otherwise, the container will fail shortly after getting started.