Contributing to Ibis
Set up a development environment
There are two primary ways to setup a development environment.
nix: fewer steps, isolated
conda: more steps, not isolated
At least one of
conda is required to contribute to ibis.
As of 2022-01-05 there is experimental support for Python 3.10. However, there are a number of problems with dependencies and development tools that ibis uses and we cannot offically support Python 3.10 until those are fixed.
- Download and install
nix-shell -p gh # or nix-env -iA gh
Fork and clone the ibis repository:
# you will likely need to auth, gh will guide you through the steps gh repo fork --clone --remote ibis-project/ibis
nix-shellin the checkout directory:
cd ibis # set up the cache to avoid building everything from scratch nix-shell -p cachix --run 'cachix use ibis' # start a nix-shell # # this may take awhile to download artifacts from the cache nix-shell
- Download and install Miniconda
conda install -c conda-forge gh
Fork and clone the ibis repository:
gh repo fork --clone --remote ibis-project/ibis
Create a Conda environment from a lock file in the repo:
cd ibis conda create -n ibis-dev -f conda-lock/<platform-64-pyver>.lock
Activate the environment
conda activate ibis-dev
Install your local copy of
ibisinto the Conda environment. In the root of the project run:
pip install -e .
Find an issue to work on
All contributions are welcome! Code, docs, and constructive feedback are all great contributions to the project.
If you don't have a particular issue in mind head over to the GitHub issue
tracker for Ibis and look for open issues with the label
good first issue.
Feel free to help with other issues that aren't labeled as such, but they may be more challenging.
Once you find an issue you want to work on, write a comment with the text
/take on the issue. GitHub will then assign the issue to you.
This lets people know you're working on the issue. If you find an issue that has an assignee, comment on the issue and ask whether the assignee is still working on the issue.
Make a branch
The first thing you want to do is make a branch. Let's call it
git checkout -b useful-bugfix
Make the desired change
Let's say you've made a change to
ibis/expr/types.py to fix a bug reported in issue #424242 (not actually an issue).
git status should give output similar to this:
$ git status On branch useful-bugfix Your branch is up to date with 'origin/useful-bugfix'. Changes not staged for commit: (use "git add <file>..." to update what will be committed) (use "git restore <file>..." to discard changes in working directory) modified: ibis/expr/types.py no changes added to commit (use "git add" and/or "git commit -a")
Run the test suite
Next, you'll want to run a subset of the test suite.
To run a subset of the ibis tests use the following command:
PYTEST_BACKENDS="sqlite pandas" pytest ibis/tests ibis/backends/tests
You can change
PYTEST_BACKENDS="sqlite pandas" to include one or more space-separated
supported backends that you want to test.
It isn't necessary to provide
PYTEST_BACKENDS at all, but it's useful for
exercising more of the library's test suite.
Commit your changes
cz is already installed in your environment if you followed the setup instructions
Next, you'll want to commit your changes.
Ibis's commit message structure follows
It isn't necessary to use
cz commit to make commits, but it is necessary
to follow the instructions outlined in this table in this
Stage your changes and run
$ git add . $ cz commit
You should see a series of prompts about actions to take next:
- Select the type of change you're committing. In this case, we're committing a bug fix, so we'll select fix:
? Select the type of change you are committing (Use arrow keys) » fix: A bug fix. Correlates with PATCH in SemVer feat: A new feature. Correlates with MINOR in SemVer docs: Documentation only changes style: Changes that do not affect the meaning of the code (white-space, formatting, missing semi-colons, etc) refactor: A code change that neither fixes a bug nor adds a feature perf: A code change that improves performance test: Adding missing or correcting existing tests build: Changes that affect the build system or external dependencies (example scopes: pip, docker, npm) ci: Changes to our CI configuration files and scripts (example scopes: GitLabCI)
Generally you don't need to think too hard about what category to select, but note that:
featwill cause a minor version bump
fixwill cause a patch version bump
everything else will not cause a version bump, unless it's a breaking change (continue reading these instructions for more info on that)
Next, you're asked what the scope of this change is:
? What is the scope of this change? (class or file name): (press [enter] to skip)
This is optional, but if there's a clear component or single file that is
modified you should put it. In our case, let's assume the bug fixed a type
inference problem, so we'd type in
type-inference at this prompt.
- You'll then be asked to type in a short description of the change which will be the commit message title:
? Write a short and imperative summary of the code changes: (lower case and no period) fix a type inference issue where floats were incorrectly cast to ints
Let's say there was a problem with spurious casting of float to integers, so
we type in the message above. That number on the left (here
(69)) is the
length of description you've typed in.
- Next you'll be asked for a longer description, which is entirely optional unless the change is a breaking change, or you feel like a bit of prose
? Provide additional contextual information about the code changes: (press [enter] to skip) A bug was triggered by some incorrect code that caused floats to be incorrectly cast to integers.
For non breaking changes, this isn't strictly necessary but it can be very helpful when a change is large, obscure, or complex. For this example let's just reiterate most of what the commit title says.
- Next you're asked about breaking changes:
? Is this a BREAKING CHANGE? Correlates with MAJOR in SemVer (y/N)
If you answer
y, then you'll get an additional prompt asking you to
describe the breaking changes. This description will ultimately make its way
into the user-facing release notes. If there aren't any breaking changes, press enter.
Let's say this bug fix does not introduce a breaking change.
- Finally, you're asked whether this change affects any open issues (ignore the bit about breaking changes) and if yes then to reference them:
? Footer. Information about Breaking Changes and reference issues that this commit closes: (press [enter] to skip) fixes #424242
Here we typed
fixes #424242 to indicate that we fixed issue #9000.
Whew! Seems like a lot, but it's rather quick once you get used to it. After that you should have a commit that looks roughly like this, ready to be automatically rolled into the next release:
commit 4049adbd66b0df48e37ca105da0b9139101a1318 (HEAD -> useful-bugfix) Author: Phillip Cloud <email@example.com> Date: Tue Dec 21 10:30:50 2021 -0500 fix(type-inference): fix a type inference issue where floats were incorrectly cast to ints A bug was triggered by some incorrect code that caused floats to be incorrectly cast to integers. fixes #424242
Push your changes
Now that you've got a commit, you're ready to push your changes and make a pull request!
gh pr create
Follow the prompts, and
gh will print a link to your PR upon successfuly submission.
Automatic dependency updates
WhiteSource Renovate will run at some cadence (outside of traditional business hours) and submit PRs that update dependencies.
These upgrades use a conservative update strategy, which is currently to increase the upper bound of a dependency's range.
The PRs it generates will regenerate a number of other files so that in most cases contributors do not have to remember to generate and commit these files.
Manually updating dependencies
Do not manually edit
setup.py, it is automatically generated from
# if using nix $ ./dev/poetry2setup -o setup.py # it not using nix, requires installation of tomli and poetry-core $ PYTHONHASHSEED=42 python ./dev/poetry2setup.py -o setup.py
from the repository root.
Updates of minor and patch versions of dependencies are handled automatically by
Style and formatting
The following tools are run in both CI and
pre-commit checks to ensure codebase hygiene:
||Formatting Python code|
||Formatting and sorting
||Linting Python code|
||Linting nix files|
||Formatting nix files|
||Linting shell scripts|
||Formatting shell scripts|
||Ensuring the latest available Python syntax is used|
If you use
nix-shell all of these are setup for you and ready to use, you don't
need to install any of these tools.
We use numpydoc as our standard format for docstrings.
We aim to make our individual commits small and tightly focused on the feature they are implementing or bug being fixed. If you find yourself making functional changes to different areas of the codebase, we prefer you break up your changes into separate Pull Requests. In general, a philosophy of one Github Issue per Pull Request is a good rule of thumb.
Maintainers should be performing a minimum number of tasks, deferring to automation as much as possible:
- Reviewing pull requests
- Merging pull requests
A number of tasks that are typically associated with maintenance are partially or fully automated:
- Updating library dependencies: this is handled automatically by WhiteSource Renovate
- Updating github-actions: this is handled automatically by WhiteSource Renovate
- Updating nix dependencies: this is a job run at a regular cadence to update nix dependencies
Occasionally you may need to manually lock poetry dependencies, which can be done by running
poetry update --lock
If a dependency was updated, you'll see changes to
poetry.lock in the current directory.
PRs can be merged using the
gh command line tool
or with the GitHub web UI.
Ibis is released in two places:
The conda-forge package is released using the conda-forge feedstock repository
After a release to PyPI, the conda-forge bot automatically updates the ibis package.