Ibis: Python Data Analysis Productivity Framework¶
Ibis is a toolbox to bridge the gap between local Python environments (like pandas and scikit-learn) and remote storage and execution systems like Hadoop components (like HDFS, Impala, Hive, Spark) and SQL databases (Postgres, etc.). Its goal is to simplify analytical workflows and make you more productive.
We have a handful of specific priority focus areas:
Enable data analysts to translate local, single-node data idioms to scalable computation representations (e.g. SQL or Spark)
Integration with pandas and other Python data ecosystem components
Provide high level analytics APIs and workflow tools to enhance productivity and streamline common or tedious tasks.
Integration with community standard data formats (e.g. Parquet and Avro)
Abstract away database-specific SQL differences
As the Apache Arrow project develops, we will look to use Arrow to enable computational code written in Python to be executed natively within other systems like Apache Spark and Apache Impala (incubating).
To learn more about Ibis’s vision, roadmap, and updates, please follow http://ibis-project.org.
Source code is on GitHub: https://github.com/ibis-project/ibis
Ibis requires a working Python 3.6+ installation. We recommend using Anaconda to manage Python versions and environments.
Installing the Python Package¶
Install Ibis from PyPI with:
pip install ibis-framework
Or from conda-forge with
conda install ibis-framework -c conda-forge
This installs the
ibis library to your configured Python environment.
Ibis can also be installed with Kerberos support for its HDFS functionality:
pip install ibis-framework[kerberos]
Some platforms will require that you have Kerberos installed to build properly.
Redhat / CentOS:
yum install krb5-devel
Ubuntu / Debian:
apt-get install libkrb5-dev
Arch Linux :
pacman -S krb5
At this time, Ibis offers some level of support for the following systems:
Direct execution of ibis expressions against Pandas objects
PySpark/Spark SQL (Experimental)
Coming from SQL? Check out Ibis for SQL Programmers.
Architecturally, Ibis features:
A pandas-like domain specific language (DSL) designed specifically for analytics, aka Ibis expressions, that enable composable, reusable analytics on structured data. If you can express something with a SQL SELECT query, you can write it with Ibis.
Integrated user interfaces to HDFS and other storage systems.
An extensible translator-compiler system that targets multiple SQL systems
SQL engine support needing code contributors:
We collect Jupyter notebooks for learning how to use ibis here: https://github.com/ibis-project/ibis/tree/master/docs/source/notebooks/tutorial. Some of these notebooks will be reproduced as part of the documentation in the tutorial section.