Categories

All (53)

arrays (2)

athena (1)

benchmark (2)

bigquery (4)

blog (50)

case study (6)

chat (1)

clickhouse (1)

cloud (2)

community (4)

continuous integration (1)

data analysis (1)

data engineering (7)

datafusion (3)

dbt (1)

dogfood (1)

duckdb (17)

ecosystem (5)

feature engineering (2)

flink (2)

geospatial (3)

hamilton (1)

internals (1)

io (3)

kedro (1)

llms (2)

lonboard (1)

machine learning (4)

new feature (5)

overturemaps (1)

pandas (1)

performance (3)

polars (3)

portability (3)

productivity (3)

puzzle (1)

release (7)

risingwave (1)

roadmap (1)

serious (1)

shiny (1)

sneak peek (2)

snowflake (3)

sql (1)

sqlmesh (1)

stream processing (1)

streaming (1)

substrait (1)

time series (1)

udfs (1)

unix (1)

web-scale (1)

window functions (1)

Posts

Dynamic UDF Rewriting with Predicate Pushdowns

machine learning

In an ideal world, deploying machine learning models within SQL queries would be as simple as calling a built-in function. Unfortunately, many ML predictions live inside User…

Does Ibis understand SQL?

Last month, an insightful article on the dbt Developer Blog on what SQL comprehension really means came across my LinkedIn feed. The big deal about SDF is that it, unlike…

Deepyaman Datta

Querying Amazon Athena from the comfort of your Python interpreter

Have you ever wanted to harness the power of AWS Athena, but found yourself tangled up in Presto SQL syntax? Good news! Ibis now supports Amazon Athena as its newest backend…

Classification metrics on the backend

machine learning

A review of binary classification models, metrics used to evaluate them, and corresponding metric calculations with Ibis.

Taking a random cube for a walk and making it talk

Synthetic data with Ibis, DuckDB, Python UDFs, and Faker.

From query to plot: Exploring GeoParquet Overture Maps with Ibis, DuckDB, and Lonboard

With the release of DuckDB 1.1.1, now we have support for reading GeoParquet files! With this exciting update we can query rich datasets from Overture Maps using python via…

Naty Clementi and Kyle Barron

Better PyPI stats with Python

Ibis + ClickHouse + Shiny for Python = better PyPI stats.

Farewell pandas, and thanks for all the fish.

TL; DR: we are deprecating the pandas and dask backends and will be removing them in version 10.0.

Using IbisML and DuckDB for a Kaggle competition: credit risk model stability

machine learning

feature engineering

In this post, we’ll demonstrate how to use Ibis and IbisML end-to-end for the credit risk model stability Kaggle competition.

Querying 1TB on a laptop with Python dataframes

TPC-H benchmark at sf=1024 via DuckDB, DataFusion, and Polars on a MacBook Pro with 96GiB of RAM.

Ibis benchmarking: DuckDB, DataFusion, Polars

The best benchmark is your own workload on your own data.

Ibis - Now flying on Snowflake

Ibis allows you to push down compute operations on your data where it lives, with the performance being as powerful as the backend you’re connected to. But what happens if…

Phillip Cloud, Tyler White

Unlocking data insights with Ibis and SQLMesh

data engineering

Have you ever needed to learn new dialects of database languages as a data scientist or struggled with the differences between database languages? Does your company manage…

Ibis 9.0: SQLGlot-ification

Ibis 9.0 wraps up “the big refactor”, completing the transition from SQLAlchemy to SQLGlot and drastically simplifying the codebase. This is a big step toward stabilized…

Varchar in a haystack

You’re a data analyst, and a new ticket landed in your queue.

Portable dataflows with Ibis and Hamilton

data engineering

feature engineering

This post showcases how Ibis and Hamilton enable dataflows that span execution over SQL and Python. Ibis is a portable dataframe library to write procedural data…

Scaling to infinity and beyond: the Unix backend

We’re happy to announce a new Ibis backend built on the world’s best known web scale technology: Unix pipes.

Snow IO: loading data from other DBs into Snowflake

We’ve blogged about Snowflake IO before, in the context of getting local files into Snowflake as fast as possible.

Analysis of World of Warcraft data

data engineering

I grew up playing games, and with the recent re-release of World of Warcraft Classic, it seems like a perfect time to analyze some in-game data!

Stream-batch unification through Ibis

One of my focuses in the past 10 months has been to implement the Flink backend for Ibis. I was working with Apache Flink and building a feature engineering tool, and we…

Using DuckDB + Ibis for RAG

In this post, we’ll demonstrate traditional retrieval-augmented generation (RAG) with DuckDB and OpenAI via Ibis and discuss the pros and cons. Notice that because Ibis is…

Why is DuckDB the default backend for Ibis?

Occasionally people ask us why DuckDB is the default backend.

Ibis project 2024 roadmap

Welcome to the first public roadmap for the Ibis project! If you aren’t familiar with the background of Ibis or who supports it nowadays, we recommend reading why Voltron…

Ibis goes real-time! Introducing the new Flink backend for Ibis

stream processing

Ibis 8.0 marks the official release of the Apache Flink backend for Ibis. Ibis users can now manipulate data across streaming and batch contexts using the same interface.…

Deepyaman Datta

Ibis 8.0: streaming and more!

Ibis 8.0 marks the first release of stream processing backends in Ibis! This enhances the composable data ecosystem vision by allowing users to implement data transformation…

Why Voltron Data supports Ibis

The Ibis project is an independently governed open source community project to build and maintain the portable Python dataframe library. Ibis has contributors across a range…

Cody Peterson + Ian Cook

Using language models for data

This post will give an overview of how (large) language models (LMs) fit into data engineering, data analyst, and data science workflows.

Building scalable data pipelines with Kedro

data engineering

Kedro is a toolbox for production-ready data science. It is an open-source Python framework like Ibis, and together you can bring the portability and scale of Ibis to the…

Modern, hybrid, open analytics

As a Python data user, I’ve wanted a more modular, composable, and scalable ecosystem. I think it’s here. Wes McKinney released pandas c. 2009 to bring dataframes into…

Using one Python dataframe API to take the billion row challenge with DuckDB, Polars, and DataFusion

This is an implementation of the The One Billion Row Challenge:

Backend agnostic arrays

This is a redux of a previous post showing Ibis’s portability in action.

Geospatial analysis with Ibis and DuckDB (redux)

Spatial Dev Guru wrote a great tutorial that walks you through a step-by-step geospatial analysis of bike sharing data using DuckDB.

Naty Clementi and Gil Forsyth

Announcing Zulip for Ibis community chat

The Ibis project has moved to Zulip for its community chat! We’ve been testing it out for a few months and are happy with the results. From the Zulip repository’s README:

Ibis versus X: Performance across the ecosystem part 2

TL; DR: Ibis supports both Polars and DataFusion. Both backends are have about the same runtime performance, and lag far behind DuckDB on this workload. There’s negligible…

Ibis + DuckDB geospatial: a match made on Earth

Ibis now has support for DuckDB geospatial functions!

Ibis versus X: Performance across the ecosystem part 1

TL; DR: Ibis has a lot of great backends. They’re all good at different things. For working with local data, it’s hard to beat DuckDB on feature set and performance.

dbt-ibis: Write your dbt models using Ibis

data engineering

dbt has revolutionized how transformations are orchestrated and managed within modern data warehouses. Initially released in 2016, dbt quickly gained traction within the…

Querying every file in every release on the Python Package Index (redux)

Seth Larson wrote a great blog post on querying a PyPI dataset to look for trends in the use of memory-safe languages in Python.

Working with arrays in Google BigQuery

Ibis and BigQuery have worked well together for years.

Icy IO: loading local files with Snowflake

It can be challenging to load local files into Snowflake from Python.

Ibis v6.1.0

Ibis 6.1.0 is a minor release that includes new features, backend improvements, bug fixes, documentation improvements, and refactors. We are excited to see further adoption…

Ibis v6.0.0

Ibis 6.0.0 adds the Oracle backend, revamped UDF support, and many new features. This release also includes a number of refactors, bug fixes, and performance improvements.…

Ibis on 🔥: Supercharge Your Workflow with DuckDB and PyTorch

machine learning

In this blog post we show how to leverage ecosystem tools to build an end-to-end ML pipeline using Ibis, DuckDB and PyTorch.

Exploring campaign finance data

data engineering

Hi! My name is Nick Crews, and I’m a data engineer that looks at public campaign finance data.

Ibis sneak peek: writing to files

Ibis 5.0 is coming soon and will offer new functionality and fixes to users. To enhance clarity around this process, we’re sharing a sneak peek into what we’re working on.

Ibis sneak peek: examples

Ibis has been moving quickly to provide a powerful but easy-to-use interface for interacting with analytical engines. However, as we’re approaching the 5.0 release of Ibis…

Maximizing productivity with selectors

Before Ibis 5.0 it’s been challenging to concisely express whole-table operations with ibis. Happily this is no longer the case in ibis 5.0.

Ibis + Substrait + DuckDB

Ibis strives to provide a consistent interface for interacting with a multitude of different analytical execution engines, most of which (but not all) speak some dialect of…

Ibis v4.0.0

Ibis 4.0 has officially been released as the latest version of the package. This release includes several new backends, improved functionality, and some major internal…

Analysis of Ibis’s CI performance

continuous integration

data engineering

This notebook takes you through an analysis of Ibis’s CI data using ibis on top of Google BigQuery.

`ffill` and `bfill` using Ibis

window functions

Suppose you have a table of data mapping events and dates to values, and that this data contains gaps in values.

Ibis v3.1.0

Ibis 3.1 has officially been released as the latest version of the package. With this release comes new convenience features, increased backend operation coverage and a…

Marlene Mhangami

Ibis v3.0.0

The latest version of Ibis, version 3.0.0, has just been released! This post highlights some of the new features, breaking changes, and performance improvements that come…

Marlene Mhangami