Getting started

In this tutorial, you will learn how to set up and use Flink with Ibis. After setting up the Flink backend for Ibis in this tutorial, we’ll see a real-life example in A real-life use case: fraud detection.

Set up and connect to Flink

Install the Flink backend for Ibis:

Using pip

pip install ibis-framework apache-flink

Warning

You need to install the Flink backend for Ibis alongside the apache-flink package.

Using conda

conda install -c conda-forge ibis-flink

Using mamba

mamba install -c conda-forge ibis-flink

Using pixi

pixi add ibis-flink

To connect to a Flink session, simply create a pyflink.table.TableEnvironment and pass that to ibis.flink.connect():

from pyflink.table import EnvironmentSettings, TableEnvironment

import ibis

env_settings = EnvironmentSettings.in_streaming_mode()
table_env = TableEnvironment.create(env_settings)
connection = ibis.flink.connect(table_env)

Tip

If you’re working on a batch data pipeline, simply change the TableEnvironment settings to batch mode before connecting to it:

env_settings = EnvironmentSettings.in_batch_mode()

Now you can connect to data sources, create transformations, and write the results into sinks!

Next steps

Now that you’re connected to Flink, you can continue this tutorial to learn the basics of Ibis or query your own data. See the rest of the Ibis documentation or Flink documentation. You can open an issue if you run into one!