from pyflink.table import EnvironmentSettings, TableEnvironment
import ibis
= EnvironmentSettings.in_streaming_mode()
env_settings = TableEnvironment.create(env_settings)
table_env = ibis.flink.connect(table_env) connection
Getting started
In this tutorial, you will learn how to set up and use Flink with Ibis. After setting up the Flink backend for Ibis in this tutorial, we’ll see a real-life example in A real-life use case: fraud detection.
Set up and connect to Flink
Install the Flink backend for Ibis:
pip install ibis-framework apache-flink
You need to install the Flink backend for Ibis alongside the apache-flink
package.
conda install -c conda-forge ibis-flink
mamba install -c conda-forge ibis-flink
pixi add ibis-flink
To connect to a Flink session, simply create a pyflink.table.TableEnvironment
and pass that to ibis.flink.connect()
:
If you’re working on a batch data pipeline, simply change the TableEnvironment settings to batch mode before connecting to it:
= EnvironmentSettings.in_batch_mode() env_settings
Now you can connect to data sources, create transformations, and write the results into sinks!
Next steps
Now that you’re connected to Flink, you can continue this tutorial to learn the basics of Ibis or query your own data. See the rest of the Ibis documentation or Flink documentation. You can open an issue if you run into one!