PySpark

Install dependencies for Ibis’s PySpark dialect:

pip install 'ibis-framework[pyspark]'

Note

When using the PySpark backend with PySpark 2.4.x and pyarrow >= 0.15.0, you need to set ARROW_PRE_0_15_IPC_FORMAT=1. See here for details

PySpark client

The PySpark client is accessible through the ibis.pyspark namespace.

Use ibis.pyspark.connect to create a client.

Backend.connect(session)

Create a SparkClient for use with Ibis.

PySparkClient.database([name])

PySparkClient.list_databases([like])

PySparkClient.list_tables([like, database])

PySparkClient.table(name[, database])

Create a table expression that references a particular table or view in the database.