PySpark/Spark SQL

Install dependencies for Ibis’s Spark dialect:

pip install ibis-framework[spark]

Create a client by passing in the spark session as a parameter to ibis.spark.connect():

con = ibis.spark.connect(spark_session)

API

SparkSQL client (Experimental)

The Spark SQL client is accessible through the ibis.spark namespace.

Use ibis.spark.connect to create a client.

connect(spark_session)

Create a SparkClient for use with Ibis.

SparkClient.database([name])

Create a database object.

SparkClient.list_databases([like])

List databases in the Spark SQL cluster.

SparkClient.list_tables([like, database])

List tables in the current (or indicated) database.

SparkClient.table(name[, database])

Create a table expression that references a particular table or view in the database.

PySpark client (Experimental)

The PySpark client is accessible through the ibis.pyspark namespace.

Use ibis.pyspark.connect to create a client.

connect(session)

Create a SparkClient for use with Ibis.

PySparkClient.database([name])

Create a database object.

PySparkClient.list_databases([like])

List databases in the Spark SQL cluster.

PySparkClient.list_tables([like, database])

List tables in the current (or indicated) database.

PySparkClient.table(name[, database])

Create a table expression that references a particular table or view in the database.