PySpark/Spark SQL¶
Install dependencies for Ibis’s Spark dialect:
pip install ibis-framework[spark]
Create a client by passing in the spark session as a parameter to
ibis.spark.connect()
:
con = ibis.spark.connect(spark_session)
API¶
SparkSQL client (Experimental)¶
The Spark SQL client is accessible through the ibis.spark
namespace.
Use ibis.spark.connect
to create a client.
|
Create a SparkClient for use with Ibis. |
|
Create a database object. |
|
List databases in the Spark SQL cluster. |
|
List tables in the current (or indicated) database. |
|
Create a table expression that references a particular table or view in the database. |
PySpark client (Experimental)¶
The PySpark client is accessible through the ibis.pyspark
namespace.
Use ibis.pyspark.connect
to create a client.
|
Create a SparkClient for use with Ibis. |
|
Create a database object. |
|
List databases in the Spark SQL cluster. |
|
List tables in the current (or indicated) database. |
|
Create a table expression that references a particular table or view in the database. |