{
“cells”: [
{

“cell_type”: “markdown”, “metadata”: {}, “source”: [

“# Additional Analytics Tools”

]

}, {

“cell_type”: “markdown”, “metadata”: {}, “source”: [

“## Setup”

]

}, {

“cell_type”: “code”, “execution_count”: null, “metadata”: {}, “outputs”: [], “source”: [

“import ibisn”, “import osn”, “hdfs_port = os.environ.get(‘IBIS_WEBHDFS_PORT’, 50070)n”, “hdfs = ibis.impala.hdfs_connect(host=’impala’, port=hdfs_port)n”, “con = ibis.impala.connect(host=’impala’, database=’ibis_testing’,n”, ” hdfs_client=hdfs)n”, “ibis.options.interactive = True”

]

}, {

“cell_type”: “markdown”, “metadata”: {}, “source”: [

“## Frequency tablesn”, “n”, “Ibis provides the value_counts API, just like pandas, for computing a frequency table for a table column or array expression. You might have seen it used already earlier in the tutorial. “

]

}, {

“cell_type”: “code”, “execution_count”: null, “metadata”: {}, “outputs”: [], “source”: [

“lineitem = con.table(‘tpch_lineitem’)n”, “orders = con.table(‘tpch_orders’)n”, “n”, “items = (orders.join(lineitem, orders.o_orderkey == lineitem.l_orderkey)n”, ” [lineitem, orders])n”, “n”, “items.o_orderpriority.value_counts()”

]

}, {

“cell_type”: “markdown”, “metadata”: {}, “source”: [

“This can be customized, of course:”

]

}, {

“cell_type”: “code”, “execution_count”: null, “metadata”: {}, “outputs”: [], “source”: [

“freq = (items.group_by(items.o_orderpriority)n”, ” .aggregate([items.count().name(‘nrows’),n”, ” items.l_extendedprice.sum().name(‘total $’)]))n”, “freq”

]

}, {

“cell_type”: “markdown”, “metadata”: {}, “source”: [

“## Binning and histogramsn”, “n”, “n”, “Numeric array expressions (columns with numeric type and other array expressions) have bucket and histogram methods which produce different kinds of binning. These produce category values (the computed bins) that can be used in grouping and other analytics.n”, “n”, “Let’s have a look at a few examplesn”, “n”, “I’ll use the summary function to see the general distribution of lineitem prices in the order data joined above:”

]

}, {

“cell_type”: “code”, “execution_count”: null, “metadata”: {}, “outputs”: [], “source”: [

“items.l_extendedprice.summary()”

]

}, {

“cell_type”: “markdown”, “metadata”: {}, “source”: [

“Alright then, now suppose we want to split the item prices up into some buckets of our choosing:”

]

}, {

“cell_type”: “code”, “execution_count”: null, “metadata”: {}, “outputs”: [], “source”: [

“buckets = [0, 5000, 10000, 50000, 100000]”

]

}, {

“cell_type”: “markdown”, “metadata”: {}, “source”: [

“The bucket function creates a bucketed category from the prices:”

]

}, {

“cell_type”: “code”, “execution_count”: null, “metadata”: {}, “outputs”: [], “source”: [

“bucketed = items.l_extendedprice.bucket(buckets).name(‘bucket’)”

]

}, {

“cell_type”: “markdown”, “metadata”: {}, “source”: [

“Let’s have a look at the value counts:”

]

}, {

“cell_type”: “code”, “execution_count”: null, “metadata”: {}, “outputs”: [], “source”: [

“bucketed.value_counts()”

]

}, {

“cell_type”: “markdown”, “metadata”: {}, “source”: [

“The buckets we wrote down define 4 buckets numbered 0 through 3. The NaN is a pandas NULL value (since that’s how pandas represents nulls in numeric arrays), so don’t worry too much about that. Since the bucketing ends at 100000, we see there are 4122 values that are over 100000. These can be included in the bucketing with include_over:”

]

}, {

“cell_type”: “code”, “execution_count”: null, “metadata”: {}, “outputs”: [], “source”: [

“bucketed = (items.l_extendedpricen”, ” .bucket(buckets, include_over=True)n”, ” .name(‘bucket’))n”, “bucketed.value_counts()”

]

}, {

“cell_type”: “markdown”, “metadata”: {}, “source”: [

“The bucketed object here is a special _category_ type”

]

}, {

“cell_type”: “code”, “execution_count”: null, “metadata”: {}, “outputs”: [], “source”: [

“bucketed.type()”

]

}, {

“cell_type”: “markdown”, “metadata”: {}, “source”: [

“Category values can either have a known or unknown _cardinality_. In this case, there’s either 4 or 5 buckets based on how we used the bucket function.n”, “n”, “Labels can be assigned to the buckets at any time using the label function:”

]

}, {

“cell_type”: “code”, “execution_count”: null, “metadata”: {}, “outputs”: [], “source”: [

“bucket_counts = bucketed.value_counts()n”, “n”, “labeled_bucket = (bucket_counts.bucketn”, ” .label([‘0 to 5000’, ‘5000 to 10000’, ‘10000 to 50000’,n”, ” ‘50000 to 100000’, ‘Over 100000’])n”, ” .name(‘bucket_name’))n”, “n”, “expr = (bucket_counts[labeled_bucket, bucket_counts]n”, ” .sort_by(‘bucket’))n”, “expr”

]

}, {

“cell_type”: “markdown”, “metadata”: {}, “source”: [

“Nice, huh?n”, “n”, “histogram is a linear (fixed size bin) equivalent:”

]

}, {

“cell_type”: “code”, “execution_count”: null, “metadata”: {}, “outputs”: [], “source”: [

“t = con.table(‘functional_alltypes’)n”, “n”, “d = t.double_coln”, “n”, “tier = d.histogram(10).name(‘hist_bin’)n”, “expr = (t.group_by(tier)n”, ” .aggregate([d.min(), d.max(), t.count()])n”, ” .sort_by(‘hist_bin’))n”, “expr”

]

}, {

“cell_type”: “markdown”, “metadata”: {}, “source”: [

“## Filtering in aggregationsn”, “n”, “n”, “Suppose that you want to compute an aggregation with a subset of the data for _only one_ of the metrics / aggregates in question, and the complete data set with the other aggregates. Most aggregation functions are thus equipped with a where argument. Let me show it to you in action:”

]

}, {

“cell_type”: “code”, “execution_count”: null, “metadata”: {}, “outputs”: [], “source”: [

“t = con.table(‘functional_alltypes’)n”, “n”, “d = t.double_coln”, “s = t.string_coln”, “n”, “cond = s.isin([‘3’, ‘5’, ‘7’])n”, “n”, “metrics = [t.count().name(‘# rows total’), n”, ” cond.sum().name(‘# selected’),n”, ” d.sum().name(‘total’),n”, ” d.sum(where=cond).name(‘selected total’)]n”, “n”, “color = (t.float_coln”, ” .between(3, 7)n”, ” .ifelse(‘red’, ‘blue’)n”, ” .name(‘color’))n”, “n”, “t.group_by(color).aggregate(metrics)”

]

}

], “metadata”: {

“kernelspec”: {

“display_name”: “Python 3”, “language”: “python”, “name”: “python3”

}, “language_info”: {

“codemirror_mode”: {

“name”: “ipython”, “version”: 3

}, “file_extension”: “.py”, “mimetype”: “text/x-python”, “name”: “python”, “nbconvert_exporter”: “python”, “pygments_lexer”: “ipython3”, “version”: “3.6.3”

}

}, “nbformat”: 4, “nbformat_minor”: 1

}