Advanced Topics: Analytics Tools

Setup

[1]:
import os
import ibis

ibis.options.interactive = True

connection = ibis.sqlite.connect(os.path.join('data', 'geography.db'))

Frequency tables

Ibis provides the value_counts API, just like pandas, for computing a frequency table for a table column or array expression. You might have seen it used already earlier in the tutorial.

[2]:
countries = connection.table('countries')
countries.continent.value_counts()
[2]:
  continent  count
0        AF     58
1        AN      5
2        AS     51
3        EU     54
4        NA     42
5        OC     28
6        SA     14

This can be customized, of course:

[3]:
freq = (countries.group_by(countries.continent)
                 .aggregate([countries.count().name('# countries'),
                             countries.population.sum().name('total population')]))
freq
[3]:
  continent  # countries  total population
0        AF           58        1021238685
1        AN            5               170
2        AS           51        4130584841
3        EU           54         750724554
4        NA           42         540204371
5        OC           28          36067549
6        SA           14         400143568

Binning and histograms

Numeric array expressions (columns with numeric type and other array expressions) have bucket and histogram methods which produce different kinds of binning. These produce category values (the computed bins) that can be used in grouping and other analytics.

Some backends implement the .summary() method, which can be used to see the general distribution of a column.

Let’s have a look at a few examples.

Alright then, now suppose we want to split the countries up into some buckets of our choosing for their population:

[4]:
buckets = [0, 1e6, 1e7, 1e8, 1e9]

The bucket function creates a bucketed category from the prices:

[5]:
bucketed = countries.population.bucket(buckets).name('bucket')

Let’s have a look at the value counts:

[6]:
bucketed.value_counts()
[6]:
  bucket  count
0    NaN      2
1    0.0     93
2    1.0     76
3    2.0     72
4    3.0      9

The buckets we wrote down define 4 buckets numbered 0 through 3. The NaN is a pandas NULL value (since that’s how pandas represents nulls in numeric arrays), so don’t worry too much about that. Since the bucketing ends at 100000, we see there are 4122 values that are over 100000. These can be included in the bucketing with include_over:

[7]:
bucketed = (countries.population
            .bucket(buckets, include_over=True)
            .name('bucket'))
bucketed.value_counts()
[7]:
  bucket  count
0      0     93
1      1     76
2      2     72
3      3      9
4      4      2

The bucketed object here is a special category type

[8]:
bucketed.type()
[8]:
Category(cardinality=5)

Category values can either have a known or unknown cardinality. In this case, there’s either 4 or 5 buckets based on how we used the bucket function.

Labels can be assigned to the buckets at any time using the label function:

[9]:
bucket_counts = bucketed.value_counts()

labeled_bucket = (bucket_counts.bucket
                  .label(['< 1M', '> 1M', '> 10M', '> 100M', '> 1B'])
                  .name('bucket_name'))

expr = (bucket_counts[labeled_bucket, bucket_counts]
        .sort_by('bucket'))
expr
[9]:
  bucket_name bucket  count
0        < 1M      0     93
1        > 1M      1     76
2       > 10M      2     72
3      > 100M      3      9
4        > 1B      4      2

Nice, huh?

Some backends implement histogram(num_bins), a linear (fixed size bin) equivalent.