Ibis sneak peek: writing to files

blog
io
new feature
sneak peek
Author

Kae Suarez

Published

March 9, 2023

Ibis 5.0 is coming soon and will offer new functionality and fixes to users. To enhance clarity around this process, we’re sharing a sneak peek into what we’re working on.

In Ibis 4.0, we added the ability to read CSVs and Parquet via the Ibis interface. We felt this was important because, well, the ability to read files is simply necessary, be it on a local scale, legacy data, data not yet in a database, and so on. However, for a user, the natural next question was “can I go ahead and write when I’m done?” The answer was no. We didn’t like that, especially since we do care about file-based use cases.

So, we’ve gone ahead and fixed that for Ibis 5.0.

Files in, Files out

Before we can write a file, we need data — so let’s read in a file, to start this off:

import ibis

ibis.options.interactive = True

t = ibis.read_csv(
    "https://storage.googleapis.com/ibis-examples/data/penguins.csv.gz"
)
t
┏━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━┓
┃ species ┃ island    ┃ bill_length_mm ┃ bill_depth_mm ┃ flipper_length_mm ┃ body_mass_g ┃ sex    ┃ year  ┃
┡━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━┩
│ string  │ string    │ float64        │ float64       │ int64             │ int64       │ string │ int64 │
├─────────┼───────────┼────────────────┼───────────────┼───────────────────┼─────────────┼────────┼───────┤
│ Adelie  │ Torgersen │           39.1 │          18.7 │               181 │        3750 │ male   │  2007 │
│ Adelie  │ Torgersen │           39.5 │          17.4 │               186 │        3800 │ female │  2007 │
│ Adelie  │ Torgersen │           40.3 │          18.0 │               195 │        3250 │ female │  2007 │
│ Adelie  │ Torgersen │            nan │           nan │              NULL │        NULL │ NULL   │  2007 │
│ Adelie  │ Torgersen │           36.7 │          19.3 │               193 │        3450 │ female │  2007 │
│ Adelie  │ Torgersen │           39.3 │          20.6 │               190 │        3650 │ male   │  2007 │
│ Adelie  │ Torgersen │           38.9 │          17.8 │               181 │        3625 │ female │  2007 │
│ Adelie  │ Torgersen │           39.2 │          19.6 │               195 │        4675 │ male   │  2007 │
│ Adelie  │ Torgersen │           34.1 │          18.1 │               193 │        3475 │ NULL   │  2007 │
│ Adelie  │ Torgersen │           42.0 │          20.2 │               190 │        4250 │ NULL   │  2007 │
│ …       │ …         │              … │             … │                 … │           … │ …      │     … │
└─────────┴───────────┴────────────────┴───────────────┴───────────────────┴─────────────┴────────┴───────┘

Of course, we could just write out, but let’s do an operation first — how about using selectors, which you can read more about here? Self-promotion aside, here’s an operation:

from ibis import _
import ibis.selectors as s

expr = (
    t.group_by("species")
     .mutate(s.across(s.numeric() & ~s.c("year"), (_ - _.mean()) / _.std()))
)
expr
┏━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━┓
┃ species ┃ island    ┃ bill_length_mm ┃ bill_depth_mm ┃ flipper_length_mm ┃ body_mass_g ┃ sex    ┃ year  ┃
┡━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━┩
│ string  │ string    │ float64        │ float64       │ float64           │ float64     │ string │ int64 │
├─────────┼───────────┼────────────────┼───────────────┼───────────────────┼─────────────┼────────┼───────┤
│ Adelie  │ Biscoe    │      -1.423513 │     -0.366874 │          0.312925 │    0.053074 │ female │  2009 │
│ Adelie  │ Torgersen │       0.979426 │      0.126283 │          1.842104 │    0.380180 │ male   │  2009 │
│ Adelie  │ Torgersen │       1.542615 │     -0.613453 │          0.924596 │    2.179266 │ male   │  2008 │
│ Adelie  │ Biscoe    │       0.641513 │     -0.366874 │         -0.451665 │   -1.091799 │ female │  2007 │
│ Adelie  │ Biscoe    │      -0.222043 │      1.359177 │          0.007089 │    0.434698 │ male   │  2009 │
│ Adelie  │ Torgersen │       1.392432 │      1.934527 │          1.077514 │    1.743124 │ male   │  2007 │
│ Adelie  │ Torgersen │       1.129610 │      0.866019 │          1.230432 │    1.634089 │ male   │  2008 │
│ Adelie  │ Dream     │      -0.747686 │      0.126283 │          0.465843 │   -0.437586 │ female │  2009 │
│ Adelie  │ Dream     │      -0.860324 │     -0.284681 │         -1.216254 │   -1.200835 │ female │  2007 │
│ Adelie  │ Dream     │       0.754151 │      0.044090 │          0.771678 │    0.434698 │ male   │  2007 │
│ …       │ …         │              … │             … │                 … │           … │ …      │     … │
└─────────┴───────────┴────────────────┴───────────────┴───────────────────┴─────────────┴────────┴───────┘

Now, finally, time to do the exciting part:

expr.to_parquet("normalized.parquet")

Like many things in Ibis, this is as simple and plain-looking as it is important. Being able to create files from Ibis instead of redirecting into other libraries first enables operation at larger scales and fewer steps. Where desired, you can address a backend directly to use its native export functionality — we want to make sure you have the flexibility to use Ibis or the backend as you see fit.

Wrapping Up

Ibis is an interface tool for analytical engines that can reach scales far beyond a laptop. Files are important to Ibis because:

  • Ibis also supports local execution, where files are the standard unit of data — we want to support all our users.
  • Files are useful for moving between platforms, and long-term storage that isn’t tied to a particular backend.
  • Files can move more easily between our backends than database files, so we think this adds some convenience for the multi-backend use case.

We’re excited to release this functionality in Ibis 5.0.

Interested in Ibis? Docs are available on this very website, at:

and the repo is always at:

Please feel free to reach out on GitHub!

Back to top