PyData Seattle 2023

Background

Altair [1] is a popular data visualization library for Python that makes it easy to build a wide variety of statistical charts with just a few lines of code. By integrating data transformations, like binning and aggregation, into chart specifications Altair often eliminates the need to perform external data pre-processing in pandas. Altair also supports a powerful selections framework [2] that enables the creation of advanced interactive charts.

This talk will include a brief overview of Altair, with several examples of the kinds of static and interactive visualizations it can be used to create.

Problem

While Altair provides many benefits, a significant limitation is that it requires transferring the entire dataset to the browser for data processing and rendering. Consider a histogram of a one million row dataset. While it is convenient for the Altair chart specification to include the data transforms necessary to perform the histogram binning and aggregation, it's not practical to serialize one million rows to JSON and transfer them from the Python kernel to the browser. To protect against crashing the browser, Altair imposes a default limit of five thousand rows, raising an error when this limit is exceeded.

Solution

VegaFusion [3] integrates with Altair to automatically extract data transformations from chart specifications. The extracted transformations are then evaluated by efficient multi-threaded Rust implementations in the Python kernel. The transformed data is then sent to the browser, along with the modified chart specification, for rendering.

As an example, in the case of a twenty bin histogram of a one million row dataset, this approach results in sending a twenty row dataset to the browser rather than the one million row dataset that plain Altair would attempt to send.

Features

VegaFusion provides three main features for Altair users: A mime renderer, a widget renderer, and a function to extract transformed data from Altair charts.

Mime renderer

The VegaFusion mime renderer [4] pre-evaluates the data transformations associated with a chart, inlining the transformed data back into the chart specification. This approach passes the modified chart specification for display in the browser exactly the same way that plain Altair does. The mime renderer is compatible with every jupyter-based environment that Altair already works in, and it's a great choice for non-interactive charts.

Widget Renderer

The VegaFusion widget renderer [5] includes a custom Jupyter widget extension [6] that maintains a live connection between the browser and the Python kernel. This makes it possible to support interactive charts that transform data in response to interactive selection events. For example, the widget renderer enables linked histogram brushing across subplots for datasets with over one million rows.

Extract Transformed Data

In plain Altair, data transformations are evaluated in the browser, and so it's not possible to access the transformed data from Python. VegaFusion adds the ability to access transformed datasets as pandas DataFrames [7], making it possible to use Altair transforms as part of larger data analysis pipelines without needing to reimplement these data transformations in Python.

Demo

VegaFusion renderers can be enabled for any Altair visualization with the addition of two lines of code. The presentation will include a live demonstration of the interactive exploration of a dataset with over a million rows.

License

As of version 1.0, VegaFusion is available under the same license as Altair: BSD-3.

Expected Experience

To get the most out of this talk, some experience with data visualization in Python will be helpful, but experience with Altair is not required.

References

• [1] https://altair-viz.github.io
• [2] https://altair-viz.github.io/user_guide/interactions.html
• [3] https://vegafusion.io/
• [4] https://vegafusion.io/mime_renderer.html
• [5] https://vegafusion.io/widget_renderer.html
• [6] https://ipywidgets.readthedocs.io/
• [7] https://vegafusion.io/transformed_data.html