Jun Liu
Jun Liu is the science tech lead of Lyft Rider App, focusing on developing large-scale machine learning solutions for recommendations and purchasing. Prior to joining Lyft, Jun received her Ph.D. in Applied Mathematics from Michigan State University.
Sessions
Using Spark, Dask, or Ray is not an all-or-nothing thing. It may seem daunting for new practitioners expecting to translate existing Pandas pipelines to these big data frameworks. In reality, distributed computing can be incrementally adopted. There are many use cases where only one or two steps of a pipeline require expensive computation. This talk covers the strategies and best practices around moving portions of workloads to distributed computing through the open-source Fugue project. The Fugue API has a suite of standalone functions compatible with Pandas, Spark, Dask, and Ray. Collectively, these functions allow users to scale any part of their pipeline when ready for full-scale production workloads on big data.