Building Big Data Pipelines For Audio With Klio

Summary Technologies for building data pipelines have been around for decades, with many mature options for a variety of workloads. However, most of those tools are focused on processing of text based data, both structured and unstructured. For projects that need to manage large numbers of binary and audio files the list of options is much shorter. In this episode Lynn Root shares the work that she and her team at Spotify have done on the Klio project to make that list a bit longer. She discusses the problems that are specific to working with binary data, how the Klio project is architected to allow for scalable and efficient processing of massive numbers of audio files, why it was released as open source, and how you can start using it today for your own projects. If you are struggling with ad-hoc infrastructure and a medley of tools that have been cobbled together for analyzing large or numerous binary assets then this is definitely a tool worth testing out. Announcements Hello and welcome to Podcast.__init__, the podcast about Python and the people who make it great. When you’re ready to launch your next app or want to try a project you hear about on the show, you’ll need somewhere to deploy it, so take a look at our friends over at Linode. With the launch of their managed Kubernetes platform it’s easy to get started with the next generation of deployment and scaling, powered by the battle tested Linode platform, including simple pricing, node balancers, 40Gbit networking, dedicated CPU and GPU instances, and worldwide data centers. Go to pythonpodcast.com/linode and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show! Do you want to get better at Python? Now is an excellent time to take an online course. Whether you’re just learning Python or you’re looking for deep dives on topics like APIs, memory mangement, async and await, and more, our friends at Talk Python Training have a top-notch course for you. If you’re just getting started, be sure to check out the Python for Absolute Beginners course. It’s like the first year of computer science that you never took compressed into 10 fun hours of Python coding and problem solving. Go to pythonpodcast.com/talkpython today and get 10% off the course that will help you find your next level. That’s pythonpodcast.com/talkpython, and don’t forget to thank them for supporting the show. Python has become the default language for working with data, whether as a data scientist, data engineer, data analyst, or machine learning engineer. Springboard has launched their School of Data to help you get a career in the field through a comprehensive set of programs that are 100% online and tailored to fit your busy schedule. With a network of expert mentors who are available to coach you during weekly 1:1 video calls, a tuition-back guarantee that means you don’t pay until you get a job, resume preparation, and interview assistance there’s no reason to wait. Springboard is offering up to 20 scholarships of $500 towards the tuition cost, exclusively to listeners of this show. Go to pythonpodcast.com/springboard today to learn more and give your career a boost to the next level. Your host as usual is Tobias Macey and today I’m interviewing Lynn Root about Klio, an open source pipeline for processing audio and binary data Interview Introductions How did you get introduced to Python? Can you start by describing what Klio is and how it got started? What are some of the challenges that are unique to processing audio data as compared to text? What use cases does Klio enable? What are some of the alternative options available for working with binary data? What capabilities were lacking in other solutions that made it worthwhile to build a new system from scratch? Can you describe the design and architecture of Klio? What was the motivation for implementing Klio as a Python framework, rather than building on top of the Scio project? How much of a challenge has it been to interface to the Beam framework from Python? (Java <-> Python impedance mismatch) One of the interesting optimizations in Klio is the option for bottom up execution of a job to avoid processing a given file unless absolutely necessary. What are some of the other useful or interesting capabilities that are built into Klio? What was the motivation and process for releasing Klio as open source? For someone who is building a pipeline with Klio, can you talk through the workflow? What are the extension and integration points that are exposed? How does Klio handle third party dependencies for a given job? What are some of the challenges, misunderstandings, or edge cases that users of Klio should be aware of? What are some of the most interesting, unexpected, or challenging lessons that you have learned while building and growing the Klio project? What are some of the most interesting, innovative, or unexpected ways that you have seen Klio used? What do you have planned for the future of the project? Keep In Touch GitHub Twitter LinkedIn Picks Tobias PSF Fundraiser Lynn Roam note-taking tool Closing Announcements Thank you for listening! Don’t forget to check out our other show, the Data Engineering Podcast for the latest on modern data management. Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes. If you’ve learned something or tried out a project from the show then tell us about it! Email hosts@podcastinit.com) with your story. To help other people find the show please leave a review on iTunes and tell your friends and co-workers Join the community in the new Zulip chat workspace at pythonpodcast.com/chat Links Klio Announcement Blog Post Docs GitHub Spotify PyLadies SF Luigi RAML ramlfications Interrogate Apache Beam Librosa PyAudio Pillow Podcast Episode FFMPeg ImageMagick Music Information Retrieval Machine Hearing Data Engineering Podcast Episode Scio Microsoft Azure Google Cloud Platform Google Cloud Dataflow Protocol Buffers Apache Spark PySpark DAG == Directed Acyclic Graph ISMIR Conference Digital Signal Processing (DSP) Python Pickle Research paper on separating vocals from instrumentals of a song New York Times: Why songs of the summer sound the same Microsoft’s Rocket Platform for video analytics The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA