Add Anomaly Detection To Your Time Series Data With Luminaire

Summary When working with data it’s important to understand when it is correct. If there is a time dimension, then it can be difficult to know when variation is normal. Anomaly detection is a useful tool to address these challenges, but a difficult one to do well. In this episode Smit Shah and Sayan Chakraborty share the work they have done on Luminaire to make anomaly detection easier to work with. They explain the complexities inherent to working with time series data, the strategies that they have incorporated into Luminaire, and how they are using it in their data pipelines to identify errors early. If you are working with any kind of time series then it’s worth giving Luminaure a look. Announcements Hello and welcome to Podcast.__init__, the podcast about Python and the people who make it great. When you’re ready to launch your next app or want to try a project you hear about on the show, you’ll need somewhere to deploy it, so take a look at our friends over at Linode. With the launch of their managed Kubernetes platform it’s easy to get started with the next generation of deployment and scaling, powered by the battle tested Linode platform, including simple pricing, node balancers, 40Gbit networking, dedicated CPU and GPU instances, and worldwide data centers. Go to pythonpodcast.com/linode and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show! Python has become the default language for working with data, whether as a data scientist, data engineer, data analyst, or machine learning engineer. Springboard has launched their School of Data to help you get a career in the field through a comprehensive set of programs that are 100% online and tailored to fit your busy schedule. With a network of expert mentors who are available to coach you during weekly 1:1 video calls, a tuition-back guarantee that means you don’t pay until you get a job, resume preparation, and interview assistance there’s no reason to wait. Springboard is offering up to 20 scholarships of $500 towards the tuition cost, exclusively to listeners of this show. Go to pythonpodcast.com/springboard today to learn more and give your career a boost to the next level. Your host as usual is Tobias Macey and today I’m interviewing Smit Shah and Sayan Chakraborty about Luminaire, a machine learning based package for anomaly detection on timeseries data Interview Introductions How did you get introduced to Python? Can you start by describing what Luminaire is and how the project got started? Where does the name come from? How does Luminaire compare to other frameworks for working with timeseries data such as Prophet? What are the main use cases that Luminaire is powering at Zillow? What are some of the complexities inherent to anomaly detection that are non-obvious at first glance? How are you addressing those challenges in Luminaire? Can you describe how Luminaire is implemented? How has the design of the project evolved since it was first started? What was the motivation for releasing Luminaire as open source? For someone who is using Luminaire, what is the process for training and deploying a model with it? What are some common ways that it is used within a larger system? How do sustained anomalies such as the current pandemic affect the work of identifying other sources of meaningful outliers? What are some of the most interesting, innovative, or unexpected ways that you have seen Luminaire being used? What are some of the most interesting, unexpected, or challening lessons that you have learned while building and using Luminaire? When is Luminaire the wrong choice? What do you have planned for the future of the project? Keep In Touch Smit LinkedIn shahsmit14 on GitHub Sayan LinkedIn Website @tweettosayan on Twitter Picks Tobias Flakehell Smit Apache Ranger Sayan Prediction Machines: The Simple Economics Of Artificial Intelligence Closing Announcements Thank you for listening! Don’t forget to check out our other show, the Data Engineering Podcast for the latest on modern data management. Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes. If you’ve learned something or tried out a project from the show then tell us about it! Email hosts@podcastinit.com) with your story. To help other people find the show please leave a review on iTunes and tell your friends and co-workers Join the community in the new Zulip chat workspace at pythonpodcast.com/chat Links Luminaire Zillow Anomaly Detection Facebook Prophet IEEE Big Data Conference Unsupervised Learning ARIMA (Autoregressive Integrated Moving Average) Model Airflow The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA