Making Complex Software Fun And Flexible With Plugin Oriented Programming

Summary Starting a new project is always exciting because the scope is easy to understand and adding new features is fun and easy. As it grows, the rate of change slows down and the amount of communication necessary to introduce new engineers to the code increases along with the complexity. Thomas Hatch, CTO and creator of SaltStack, didn’t want to accept that as an inevitable fact of software, so he created a new paradigm and a proof-of-concept framework to experiment with it. In this episode he shares his thoughts and findings on the topic of plugin oriented programming as a way to build and scale complex projects while keeping them fun and flexible. Announcements Hello and welcome to Podcast.__init__, the podcast about Python and the people who make it great. When you’re ready to launch your next app or want to try a project you hear about on the show, you’ll need somewhere to deploy it, so take a look at our friends over at Linode. With 200 Gbit/s private networking, scalable shared block storage, node balancers, and a 40 Gbit/s public network, all controlled by a brand new API you’ve got everything you need to scale up. And for your tasks that need fast computation, such as training machine learning models, they just launched dedicated CPU instances. Go to pythonpodcast.com/linode to get a $20 credit and launch a new server in under a minute. And don’t forget to thank them for their continued support of this show! You listen to this show to learn and stay up to date with the ways that Python is being used, including the latest in machine learning and data analysis. For even more opportunities to meet, listen, and learn from your peers you don’t want to miss out on this year’s conference season. We have partnered with organizations such as O’Reilly Media, Dataversity, Corinium Global Intelligence, Alluxio, and Data Council. Upcoming events include the combined events of the Data Architecture Summit and Graphorum, the Data Orchestration Summit, and Data Council in NYC. Go to pythonpodcast.com/conferences to learn more about these and other events, and take advantage of our partner discounts to save money when you register today. Your host as usual is Tobias Macey and today I’m interviewing Thomas Hatch about his work on the POP library and how he is using plugin oriented programming in his work at SaltStack Interview Introductions How did you get introduced to Python? Can you start by giving your definition of Plugin Oriented Programming and your thoughts on what benefits it provides? You created the POP library as a framework for enabling developers to incorporate this pattern into their own projects. What capabilities does that framework provide and what was your motivation for creating it? How has your work on Salt influenced your thinking on how to implement plugins for software projects? How does POP fit into the future of the SaltStack project? What are some of the advanced patterns or paradigms that the POP model allows for? Can you describe how the POP library itself is implemented and some of the ways that its design has evolved since you first began experimenting with it? What are some of the languages or libraries that you have looked at for inspiration in your design and philosophy around this development pattern? For someone who is building a project on top of POP what does their workflow look like and what are some of the up-front design considerations they should be thinking of? How do you define and validate the contract exposed by or expected from a plugin subsystem? One of the interesting capabilities that you highlight in the documentation is the concept of merging applications. What are your thoughts on the challenges that an engineer might face when merging library or microservice applications built with POP into a single deployable artifact? What would be involved in going the other direction to split a single application into independently runnable microservices? When extracting common functionality from a group of existing applications, what are the relative merits of creating a plugin sybsystem vs writing a library? How does the system design of a POP application impact the available range of communication patterns for software and the teams building it? What are some antipatterns that you anticipate for teams building their projects on top of POP? In the documentation you mention that POP is just an example implementation of the broader pattern and that you hope to see other languages and developer communities adopt it. What are some of the barriers to adoption that you foresee? What are some of the limitations of POP or cases where you would recommend against following this paradigm? What are some of the most interesting, innovative, or unexpected ways that you have seen POP used? What have been some of the most interesting, unexpected, or challenging aspects of building POP? What do you have planned for the future of the POP library, or any applications where you plan to employ this pattern? Keep In Touch thatch45 on GitHub @thatch45 on Twitter Picks Tobias The Man In The High Castle TV series Thomas Jack Ryan TV Series Closing Announcements Thank you for listening! Don’t forget to check out our other show, the Data Engineering Podcast for the latest on modern data management. Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes. If you’ve learned something or tried out a project from the show then tell us about it! Email hosts@podcastinit.com) with your story. To help other people find the show please leave a review on iTunes and tell your friends and co-workers Join the community in the new Zulip chat workspace at pythonpodcast.com/chat Links Episode 1 POP SaltStack Ruby Microservices Linus Torvalds SaltConf SaltStack Thorium Salt Beacons Salt Reactors Salt Grains Idem AsyncIO Nim OCaml Julia LLVM Object Oriented Programming Go Language Rust RBAC == Role Based Access Control The Mythical Man Month Linux Kernel Heist Umbra Flow Programming Magic The Gathering The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA