Bringing AI Into The Inner Loop of Data Engineering With Ascend

Summary In this episode of the Data Engineering Podcast Sean Knapp, CEO of Ascend.io, explores the intersection of AI and data engineering. He discusses the evolution of data engineering and the role of AI in automating processes, alleviating burdens on data engineers, and enabling them to focus on complex tasks and innovation. The conversation covers the challenges and opportunities presented by AI, including the need for intelligent tooling and its potential to streamline data engineering processes. Sean and Tobias also delve into the impact of generative AI on data engineering, highlighting its ability to accelerate development, improve governance, and enhance productivity, while also noting the current limitations and future potential of AI in the field. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Data migrations are brutal. They drag on for months—sometimes years—burning through resources and crushing team morale. Datafold's AI-powered Migration Agent changes all that. Their unique combination of AI code translation and automated data validation has helped companies complete migrations up to 10 times faster than manual approaches. And they're so confident in their solution, they'll actually guarantee your timeline in writing. Ready to turn your year-long migration into weeks? Visit dataengineeringpodcast.com/datafold today for the details. Your host is Tobias Macey and today I'm interviewing Sean Knapp about how Ascend is incorporating AI into their platform to help you keep up with the rapid rate of change Interview Introduction How did you get involved in the area of data management? Can you describe what Ascend is and the story behind it? The last time we spoke was August of 2022. What are the most notable or interesting evolutions in your platform since then? In that same time "AI" has taken up all of the oxygen in the data ecosystem. How has that impacted the ways that you and your customers think about their priorities? The introduction of AI as an API has caused many organizations to try and leap-frog their data maturity journey and jump straight to building with advanced capabilities. How is that impacting the pressures and priorities felt by data teams? At the same time that AI-focused product goals are straining data teams capacities, AI also has the potential to act as an accelerator to their work. What are the roadblocks/speedbumps that are in the way of that capability? Many data teams are incorporating AI tools into parts of their workflow, but it can be clunky and cumbersome. How are you thinking about the fundamental changes in how your platform works with AI at its center? Can you describe the technical architecture that you have evolved toward that allows for AI to drive the experience rather than being a bolt-on? What are the concrete impacts that these new capabilities have on teams who are using Ascend? What are the most interesting, innovative, or unexpected ways that you have seen Ascend + AI used? What are the most interesting, unexpected, or challenging lessons that you have learned while working on incorporating AI into the core of Ascend? When is Ascend the wrong choice? What do you have planned for the future of AI in Ascend? Contact Info LinkedIn Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? Closing Announcements Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. The AI Engineering Podcast is your guide to the fast-moving world of building AI systems. Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes. If you've learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com with your story. Links Ascend Cursor AI Code Editor Devin GitHub Copilot OpenAI DeepResearch S3 Tables AWS Glue AWS Bedrock Snowpark Co-Intelligence: Living and Working with AI by Ethan Mollick (affiliate link) OpenAI o3 The intro and outro music is from The Hug by The Freak Fandango Orchestra / CC BY-SA