GenSim with Radim Řehůřek

Summary Being able to understand the context of a piece of text is generally thought to be the domain of human intelligence. However, topic modeling and semantic analysis can be used to allow a computer to determine whether different messages and articles are about the same thing. This week we spoke with Radim Řehůřek about his work on GenSim, which is a Python library for performing unsupervised analysis of unstructured text and applying machine learning models to the problem of natural language understanding. Brief Introduction Hello and welcome to Podcast.__init__, the podcast about Python and the people who make it great. I would like to thank everyone who has donated to the show. Your contributions help us make the show sustainable. For details on how to support the show you can visit our site at pythonpodcast.com Linode is sponsoring us this week. Check them out at linode.com/podcastinit and get a $20 credit to try out their fast and reliable Linux virtual servers for your next project We are also sponsored by Sentry this week. Stop hoping your users will report bugs. Sentry’s real-time tracking gives you insight into production deployments and information to reproduce and fix crashes. Check them out at getsentry.com and use the code podcastinit at signup to get a $50 credit on your account. Visit our site to subscribe to our show, sign up for our newsletter, read the show notes, and get in touch. To help other people find the show you can leave a review on iTunes, or Google Play Music, and tell your friends and co-workers Join our community! Visit discourse.pythonpodcast.com for your opportunity to find out about upcoming guests, suggest questions, and propose show ideas. Your hosts as usual are Tobias Macey and Chris Patti Today we’re interviewing Radim Řehůřek about Gensim, a library for topic modeling and semantic analysis of natural language. Interview with Radim Řehůřek Introductions How did you get introduced to Python? – Chris Can you start by giving us an explanation of topic modeling and semantic analysis? – Tobias What is Gensim and what inspired you to create it? – Tobias What facilities does Gensim provide to simplify the work of this kind of language analysis? – Tobias Can you describe the features that set it apart from other projects such as the NLTK or Spacy? – Tobias What are some of the practical applications that Gensim can be used for? – Tobias One of the features that stuck out to me is the fact that Gensim can process corpora on disk that would be too large to fit into memory. Can you explain some of the algorithmic work that was necessary to allow for this streaming process to be possible? – Tobias Given that it can handle streams of data, could it also be used in the context of something like Spark? – Tobias Gensim also supports unsupervised model building. What kinds of limitations does this have and when would you need a human in the loop? – Tobias Once a model has been trained, how does it get saved and reloaded for subsequent use? – Tobias What are some of the more unorthodox or interesting uses people have put Gensim to that you’ve heard about? – Chris In addition to your work on Gensim, and partly due to its popularity, you have started a consultancy for customers who are interested in improving their data analysis capabilities. How does that feed back into Gensim? – Tobias Are there any improvements in Gensim or other libraries that you have made available as a result of issues that have come up during client engagements? – Tobias Is it difficult to find contributors to Gensim because of its advanced nature? – Tobias Are there any resources you’d like to recommend our listeners explore to get a more in depth understanding of topic modeling and related techniques? – Chris Keep In Touch RaRe Technologies Twitter Email Github Mailing List Picks Tobias Dark Matter and the Dinosaurs by Lisa Randall Chris m-cli Radim 1177 BC: The Year Civilization Collapsed Links Nadia Eghbal Gensim SQL Addict NLTK Spacy Latent Dirichlet Allocation (LDA) LSI Keynote in Italy on distributed processing Google Scholar references for Gensim Stylometric analysis On Writing Well Student Incubator Wikipedia on topic modeling The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA