Automating Delivery of RSS Feed with Integromat
As practicing scientists and engineers are keenly aware, keeping up with the ever-growing body of technical literature is fundamental to any research endeavor. Many tools exist to help organize and curate literature for consumption, including Twitter Bots, Google Scholar Alerts, and simple RSS feeds. While all of these can be great ways to be notified of any field’s current state of the art, my company has (under)-utilized a Microsoft Teams channel aptly called “Industry News.” After seeing the channel go practically unused for six months, I decided to solve the problem with automation.
When I began researching for this project, I discovered many researchers who have turned to Twitter as a preferred method of curating and disseminating industry news. I stumbled upon the physpaper project by Robert Lanfear on GitHub, which provided excellent documentation on creating a Twitter bot PubMed, arxiv, and just about any other service offering an RSS feed. However, I don’t tweet or use much of any social media – making the dlvrit.com solution that pyspaper suggested unviable.
That was when I discovered integromat.com, a low code automation website that could do (almost) exactly what I wanted. I originally wanted to post directly to Microsoft Teams, however opening the integration with integromat required administrator privileges that I did not have. To overcome this, I instead designed a flow to send me an email every Monday morning with the top five hits for Concentrated Solar Power in the Springer Journals. I easily can copy this email into Teams until administrator privileges can be figured out.
Step 1: Getting the Springer RSS Feed
To connect as RSS feed as the datasource I first dragged the RSS module Integromat’s list of connections: “Retrieve and RSS feed.” Springer offers a convenient RSS feed generator for any search topic: https://link.springer.com/search
After making your query and selecting “newest first” to get the newest entries, click the RSS icon to get a custom feed link that can be copied into the URL field in Integromat’s RSS block. I filled out the rest of the RSS block to look at all articles from the moment and query to a week prior and limit the results to the most recent 5. These parameters can be adjusted however you see fit.
Step 2: Building the Pipeline to Convert Email to Text
One of the most difficult aspects of setting the pipeline up was figuring out what “tool” blocks I would need from the pipeline in order to transform the RSS feed into something readable for email. The simplest way I found was the use a string aggregator in order to produce an array of Titles and URLs from the RSS feed.
I then used an iterator block to iterate through every entry into the array, coupled with a Text Aggregator block to strip the Titles and URLs from the array into HTML formatting. Figuring out that the Text aggregator could accept HTML was the lightbulb for me to figure out how to get this pipeline to work.
Step 3: Connecting the Gmail API to Integromet
Unfortunately, Google’s API integrations make it rather difficult to connect integromet.com to a user account in order to send an email. Essentially, it is necessary to set up a custom OAuth Client. Luckily, integromat has detailed instructions on how to do this. As a note of precaution be careful with the security of this account. I personally recommend setting up a separate dedicated Gmail account for this connection just in case.
After connecting the OAuth client, it was easy to take the text output in the process flow and send it straight to email.
Finally, I ran a quick test to make sure the pipeline was functional and set an automated recurring activation once a week every Monday. Checked my email and got a nifty reading list!
So, what’s next? I’m leaving it as is for now, but I could easily see two great extensions:
- Accumulating more RSS feeds than just Springer to look through many different news and journal sources.
- Adjusting queries to be smarter than just picking the top 5 most recent entries.