Any Yahoo Pipes true substitute out there ?
Tools for RSS feed merging and filtering
The loss of Yahoo Pipes in August 2015 [1] was a major blow to the monitoring/research community and also web site authors looking for selected content. Its ability to combine and filter multiple RSS feeds into one feed was unique among free hosted services.
This post will focus on how we could replace Yahoo Pipes for these functions [2]. We will not deal here with the web page scraping Pipes was able to do too [3].
So I went looking for substitutes which could merge and filter feeds. I mean, real ones : as simple as Pipes. For instance, they shall have a graphic user interface (GUI) ; coding shouldn’t be necessary. The only difficulty allowed should be the use of regular expressions (Regex).
Free hosted substitutes
- FeedCombine, RSSMix, FeedRinse or ChimpFeedr (to merge feeds into one feed) then FeedSifter or FeedRinse (to filter the mega-feed) can answer simple needs.
FeedRinse can do more, as it accepts Regex but it can’t merge more than 5 feeds, is slow and bugs according to La Bibliothèque du CHUM’s test.
FeedCombine cannot combine more than 5 feeds, say Anik Dumont-Bissonnette.
ChimpFeedr is very simple at "chomping" (as their web site says) feeds.
RSSMix, according to the same test, works perfectly, merging up to 100 feeds - Feed Informer combines and filters feeds, but filtering can only be done using AND or OR operators
- IFTTT doesn’t seem to do exactly that : merging and filtering RSS feeds into one feed. Which is not to say it does not have its (numerous) advantages in the field of monitoring services.
Paid hosted substitutes
- FeedsAPI does not seem to merge RSS feeds but can filter them. Afterwards you can use an IFTTT recipe such as Feedly Category to Twitter. FeedsAPI and Inoreader seem the easiest to use solutions among the paid ones
- Tadaweb from Luxembourg, according to some blog posts, is able to create from web pages, filter and organize RSS feeds. The ouput is in RSS format only if the input is RSS too. NB : following a recent new policy, Tadaweb does no longer give access to its Creator module, its team is doing the work in lieu and place of the information professional and its pricing has accordingly evolved
- the paid version of dlvr.it can filter and mash up RSS feeds
- We Wired Web offers an IFTTT-like interface, RSS in and out and is able to filter it. But it does not merge different RSS feeds
- the VIP and Premium versions of the Netvibes RSS portal are able to merge and filter RSS feeds by keywords, using Universal Search (which allows to query only certain groups of feeds — they’re called "tabs") and then Tracked Topics with RSS export
- the Plus and Pro version of the RSS reader Inoreader [4] allow merging and filter features throught the use of "active searches" : create a folder of feeds then an "active search" (an automated keyword search) on it and finally right click on the active search and click "Get RSS feed" [5]. It’s a pity Inoreader filter rules does not seem to offer RSS feeds as an outcome ; it would be interesting because those filter rules are much more precise than actives searches since they accept complex boolean queries and Regex [6] [7].
Self-hosted solutions (need coding skills)
In the sources listed below, I found other free or paid total or partial replacements — but you have to host them on your own server and/or they’re far too complicated for the average information professional :
- Tiny Tiny RSS (TT-RSS) is an RSS reader with very interesting RSS feeds filtering features (open source)
- Huginn by cantino (open source on GitHub). According to a MakeUseOf article, « this is the most popular of all the GitHub solutions mentioned, and has a huge, active support community behind it. It’s well documented and easy to use »
- WebHookit, based on Node.js and mongoDB (open source)
- Pypes (open source on GitHub), a Python application
- Rss Percolator (open source on GitHub). Downloads, aggregates and filters RSS feeds
- ClickScripts (open source on GitHub)
- Bipio (open source). See the Getting started page (developer documentation) for a quick presentation. Bipio is a graph pipelining API talking RESTful JSON
- Neddick (open source on GitHub).
- pipe2py (open source on GitHub). You need to backup your YP pipes before. It uses JSON and the Google App Engine
- Pipes2js (open source on GitHub)
- Superpipes (open source), to be hosted on a Debian server
- open source edition of MuleSoft’s Anypoint Studio
- Quadrigram seems too graphics-oriented to me (apparently free).
NB : the first four services are recommended [8] by Serge Courrier, a French consultant specialising in RSS readers and the use of RSS feeds for monitoring web sites [9].
Walled gardens
RSS is important to information professionals and, as we’ve seen, much used by people whose job is to monitor web resources. But this does not prevent RSS implementation from receding. If you look into the list of applications which can be connected by IFTTT, you will find that RSS is just one among more than a hundred of mostly proprietary ones ... Walled gardens again.
Did I miss something ? Do you know about any simple tool that could replace the "RSS mashup" function of Yahoo Pipes ? I’m interested in free as well as paid for services.
Emmanuel Barthe
law librarian researcher, open law specialist and monitoring solutions specialist
Sources for this article
Here are the links I found to some articles/blog posts and a number of forum discussions about Yahoo Pipes substitutes :
- Par quoi remplacer Yahoo Pipes ? : Grille d’évaluation d’outils pouvant fusionner et filtrer des fils RSS (in French) / Anik Dumont-Bissonnette, La Bibliothèque du CHUM 22 June 2015
- Modèle d’une plateforme de veille visant à automatiser la rediffusion de l’information (in French) / Serge Courrier, November 2016. See especially pp. 20-21 and 35-37
- The 13 Best Yahoo Pipes Alternatives You Should Look at Today / Angela Alcom, MakeUseOf 8 June 2015
- Yahoo Pipes s’arrête… Par quoi le remplacer ? (in French) / Korben, 5 June 2015
- TaDaweb, un outil de veille innovant et aux fonctionnalités impressionnantes (in French) / Christophe Deschamps, Outils Froids 28 February 2013
- Need a Yahoo ! Pipes Alternative ? Your Search Is Over / Bill Flitter, dlvr.it blog 11 June 2015
- Does Netvibes provide exportable RSS ?, Netvibes Knowledge base
- How-To : Universal Search for VIP, Netvibes blog 3 April 2014
- Inoreader, enfin le digne successeur de Google Reader ? (in French) / Corinne Habarou, UrfistInfo 24 November 2014. Do read Serge Courrier’s comment
- Yahoo Pipes End-Of-life Announcement (forum thread), Hacker News 6 June 2015
- Yahoo Pipes : Any good alternatives out there ? (forum thread), Yahoo Developer network 4-7 June 2015
- Is there a decent alternative to Yahoo Pipes (forum thread), Stack Overflow 13 April 2010-7 July 2011
- Is there any good alternative to Yahoo ! Pipes ? (forum thread), Quora 6 May 2012-...
- What paid/commercial alternatives to Yahoo Pipes exist in the market today ? (forum thread), Quora February 2011
- Alternatives to Yahoo Pipes ? (forum thread), Buzztouch 6 April 2014
- Alternatives to Yahoo ! Pipes, Meta-Guide.com (bot created directory) (regularly updated)
- What are some open-source alternatives to Zapier ? (forum thread), Quora 8 October 2013-9 August 2014
- 5 Intuitive Solutions for Integrating Cloud Apps, 5 August 2014
- Bringing ETL to the Masses with APIs, API Evangelist 10 February 2013.
This article was written with the the help of Serge Courrier.
Notes
[1] Starting end of August 2015, one was no longer able to create new pipes. At the end of September 2015, the service closed.
[2] Yahoo Pipes could do much more but we will skip that here.
[3] To easily scrape web pages that do not have an RSS feed and so create their RSS feed, use paid services such as the simple, hosted and often recommended Feedity and FeedsAPI, the hosted but complex Feed43 — you need to master regular expresssions (Regex) —, or the self-hosted Feed Creator by FiveFilters.org.
[4] Inoreader is developped by a team of Bulgarians under the name of Innologica Ltd directed by its co-founder Yordan Yordanov.
[5] In the Plus version, active searches are limited to 20. The Pro version makes them illimited. Thanks to Marjolein Hoekstra (CleverClogs) for pointing me to this Inoreader feature. See her comments here under.
[6] Les filtres RSS dans Inoreader : détail de la syntaxe à utiliser (in French) / Serge Courrier, 30 November 2016.
[7] CloudWork, Zapier or elastic.io (paid for, IFTTT-likes) are not able to do the job — I had a close look at the list of their standard "actions" with RSS. The thread on Hacker News says the same with different words. They know how to transform one RSS feed into something else or sometimes into another RSS feed but they don’t know how to merge multiple RSS feeds into one feed :
- Zapier has a "zap" named "Curate RSS Feed into Your Own RSS Feed" but it does not take in multiple feeds : "Create a customized, curated RSS feed from an already existing RSS feed" says the recipe
- Cloudwork has no RSS to RSS "integration" and seems to be dead (see this post’s comments)
- elastic.io has no RSS to RSS "connector" either.
[8] See page 20 of these slides.
[9] See Serge’s Scoop.it : RSS Circus.
Commentaires
14 commentaires
Any Yahoo Pipes true substitute out there ?
Hi Emmanuel,
I noticed your mention of using the Inoreader Rules to filter feeds. May I ask why you didn’t use the Search feature ? It lets you filter within the scope of a feed, a feed folder, even lets you search against all feeds by all subscribers. Search queries have many customization options and support advanced search syntax.
Active Search queries can be saved as an Active Search. Like many other entities in Inoreader, Active Searches are RSS-enabled.
I hope this helps.
Kind greetings,
Marjolein Hoekstra
@CleverClogs
Any Yahoo Pipes true substitute out there ?
Hi Emmanuel,
One tiny correction :
I wrote :
Active Search queries can be saved as an Active Search.
Of course this should be
Search queries can be saved as an Active Search.
Kindest,
Marjolein
Major updates to this post
Any Yahoo Pipes true substitute out there ?
Hi Emmanuel, I noticed you mentioned CloudWork in your article. Have you actually checked if this company is still offering its product ? I’m curious, because currently, it’s not possible to sign up or sign in on CloudWork, and I got the overall feeling that this company shut down its services quite a while ago. But its name keeps appearing in various articles, as if it were still up and running. That’s why I asked you about checking - have you found a way to sign up at CloudWork after all or have you had an account there from the start ?
Cloudwork does seem closed in fact
Thanks for bringing up this matter, Olga.
No, I haven’t got a test code to Cloudwork. And yes, it seems the application isn’t alive anymore :
Nevertheless, I couldn’t find any piece of information on the Web about Nubera closing Cloudwork.
Any Yahoo Pipes true substitute out there ?
Two other possibilities :
bip.io (https://bip.io/)
We Wired Web (https://wewiredweb.com/)
Any Yahoo Pipes true substitute out there ?
In your article, you say "But it’s a pity Inoreader filter rules cannot have feeds as an outcome."
It’s not entirely correct. You can have a rule which would assign a specific category any time specific words are found in a feed (or in a group of feeds). Since Inoreader supports turning any user created category into a new RSS feed, doing this would effectively let you filter any number of feeds based on keywords and make the outcome a new RSS feed.
Any Yahoo Pipes true substitute out there ?
Thanks Jonathan. I’m correcting that in the post’s text.
Any Yahoo Pipes true substitute out there ?
This just in : Inoreader can now filter feeds : http://blog.inoreader.com/2015/07/see-only-relevant-content-with-filtered.html
Any Yahoo Pipes true substitute out there ?
Salut Emmanuel, Merci bcp pour ce poste ! Car la plupart des réponses est en anglais, je roule comme ça aussi.
I have several dozen pipes that do important things for me, so I spent several hours today looking for alternatives. Two that are paid but come the closest to how I use pipes are :
dlvr.it > https://blog.dlvrit.com/2015/06/yahoo-pipes-alternative/
Feedly pro > http://blog.feedly.com/2015/08/21/find-and-filter-the-best-content-on-the-web-with-power-search/
I am hoping that others will emerge ; I doubt that they will be free, though. For years, Pipes sucked all the oxygen from the market b/c it was so dominant, and other players are jockeying into position now.
Thanks again for your work !
A+ Chris
dlvr.it as a paid substitute for Pipes
Thanks to you Chris for pointing me to dlvr.it. I will include it in the paid hosted section.
Any Yahoo Pipes true substitute out there ?
Hi Emmanuel,
great article. A summary I really couldn’t do any better myself.
Please allow me to talk three sentences about the newscombinator.com, I am the founder.
It is another kind of RSS aggregation software, which can do more than just keywords filtering. Machine-Learning is built right into the core workflow. With that in mind, it is maybe a considerable alternative for people who need to stay updated on specific topics where news is spread across a lot of Feeds.
Thank you,
Thomas
Any Yahoo Pipes true substitute out there ?
Have you tried www.transmitdata.com ?
How to detect or create RSS feeds
Before filtering and merging RSS feeds, one must locate them or create one i it does not exist. So here is a long list (in French) from a very reliable source (Bryan Coder) :
Tout ce qu’il faut savoir pour obtenir des flux RSS sur le web 2.0 (All you need to know in order to get RSS feeds from social web sites and blogs].
One of the ressources listed by Bryan is very useful because it allows to build feeds from Twitter : Queryfeed (it has a limited, free version).
Laisser un commentaire