Any Yahoo Pipes true substitute out there ?

Tools for RSS feed merging and filtering

Mercredi 30 novembre 2016, par Emmanuel Barthe // Logiciels, Internet, moteurs de recherche

Pipe dreams by darwin Bell

The loss of Yahoo Pipes in August 2015 [1] was a major blow to the monitoring/research community and also web site authors looking for selected content. Its ability to combine and filter multiple RSS feeds into one feed was unique among free hosted services.

This post will focus on how we could replace Yahoo Pipes for these functions [2]. We will not deal here with the web page scraping Pipes was able to do too [3].

So I went looking for substitutes which could merge and filter feeds. I mean, real ones : as simple as Pipes. For instance, they shall have a graphic user interface (GUI) ; coding shouldn’t be necessary. The only difficulty allowed should be the use of regular expressions (Regex).

Free hosted substitutes

  • FeedCombine, RSSMix, FeedRinse or ChimpFeedr (to merge feeds into one feed) then FeedSifter or FeedRinse (to filter the mega-feed) can answer simple needs.
    FeedRinse can do more, as it accepts Regex but it can’t merge more than 5 feeds, is slow and bugs according to La Bibliothèque du CHUM’s test.
    FeedCombine cannot combine more than 5 feeds, say Anik Dumont-Bissonnette.
    ChimpFeedr is very simple at "chomping" (as their web site says) feeds.
    RSSMix, according to the same test, works perfectly, merging up to 100 feeds
  • Feed Informer combines and filters feeds, but filtering can only be done using AND or OR operators
  • IFTTT doesn’t seem to do exactly that : merging and filtering RSS feeds into one feed. Which is not to say it does not have its (numerous) advantages in the field of monitoring services.

Paid hosted substitutes

Self-hosted solutions (need coding skills)

In the sources listed below, I found other free or paid total or partial replacements — but you have to host them on your own server and/or they’re far too complicated for the average information professional :

  • Tiny Tiny RSS (TT-RSS) is an RSS reader with very interesting RSS feeds filtering features (open source)
  • Huginn by cantino (open source on GitHub). According to a MakeUseOf article, « this is the most popular of all the GitHub solutions mentioned, and has a huge, active support community behind it. It’s well documented and easy to use »
  • WebHookit, based on Node.js and mongoDB (open source)
  • Pypes (open source on GitHub), a Python application
  • Rss Percolator (open source on GitHub). Downloads, aggregates and filters RSS feeds
  • ClickScripts (open source on GitHub)
  • Bipio (open source). See the Getting started page (developer documentation) for a quick presentation. Bipio is a graph pipelining API talking RESTful JSON
  • Neddick (open source on GitHub).
  • pipe2py (open source on GitHub). You need to backup your YP pipes before. It uses JSON and the Google App Engine
  • Pipes2js (open source on GitHub)
  • Superpipes (open source), to be hosted on a Debian server
  • open source edition of MuleSoft’s Anypoint Studio
  • Quadrigram seems too graphics-oriented to me (apparently free).

NB : the first four services are recommended [8] by Serge Courrier, a French consultant specialising in RSS readers and the use of RSS feeds for monitoring web sites [9].

Walled gardens

RSS is important to information professionals and, as we’ve seen, much used by people whose job is to monitor web resources. But this does not prevent RSS implementation from receding. If you look into the list of applications which can be connected by IFTTT, you will find that RSS is just one among more than a hundred of mostly proprietary ones ... Walled gardens again.

Did I miss something ? Do you know about any simple tool that could replace the "RSS mashup" function of Yahoo Pipes ? I’m interested in free as well as paid for services.

Emmanuel Barthe
law librarian researcher, open law specialist and monitoring solutions specialist


Sources for this article

Here are the links I found to some articles/blog posts and a number of forum discussions about Yahoo Pipes substitutes :

This article was written with the the help of Serge Courrier.

Notes de bas de page

[1Starting end of August 2015, one was no longer able to create new pipes. At the end of September 2015, the service closed.

[2Yahoo Pipes could do much more but we will skip that here.

[3To easily scrape web pages that do not have an RSS feed and so create their RSS feed, use paid services such as the simple, hosted and often recommended Feedity and FeedsAPI, the hosted but complex Feed43 — you need to master regular expresssions (Regex) —, or the self-hosted Feed Creator by FiveFilters.org.

[4Inoreader is developped by a team of Bulgarians under the name of Innologica Ltd directed by its co-founder Yordan Yordanov.

[5In the Plus version, active searches are limited to 20. The Pro version makes them illimited. Thanks to Marjolein Hoekstra (CleverClogs) for pointing me to this Inoreader feature. See her comments here under.

[6Les filtres RSS dans Inoreader : détail de la syntaxe à utiliser (in French) / Serge Courrier, 30 November 2016.

[7CloudWork, Zapier or elastic.io (paid for, IFTTT-likes) are not able to do the job — I had a close look at the list of their standard "actions" with RSS. The thread on Hacker News says the same with different words. They know how to transform one RSS feed into something else or sometimes into another RSS feed but they don’t know how to merge multiple RSS feeds into one feed :

  • Zapier has a "zap" named "Curate RSS Feed into Your Own RSS Feed" but it does not take in multiple feeds : "Create a customized, curated RSS feed from an already existing RSS feed" says the recipe
  • Cloudwork has no RSS to RSS "integration" and seems to be dead (see this post’s comments)
  • elastic.io has no RSS to RSS "connector" either.

[8See page 20 of these slides.

[9See Serge’s Scoop.it : RSS Circus.

Répondre à cet article

14 Messages

  • Any Yahoo Pipes true substitute out there ? 12 juin 2015 11:03, par Marjolein Hoekstra

    Hi Emmanuel,
    I noticed your mention of using the Inoreader Rules to filter feeds. May I ask why you didn’t use the Search feature ? It lets you filter within the scope of a feed, a feed folder, even lets you search against all feeds by all subscribers. Search queries have many customization options and support advanced search syntax.
    Active Search queries can be saved as an Active Search. Like many other entities in Inoreader, Active Searches are RSS-enabled.

    I hope this helps.

    Kind greetings,

    Marjolein Hoekstra
    @CleverClogs

    repondre message

  • Any Yahoo Pipes true substitute out there ? 12 juin 2015 11:07, par Marjolein Hoekstra

    Hi Emmanuel,

    One tiny correction :
    I wrote :
    Active Search queries can be saved as an Active Search.

    Of course this should be

    Search queries can be saved as an Active Search.

    Kindest,
    Marjolein

    repondre message

  • Major updates to this post 14 juin 2015 01:52, par Emmanuel Barthe
    • took out Kimono (just a web scraper) and ClickScript (just a graphic user interface)
    • added more details about Inoreader functionalities following Marjolein Hooekstra’s post
    • added pipe2py
    • confirmed Netvibes functionalities so raised its mention from notes to content
    • added RssPercolator and Neddick (suggestions from Hacker News discussion thread).

    repondre message

  • Any Yahoo Pipes true substitute out there ? 15 juin 2015 08:39, par Olga

    Hi Emmanuel, I noticed you mentioned CloudWork in your article. Have you actually checked if this company is still offering its product ? I’m curious, because currently, it’s not possible to sign up or sign in on CloudWork, and I got the overall feeling that this company shut down its services quite a while ago. But its name keeps appearing in various articles, as if it were still up and running. That’s why I asked you about checking - have you found a way to sign up at CloudWork after all or have you had an account there from the start ?

    repondre message

    • Cloudwork does seem closed in fact 15 juin 2015 17:52, par Emmanuel Barthe

      Thanks for bringing up this matter, Olga.

      No, I haven’t got a test code to Cloudwork. And yes, it seems the application isn’t alive anymore :

      • you’re right, there’s no place on their site to create an ID nor — that’s more important — to simply connect with an existing one ...
      • the site links to lots of pages of the GetApp (an online marketplace for SaaS) site
      • the terms of service makes no reference to Cloudwork but to Nubera and GetApp.

      Nevertheless, I couldn’t find any piece of information on the Web about Nubera closing Cloudwork.

      repondre message

  • Any Yahoo Pipes true substitute out there ? 22 juin 2015 00:29, par Scott Frey

    Two other possibilities :
    bip.io (https://bip.io/)
    We Wired Web (https://wewiredweb.com/)

    repondre message

  • Any Yahoo Pipes true substitute out there ? 2 juillet 2015 16:38, par Jonathan

    In your article, you say "But it’s a pity Inoreader filter rules cannot have feeds as an outcome."

    It’s not entirely correct. You can have a rule which would assign a specific category any time specific words are found in a feed (or in a group of feeds). Since Inoreader supports turning any user created category into a new RSS feed, doing this would effectively let you filter any number of feeds based on keywords and make the outcome a new RSS feed.

    repondre message

  • Any Yahoo Pipes true substitute out there ? 24 août 2015 23:53, par Christopher Rollyson

    Salut Emmanuel, Merci bcp pour ce poste ! Car la plupart des réponses est en anglais, je roule comme ça aussi.

    I have several dozen pipes that do important things for me, so I spent several hours today looking for alternatives. Two that are paid but come the closest to how I use pipes are :

    dlvr.it > https://blog.dlvrit.com/2015/06/yahoo-pipes-alternative/

    Feedly pro > http://blog.feedly.com/2015/08/21/find-and-filter-the-best-content-on-the-web-with-power-search/

    I am hoping that others will emerge ; I doubt that they will be free, though. For years, Pipes sucked all the oxygen from the market b/c it was so dominant, and other players are jockeying into position now.

    Thanks again for your work !

    A+ Chris

    repondre message

  • Any Yahoo Pipes true substitute out there ? 23 avril 2016 17:51, par Thomas

    Hi Emmanuel,

    great article. A summary I really couldn’t do any better myself.

    Please allow me to talk three sentences about the newscombinator.com, I am the founder.

    It is another kind of RSS aggregation software, which can do more than just keywords filtering. Machine-Learning is built right into the core workflow. With that in mind, it is maybe a considerable alternative for people who need to stay updated on specific topics where news is spread across a lot of Feeds.

    Thank you,
    Thomas

    repondre message

  • How to detect or create RSS feeds 24 avril 2016 21:23, par Emmanuel Barthe

    Before filtering and merging RSS feeds, one must locate them or create one i it does not exist. So here is a long list (in French) from a very reliable source (Bryan Coder) :
    Tout ce qu’il faut savoir pour obtenir des flux RSS sur le web 2.0 (All you need to know in order to get RSS feeds from social web sites and blogs].

    One of the ressources listed by Bryan is very useful because it allows to build feeds from Twitter : Queryfeed (it has a limited, free version).

    repondre message