Any Yahoo Pipes true substitute out there ?

Tools for RSS feed merging and filtering

Mardi 29 septembre 2015, par Emmanuel Barthe // Logiciels, Internet, moteurs de recherche

Pipe dreams by darwin Bell

The loss of Yahoo Pipes in August this year [1] is a major blow to the monitoring/research community and also web site authors looking for selected content. Its ability to combine and filter multiple RSS feeds into one feed remains unique to this day among free hosted services.

This post will focus on how we could replace Yahoo Pipes for these functions [2]. We will not deal here with the web page scraping Pipes was able to do too [3].

So I went looking for substitutes which could merge and filter feeds. I mean, real ones : as simple as Pipes. For instance, they shall have a graphic user interface (GUI) ; coding shouldn’t be necessary. The only difficulty allowed should be the use of regular expressions (Regex).

Free hosted substitutes

  • FeedCombine, RSSMix, FeedRinse or ChimpFeedr (to merge feeds into one feed) then FeedSifter or FeedRinse (to filter the mega-feed) can answer simple needs.
    FeedRinse can do more, as it accepts Regex but it can’t merge more than 5 feeds, is slow and bugs according to La Bibliothèque du CHUM’s test.
    FeedCombine cannot combine more than 5 feeds, say Anik Dumont-Bissonnette.
    ChimpFeedr is very simple at "chomping" (as their web site says) feeds.
    RSSMix, according to the same test, works perfectly, merging up to 100 feeds
  • Feed Informer combines and filters feeds, but filtering can only be done using AND or OR operators
  • IF (ex-IFTTT) doesn’t seem to do exactly that : merging and filtering RSS feeds into one feed. Which is not to say it does not have its (numerous) advantages in the field of monitoring services.

Paid hosted substitutes

  • FeedsAPI does not seem to merge RSS feeds but can filter them. Afterwards you can use an IFTTT recipe such as Feedly Category to Twitter. FeedsAPI and Inoreader seem the easiest to use solutions among the paid ones
  • Tadaweb from Luxembourg, according to some blog posts, is able to create from web pages, filter and organize RSS feeds. The ouput is in RSS format only if the input is RSS too. NB : following a recent new policy, Tadaweb does no longer give access to its Creator module, its team is doing the work in lieu and place of the information professional and its pricing has accordingly evolved
  • the paid version of dlvr.it can filter and mash up RSS feeds
  • We Wired Web offers an IFTTT-like interface, RSS in and out and is able to filter it. But it does not merge different RSS feeds
  • the VIP and Premium versions of the Netvibes RSS portal are able to merge and filter RSS feeds by keywords, using Universal Search (which allows to query only certain groups of feeds — they’re called "tabs") and then Tracked Topics with RSS export
  • the Plus and Pro version of the RSS reader Inoreader allow merging and filter features : create a folder of feeds then an "active search" (an automated keyword search) on it and finally right click on the active search and click "Get RSS feed" [4]. It’s a pity Inoreader filter rules does not seem to have feeds as an outcome ; it would be interesting because they are much more precise than searches since they accept complex boolean queries and Regex [5] [6].

Self-hosted solutions (need coding skills)

In the sources listed below, I found other free or paid total or partial replacements — but you have to host them on your own server and/or they’re far too complicated for the average information professional :

  • Tiny Tiny RSS (TT-RSS) is an RSS reader with very interesting RSS feeds filtering features (open source)
  • Huginn by cantino (open source on GitHub). According to a MakeUseOf article, « this is the most popular of all the GitHub solutions mentioned, and has a huge, active support community behind it. It’s well documented and easy to use »
  • WebHookit, based on Node.js and mongoDB (open source)
  • Pypes (open source on GitHub), a Python application
  • Rss Percolator (open source on GitHub). Downloads, aggregates and filters RSS feeds
  • ClickScripts (open source on GitHub)
  • Bipio (open source). See the Getting started page (developer documentation) for a quick presentation. Bipio is a graph pipelining API talking RESTful JSON
  • Neddick (open source on GitHub).
  • pipe2py (open source on GitHub). You need to backup your YP pipes before. It uses JSON and the Google App Engine
  • Pipes2js (open source on GitHub)
  • Superpipes (open source), to be hosted on a Debian server
  • open source edition of MuleSoft’s Anypoint Studio
  • Quadrigram seems too graphics-oriented to me (apparently free).

Walled gardens

RSS is important to information professionals and, as we’ve seen, much used by people whose job is to monitor web resources. But this does not prevent RSS implementation from receding. If you look into the list of applications which can be connected by IFTTT, you will find that RSS is just one among more than a hundred of mostly proprietary ones ... Walled gardens again.

Did I miss something ? Do you know about any simple tool that could replace the "RSS mashup" function of Yahoo Pipes ? I’m interested in free as well as paid for services.

Emmanuel Barthe
independant law librarian researcher
open law specialist
monitoring solutions specialist


Sources for this article

Here are the links I found to some articles/blog posts and a number of forum discussions about Yahoo Pipes substitutes :

This article was written with the the help of Serge Courrier, a French consultant specialising in RSS readers and the use of RSS feeds for monitoring web sites [7].

Notes de bas de page

[1Starting end of August 2015, you will no longer be able to create new pipes ; Pipes will be in read only mode but you will still be able to save your pipes. At the end of September, the service will close.

[2Yahoo Pipes could do much more but we will skip that here.

[3To easily scrape web pages that do not have an RSS feed and so create their RSS feed, use paid services such as the simple, hosted and often recommended Feedity and FeedsAPI, the hosted but complex Feed43 — you need to master regular expresssions (Regex) —, or the self-hosted Feed Creator by FiveFilters.org.

[4In the Plus version, active searches are limited to 20. The Pro version makes them illimited. Thanks to Marjolein Hoekstra (CleverClogs) for pointing me to this Inoreader features. See her comments here under.

[5Inoreader is developped by a team of Bulgarians under the name of Innologica Ltd directed by its co-founder Yordan Yordanov.

[6CloudWork, Zapier or elastic.io (paid for, IFTTT-likes) are not able to do the job — I had a close look at the list of their standard "actions" with RSS. The thread on Hacker News says the same with different words. They know how to transform one RSS feed into something else or sometimes into another RSS feed but they don’t know how to merge multiple RSS feeds into one feed :

  • Zapier has a "zap" named "Curate RSS Feed into Your Own RSS Feed" but it does not take in multiple feeds : "Create a customized, curated RSS feed from an already existing RSS feed" says the recipe
  • Cloudwork has no RSS to RSS "integration" and seems to be dead (see this post’s comments)
  • elastic.io has no RSS to RSS "connector" either.

[7See Serge’s Scoop.it : RSS Circus.

Répondre à cet article

14 Messages

  • Any Yahoo Pipes true substitute out there ? 12 juin 2015 11:03, par Marjolein Hoekstra

    Hi Emmanuel,
    I noticed your mention of using the Inoreader Rules to filter feeds. May I ask why you didn’t use the Search feature ? It lets you filter within the scope of a feed, a feed folder, even lets you search against all feeds by all subscribers. Search queries have many customization options and support advanced search syntax.
    Active Search queries can be saved as an Active Search. Like many other entities in Inoreader, Active Searches are RSS-enabled.

    I hope this helps.

    Kind greetings,

    Marjolein Hoekstra
    @CleverClogs

    repondre message

  • Any Yahoo Pipes true substitute out there ? 12 juin 2015 11:07, par Marjolein Hoekstra

    Hi Emmanuel,

    One tiny correction :
    I wrote :
    Active Search queries can be saved as an Active Search.

    Of course this should be

    Search queries can be saved as an Active Search.

    Kindest,
    Marjolein

    repondre message

  • Major updates to this post 14 juin 2015 01:52, par Emmanuel Barthe
    • took out Kimono (just a web scraper) and ClickScript (just a graphic user interface)
    • added more details about Inoreader functionalities following Marjolein Hooekstra’s post
    • added pipe2py
    • confirmed Netvibes functionalities so raised its mention from notes to content
    • added RssPercolator and Neddick (suggestions from Hacker News discussion thread).

    repondre message

  • Any Yahoo Pipes true substitute out there ? 15 juin 2015 08:39, par Olga

    Hi Emmanuel, I noticed you mentioned CloudWork in your article. Have you actually checked if this company is still offering its product ? I’m curious, because currently, it’s not possible to sign up or sign in on CloudWork, and I got the overall feeling that this company shut down its services quite a while ago. But its name keeps appearing in various articles, as if it were still up and running. That’s why I asked you about checking - have you found a way to sign up at CloudWork after all or have you had an account there from the start ?

    repondre message

    • Cloudwork does seem closed in fact 15 juin 2015 17:52, par Emmanuel Barthe

      Thanks for bringing up this matter, Olga.

      No, I haven’t got a test code to Cloudwork. And yes, it seems the application isn’t alive anymore :

      • you’re right, there’s no place on their site to create an ID nor — that’s more important — to simply connect with an existing one ...
      • the site links to lots of pages of the GetApp (an online marketplace for SaaS) site
      • the terms of service makes no reference to Cloudwork but to Nubera and GetApp.

      Nevertheless, I couldn’t find any piece of information on the Web about Nubera closing Cloudwork.

      repondre message

  • Any Yahoo Pipes true substitute out there ? 22 juin 2015 00:29, par Scott Frey

    Two other possibilities :
    bip.io (https://bip.io/)
    We Wired Web (https://wewiredweb.com/)

    repondre message

  • Any Yahoo Pipes true substitute out there ? 2 juillet 2015 16:38, par Jonathan

    In your article, you say "But it’s a pity Inoreader filter rules cannot have feeds as an outcome."

    It’s not entirely correct. You can have a rule which would assign a specific category any time specific words are found in a feed (or in a group of feeds). Since Inoreader supports turning any user created category into a new RSS feed, doing this would effectively let you filter any number of feeds based on keywords and make the outcome a new RSS feed.

    repondre message

  • Any Yahoo Pipes true substitute out there ? 24 août 2015 23:53, par Christopher Rollyson

    Salut Emmanuel, Merci bcp pour ce poste ! Car la plupart des réponses est en anglais, je roule comme ça aussi.

    I have several dozen pipes that do important things for me, so I spent several hours today looking for alternatives. Two that are paid but come the closest to how I use pipes are :

    dlvr.it > https://blog.dlvrit.com/2015/06/yahoo-pipes-alternative/

    Feedly pro > http://blog.feedly.com/2015/08/21/find-and-filter-the-best-content-on-the-web-with-power-search/

    I am hoping that others will emerge ; I doubt that they will be free, though. For years, Pipes sucked all the oxygen from the market b/c it was so dominant, and other players are jockeying into position now.

    Thanks again for your work !

    A+ Chris

    repondre message

  • Hi Emmanuel,

    great article. A summary I really couldn’t do any better myself.

    Please allow me to talk three sentences about the newscombinator.com, I am the founder.

    It is another kind of RSS aggregation software, which can do more than just keywords filtering. Machine-Learning is built right into the core workflow. With that in mind, it is maybe a considerable alternative for people who need to stay updated on specific topics where news is spread across a lot of Feeds.

    Thank you,
    Thomas

    repondre message

  • How to detect or create RSS feeds 24 avril 21:23, par Emmanuel Barthe

    Before filtering and merging RSS feeds, one must locate them or create one i it does not exist. So here is a long list (in French) from a very reliable source (Bryan Coder) :
    Tout ce qu’il faut savoir pour obtenir des flux RSS sur le web 2.0 (All you need to know in order to get RSS feeds from social web sites and blogs].

    One of the ressources listed by Bryan is very useful because it allows to build feeds from Twitter : Queryfeed (it has a limited, free version).

    repondre message