If anyone’s been following my Twitter feed recently they’ll know that I’ve been getting to grips with Yahoo Pipes. I think it’s an amazing tool and offers the tantalising possibility of creating a wholly modular monitoring system, effectively through programming-like routines which, through a neat graphical interface, largely remove the need for knowing how to program.
This post isn’t going to explain how to put Pipes together – you can find that out for yourself. 😉 Although I am considering putting together a few short, sharp tutorials on the basics, without giving away too much (I’m using this system to complement our monitoring programme at Fleishman-Hillard and while we’re nice here, we’re not that nice).
What it is going to describe is some of the pitfalls of the system, and a pointer towards what I’ve put together.
Tell us something we don’t already know
The ability to bring together RSS feeds is nothing new. You’ve been able to create feeds with Google Reader since it came out. Simply bundle feeds together under a folder (or tag – Google is a bit fast and loose with its terminology), and make it public. Voila – one feed comes out from the many that came in.
But that’s where it ends. You can’t get at the contents of feeds and do smart stuff with them, such as change details within feeds or cross-reference them with others. With Yahoo Pipes, you can.
Remember the fun you had with Lego?
Let’s start with the second first: how to create a modular system.
First, take a look at the social media firehose. It’s a beast (a firehose beast). I advise you to keep calm while looking at it, and slowly take it apart piece by piece to understand what’s going on. If I tell you that all it’s doing is taking keywords, building RSS queries across certain RSS-enabled search engines and bringing them together in one feed, it should start to make sense.
But I can tell you now that you in fact can take it apart and put it together in various smart ways such that it is entirely modular. That is, you can store keywords and search engines separately.
Let’s say you want to do a search for Hewlett Packard (not a client) in the UK. You can specify keywords in one pipe – for example HP, with “Harry Potter” as an exclusion (as I know from bitter experience), and UK as an inclusion. In a search engine this query would look something like HP -“Harry Potter” +UK.
Now create a new pipe, bringing your keywords pipe in as the source, plugging that into your blog pipe and your microblog pipe.
As I’ve already said, you could create something similar in Google Reader for example. But consider what happens if you want to change your keywords. Putting a good keyword list together isn’t an exact science. You’ll need several attempts before you get the results you want, mainly because you’ll have to filter out weird stuff – “HP sauce” springs to mind. So you would have to rebuild every feed with that new keyword in.
And say you discover that a blog search doesn’t quite work properly. Or say a new microblog platform comes along – one that works better that Twitter and starts to attract significant users (it might happen). To make sure you’re monitoring properly, you’d have to build this new search into all your previous searches.
Not with modular pipes. Change the keywords once, and all your searches pick it up. Change the blog or microblog pipe once, and all your searches now go through them. It’s very cool.
So, your future searches will always be using the latest keywords and search engines. It also means you can prototype much more quickly, and that means a lot less hassle when putting together monitoring systems.
It reminds me of Lego, just putting together the bits you need in interesting new ways. Or perhaps the electrical circuits analogy is better. Or maybe it’s like upgrading a PC – just take out the old component and shove in the new.
Am I going to tell you how to do this?
No. As I said, I think I’ve spent enough time going through this to consider my systems proprietary knowledge. And I’m sure other people have figured out their own ways to do this, and that you can too.
What I will tell you about is some of the problems you might encounter along the way. Story is, Yahoo Pipes isn’t absolutely stable, and RSS is a right old pain to work with sometimes.
This time, let’s talk about the first, first.
Yahoo Pipes has some teething troubles
First (again), the interface, while immediately attractive, can break. After a slightly prolonged editing session you’ll find that modules won’t connect, so you have to save the pipe, get out, and go back in again.
On the other hand, very quick edits might not activate the ‘Save’ button, so you’ll have to do something more radical – break a link and re-link it – to activate the button so you can save it.
If you edit a pipe within another pipe, then go back to the original pipe, you have to re-link it.
And there’s no ‘undo’ command.*
Also, pipe management is non-existent. You can’t put pipes into folders, and they’re sorted by date last edited. I’m currently standing at six pages of pipes so if I want to edit a pipe I haven’t looked at for a while, I potentially have to page through six times to find it. This is surely a very easy thing for Yahoo to fix. Add to that the fact that your pipe list often doesn’t automatically refresh, and you can get quite frustrated. I know I do.
But what I’ve found really irritating – and what PR Week picked up in my rant – is that you can sometimes just be locked out of the system altogether, or you’ll get an annoying message like “Oops, error parsing response.” You can only pray that your changes were saved. On occasion, mine haven’t.
And I recently found myself unable to access any of my pipes over an entire weekend when I had a neat solution to a problem in my head, but just couldn’t get into the system to see if I was right. I wasn’t, but that’s neither here nor there.
RSS is a bit annoying to work with too
There seems to be a delay effect when using pipes. This could be another Yahoo Pipes problem, but I suspect it’s part and parcel of caching in RSS. So you can make changes to your pipe that do not reflect in the output. You can wait several minutes before they show, by which time you’ve lost that zen-like flow of concentration.
Not how to do it, but how not to do it
So I’m feeling bad now that I’m not telling you how to do things with pipes. Just to give you a little more help along the way, here’s a prescription for working with then:
- Keywords – as I say, it’s not an exact science. The matrix approach I helped develop at Porter Novelli (and yes, I did try to link to the PN blog but page 2 onwards don’t seem be loading right now) goes some way towards identifying them but you’ll need to tweak them to get useful results. In the same way you can have too few keywords, you can also have too many. Think about how best to arrange them to get good search results.
- Yahoo Pipes – edit in short sessions, and make sure you save regularly. Give your pipes meaningful names to get around the lack of management. Keep breathing.
- Debugging – if your results don’t come out right, it’s usually your fault. Check your keywords, make sure you’re putting phrasal keywords in quotes, check your spelling, be canny with your inclusions and exclusions. If it’s not your fault then give it a few minutes – this could be an RSS caching issue. Sometimes it’s better just to let the pipe do its own thing for a few hours until the meaningful information starts to come through.
- If you get too wound up, lift your arms in the air, waft them around your head lightly, and say in a calm voice “I am a tree.”
Can’t you just show us one pipe?
Oh, alright then. This is the schematic icon, increased by 400% (hence the blockiness), showing one of my modular pipes:
Interestingly, I think it actually looks like one of the hookah pipes proudly displayed (and used) in the many hostelries along one of my old stamping grounds, Queensway.
Any other suggestions?
* Am I alone in occasionally doing stupid things in real-life and thinking “CTRL-Z”?