open data

Sep 11

Open data for everyday life

TfL Countdown board at bus stop

I’ve noticed two main motivations in the campaign for government open data.

Some people want to change the world. They want to reform government and create a new era of accountability through transparent public bodies.

And some people just want to find out when the next bus is due.

The two of these aren’t mutually exclusive and there’s a point at which they meet, but today I’m going to look at how open data can improve people’s everyday lives.

My interest in local government stems from my core motivations as a designer. We can put people on the Moon and build complex and elegant skyscrapers but why do so many people struggle through their daily lives? To paraphrase William Gibson, at what point does the future become evenly distributed? When do the rest of us Earthbound ground dwellers get a slice of the techno-pie?

Local councils touch so many aspects of our everyday activities. We can go for months without having any direct contact with a central government department. We can avoid pretty much any company we don’t like, though perhaps not all of them at once. But we can barely go a day without using a service provided by our local councils.

Councils collect our bins. They maintain our streets. They educate our children. They run our parks and public libraries and leisure centres. And once a month they send us the bill. Whatever you think about your own council or local government in general, they’re unavoidable. Making councils work better directly and immediately makes people’s lives better. Designers and technologists should be all over them.

But in as much as I think that councils should always be striving to provide better services and to provide better information and access to those services, I know that they can’t do it all by themselves. People’s needs are too diverse. Technology changes too quickly. Aspirations always outstrip the ability of councils to meet them. Try their very best with the budgets and resources available and it’s impossible to keep up let alone get ahead.

Open data offers councils the opportunity to tackle one part of the problem.

So when’s the next bus due?

If you live in London this is now a much easier question to answer.

TfL have just launched their Countdown website where you can look up bus arrival times. It’s the same data that appears on the red screens at some bus stops.

TfL Countdown mobile website screenshot

In fact, there are three Countdown websites. One for computers with “normal sized” screens, your typical PC or laptop. One for smartphones and other mobile devices. And one “accessible” site for people who have difficulty using conventional screens.

These three websites tell a story in themselves. A story of diverse needs and aspirations. About wanting to provide for everyone in every context and on every device while still trying to deliver the best user experience possible in each situation.

And these sites are good. TfL has put the effort in and succeeded.

But better still, TfL has built its websites in a smart way. All three sites draw data from the same source so there’s no need to access the underlying database directly in every app. Just build an API and pull the data from there.

That API is available for anyone to use (unofficially, at least). Just request the current bus arrivals data for any stop and you get the answer in a format you can use in your own apps.

So I built something that TfL probably never considered and would almost certainly never have got approved had they thought of it: a Unix command line utility for Countdown.

Why? Because I’m in a niche group. My aspirations (if not exactly my needs) are well outside the mainstream. Like a few hundred people in London but unlike just about everyone else, I spend nearly all day every day sitting at a computer with a Unix terminal window open.

So rather than visiting a website and clicking my way around, it’s much easier for me to type:


and get a response like this:

Stonecot Hill outside Metalcraft (50799)

93    Putney Bridge        5 min
413   Morden               12 min

Ash Road (50823)

93    North Cheam          3 min
413   Sutton Garage        9 min

A staggering 33 people have downloaded this software since I published it four days ago. That’s nearly nine people a day.

This vanishingly small audience represents a technological and organisational triumph. This isn’t one size fits all. It’s one size fits one if it needs to. It’s what Clay Shirky calls situated software — extreme customisation for the most niche tastes. It’s the information you need, when you need it, in the way that suits you best. It’s a tiny public service in a place where government wouldn’t dream to reach.

If the Unix command line is your thing you’re welcome to become the 34th user.

So those three websites that TfL created for their Countdown service aren’t overkill. They aren’t even covering all the bases. There are hundreds of different ways in which this data could be used in every conceivable context and through any imaginable technology, past, present or future. You could build it into other websites and apps. Put it on screens in shops or hospital waiting rooms. Print it out on till receipts. Run a flag up a flagpole when a bus is around the corner. Whether it’s a labour of love, a business opportunity or a way to improve something that already exists, you can make it. TfL has built the API and the mainstream web apps. The rest is up to you.

So that’s London buses. Some other UK towns and cities have got similar websites though I’m not sure whether they’ve got APIs yet. They need them.

What’s next?

Socitm has recently published its plans for its annual survey of council websites, Better Connected.

Better Connected 2012 will focus on top tasks, the most common things that people visit their council websites to do.

Socitm suggests that the top tasks are:

  • Apply for a council job
  • Comment on a planning application
  • Find winter gritting routes
  • Find rubbish collection day
  • Pay a parking fine
  • Pay council tax
  • Renew a library book
  • Find out about getting help at home
  • Find out school term dates
  • Find opening times for local swimming pool
  • Find records office opening hours

Socitm will be looking to see how well councils provide this information and these services on their own websites.

Councils should work to make all these things as easy as possible to do on their own websites.

But you shouldn’t need to visit your council’s website to do any of them if something else would suit you better.

An iPhone app. An app for a tablet computer that got withdrawn after two months on sale. A nicely formatted table on a leaflet.  A desktop or web widget. A little light flashing on your bin on the day it needs to be taken out. And not forgetting the Unix command line for the three dozen people that might use it.

As councils polish up the top tasks on their websites they should be building APIs for those services too, starting with the most popular and working their way down.

There’s more than one way to do it and there’s nothing better than doing it your own way. We need open data and APIs for everyday life. Councils have the ability to make it happen.

Apr 11

Boris says bye-bye to indie Boris Bikes developers

Barclays Cycle Hire app iTunes screenshot

9 May 2011: Some of my assumptions in this post are wrong so please read it in the context of Emer Coleman’s comment below.

Courted, used and discarded in less than a year. That’s Boris’s and Transport for London’s attitude towards independent app developers for the Barclays Cycle Hire scheme.

Let’s take it from the beginning.

A month before TfL launched their new cycle hire scheme, Boris was very keen to get independent developers on board. Why? Some deep commitment to digital diversity or small government doing what it does best and leaving the rest to the market?

Not really. Just that in 2010 if you’re launching a public cycle hire scheme in a major world city you need an app. And there wasn’t any budget allocated for one so the open data line was expedient.

Let’s hear some of the bull from back in June 2010:

In build up to the launch of the Mayor’s Barclays Cycle Hire scheme on 30 July, Transport for London (TfL) has relaxed its terms and conditions to allow commercial use of official data – opening the door for developers to provide accurate and reliable information about the hundreds of locations where hire cycles will be available, smart routes around town or proximity of docking stations to Tube stations and places of interest.

Of course this wasn’t actually true anyway. TfL didn’t release any live machine-readable data about bike or dock availability at that point. In fact, they still haven’t.

More bull from TfL:

Independently produced apps will complement the wealth of information that TfL is already generating to keep users up to speed about the scheme.

So what happened? Indie developers got on board only to find that they had to screen scrape data from TfL’s web map, the only publicly-available source of data. No real API, no service level standards, no support. And very often crap data.

The incident where TfL’s map started serving up data from the Montreal cycle hire scheme being just one case in point.

The indies have muddled through, producing some good apps that very often have been held back by poor and unreliable data. When it comes to realtime information services, your app is only as good as your data.

People have invested time and money in these apps, largely in the hope that TfL would see them right soon enough.

All the while, developers have been pressing TfL for a real API. The story has always been that it’s coming… one day.

I think it’s reasonable to say that indie developers have made a big contribution towards Barclays Cycle Hire’s success. There aren’t unlimited bikes and you need to be able to find them. It’s handy to have a timer to help manage the costs. And you need a map on the go just to find the docking stations. You need an app.

TfL have been happy to take the credit for the indie cycle hire app and analysis work that they’ve done next to nothing to support.

And now we get the final confirmation of where TfL really stands on indie developers and open data: This week Barclays launched official iPhone and Android apps for the scheme.

These free apps (with all of Barclays/TfL’s marketing support behind them) wipe out the largest markets for indie apps at a stroke.


Moreover, Barclays own apps will doubtless be using a private API to which they have privileged access. So their apps get good quality data while everyone else struggles along with the leftovers.

I’m told, unofficially, that an official cycle hire API is coming soon. But I’ve heard that story before.

When it comes — if it comes — it’ll be useful for the people doing data analysis and building cycle hire data into novel apps and games like Chromaroma.

But for the mass market — indie developers making and selling standard find a bike/dock apps — TfL just doesn’t need you any more.

The parallels with Twitter’s attitude to its API are clear: Having built a successful service on the back of indie developers’ labour, it’s now time to take the good stuff in house and reap the rewards. At least Twitter provided a proper API.

The question remains: Who’s driving Barclays Cycle Hire, Barclays or the mayor?

Perhaps the clue’s in the name.

Apr 11

Waste minimisation and the quantified self

Last month, Sutton Council was looking for ways to save £925K a year in waste collection costs. There was an online discussion where residents were asked to come up with ideas for making savings and also give their views on suggestions made by the council.

Some of the ideas such as reorganising waste collection shifts to enable the council to halve the number of vehicles are efficiencies that would have a relatively minor impact on residents. Others inevitably are focussing directly on how much of their household waste is being recycled by residents and how much is being sent to landfill.

Rewards for recycling

Rewards for recycling or fines for not recycling enough are among the options. Introducing penalties for bad behaviour isn’t a very popular idea among the public and also in many councils who would rather have constructive rather than punitive relations with their residents. Moreover, there are many practical difficulties in running a punitive scheme. It’s easy enough to put your waste in someone else’s bin unless every bin is fitted with a lock. A trial in Norfolk failed due to numerous technical problems and also led to a 250% increase in fly tipping.

Windsor and Maidenhead are in the process of rolling out a borough-wide recycling rewards scheme after a successful trial with 6500 households.

You can get fantastic rewards at participating businesses like M&S, Legoland, Magnet and Windsor Leisure Centres or you can donate your points to the RecycleBank Green Schools Scheme.

Chip and bin

As with the Norfolk scheme, Windsor and Maidenhead are using so-called chip-and-bin technology. Residents’ recycling bins are fitted with an RFID chip identifying the household to which it belongs. The bin is weighed automatically as it’s emptied into the collection vehicle and the weight is added to the appropriate household’s account. RFID is a short-range radio system that unlike barcodes doesn’t require manual scanning and enables bins to be identified automatically as part of the emptying process.

While we wait to see whether Windsor and Maidenhead’s scheme will have a positive long-term impact on residents’ recycling habits, there’s another approach we could consider. Instead of the council weighing your bin, why not do it yourself?

The quantified self

We’re all familiar with large organisations collecting data about us, whether it’s some part of government, or the supermarket recording our every purchase for their loyalty card scheme. Many people are sceptical about or outright hostile to the increasing amount of intelligence gathering directed at us as citizens and shoppers. Whether well-intended and well-managed or not, this database-building nonetheless chips away another little bit of our privacy.

But technology observers have recently been following the trend for some people to collect this kind of data about themselves. This concept of the quantified self pulls together a diverse set of self-monitoring and self-improvement practices in which people collect data about themselves, analyse it and use it as evidence to support decision making and behaviour changes.

Some quantified self (QS) applications need you to type data into a website. Want to ramp up your drinking? Try DrinkingDiary. Sex life too hot to handle? Bedpost can help you cool it down. Others use common gadgets like smartphones. Runkeeper will track how far and how often you run, where you’ve been and will let you share your progress with others in its own community.

Some QS apps are sophisticated product service systems that come with their own custom hardware that takes the pain out of data collection. Fitbit gives you a tiny clip-on device that works as a pedometer to track your exercise levels while you’re awake and monitors your sleep time and quality while you’re in bed.

The Withings Body Scale is no ordinary bathroom scale. It doesn’t just let you weigh yourself, it wirelessly and automatically transmits your weight to a central database, from where you can monitor your progress on the web and through various mobile apps.

So if we can weigh our bodies, why not our bins?

DIY chip and bin — the pilot project

My own “pilot project” for tracking my waste disposal has so far taken around ten minutes of my time and cost £3.50 in hard cash. The money went on a portable luggage scale that I picked up in my local newsagent and the time went building a simple Google Spreadsheet and form that lets me easily type in the weight of each bin bag before I put it out into the bins. Here in Sutton we’ve got a brown bin for landfill waste and a green bin for recycling. I’m weighing both. When I fill in the Google form the numbers get automatically added to the spreadsheet along with the current date. Then I can just total up the columns and see how much I’m landfilling and recycling over time. It’s a good start.

Despite its low cost and relative simplicity, my system requires a fair bit of effort to get started and to diligently weigh and record each bag as it goes out to the bins. But what if we could buy kitchen bins that did the weighing and recording for us like the Withings Body Scale? I’m sure it’s only a matter of time before we can. Technologically it’s almost the same product, though the software would need adapting to the different context of having two or more bins recording weights of different waste products rather than a single scale recording many people’s weights. This isn’t rocket science.

But will you behave better?

What’s likely to be much harder than collecting this data is using it to help people actually change their behaviour. Someone who goes to buy or contrive a weighing bin is at least starting off with some motivation and a good intention. Sustaining that might be hard. As with many fitness apps, a social networking component where the user could share and compare their data with others in similar households might provide sufficient motivation and social reinforcement to keep going. A web-based system could send occasional suggestions for ways to waste less and recycle more, even in the form of advertisements for eco-friendly packaged products like those in Amazon’s 100% recyclable Frustration-Free Packaging.

Twenty minutes into the future…

Now imagine what could be done if this system had access (with your permission, of course) to your supermarket loyalty card data. Suggestions on how to substitute poorly-packaged products for comparable ones based on your actual shopping habits? Suggestions on how you could buy larger packs of existing products that would have proportionally less packaging? Having access to this info on your smartphone while you’re in the shop? Discount vouchers as a little extra nudge to get you started? It’s all perfectly feasible.

If people can do data for themselves, do councils still have a role to play? One way might be to sidestep the whole issue of people getting their own weighing bins and providing bin weight data collected through chip-and-bin straight to residents without operating a local penalty or reward scheme. Residents could then participate in any third-party reward scheme that they chose, most likely operated by a supermarket or another retailer. People would appreciate the choice and flexibility of being in control of how their data was used and could pick a reward scheme that suited them best. Retailers would have greater incentive to work with their suppliers to improve packaging. Councils would benefit from lower landfill taxes and greater recycling rates without going to the trouble of running their own reward schemes or getting out the thumbscrews.

Whose data? Our data!

The biggest barrier to making this happen isn’t the technology. We’ve already got that. The problem is getting the organisations that collect data about us to give us access. We need a new deal with the organisations that know more about us than we know ourselves: If you want my data I get to use it too and I get a secure way of sharing it with trusted third parties that can do something good with it that benefits me. If you want to log my phone calls or my purchases or the weight of my bin every time you collect it — show me your API. Quid pro quo or GTFO.

We’re starting to make good progress with the open data movement to get government to release its non-personal data for everyone to use. The next step is to get equal access to the personal data that government and business holds about us so it can work for us as well as it works for them. Then we can have an information society in which everyone benefits rather than an information technology society that just reinforces the status quo. It might start with our bins but it won’t end there.

Mar 11

Sutton Bookshare is not a library

Sutton Bookshare is a project that I’ve been designing for Sutton Council. It’s a website that lets local residents list their books on a website and then share with each other.

Bookshare is part of a wider project called Sutton Open Library that’s about opening up the library service to innovation. The whole project is funded through a grant from NESTA (a charity distributing lottery money) under their Make It Local funding scheme.

As well as the book sharing website, Sutton Open Library also opens up the main library service’s database so that independent software developers can access it and build their own apps for it.

Is Sutton Bookshare an attempt to cover for library cutbacks?

Many people have asked this question. It’s a fair question to ask.

While I work temporarily as a contractor for Sutton Council I do not speak for the council. So these are my personal views.

It would be very hard to see how Sutton Bookshare could substitute for any significant part of the library service. It has certainly not been designed with that aim in mind. In fact, the whole design direction of the project has been led by the principle that Sutton Bookshare is not a library. Every time I start transplanting library concepts into Bookshare I remind myself that Bookshare is unique and different and needs to work in a very distinct way.

One of the aims of Sutton Bookshare has been to make books that aren’t available in the library available for people to borrow. Most of my own books aren’t in the library service, so if you’re interested in design theory, urbanism and software development you now have access to books that you didn’t have previously.

Another aim of Sutton Bookshare is to build and reinforce personal relationships and social networks. Libraries can do this to an extent through clubs and other activities but the core library services are about borrowing items from the library, not other people. When you lend or borrow something in Sutton Bookshare you don’t just exchange a book, you get to meet someone who lives or works locally and almost by definition has a shared interest with you.

Sutton Bookshare also improves the library service. When you look at the page for a book in Bookshare you get a direct link to that book’s page on the main library service’s catalogue. This gives you options: Borrow it in Bookshare or borrow it from the library. For many people it will be more convenient to borrow it from the library. Bookshare provides another way to find books that are in the main library service.

What Sutton Bookshare doesn’t do

Sutton Bookshare isn’t a library.

Bookshare only lets you borrow books, hence the name. No CDs or DVDs.

Bookshare doesn’t give you a desk where you can sit down and work for a few hours in a quiet atmosphere.

Bookshare won’t let you catch up on the day’s newspapers or recent magazines.

You can borrow my books but you can’t pitch up in my living room for the afternoon. Sorry about that.

Open data makes the libraries better

The open data side of the Sutton Open Library project is all about improving the library service. We’re doing this by giving software developers the opportunity to build apps that help people find books more easily. This is nothing to do with cutting back the library service. It’s about making the library service better. Sutton Council has been fortunate to be able to attract outside funding for this work that will not just pay off in Sutton but will help to set standards and make similar work easier in other councils.

All software developed under this project is free and open source. Anyone can use or modify it themselves for any purpose. The code is here on Github.

So what about the cutbacks then?

Like all councils, Sutton Council is reviewing its services in the light of funding cuts from central government. This includes the library service. If you’re a local resident and you want to get involved in the discussions about the future of the library service you can start here on the Speak Out Sutton website, the council’s consultation site. I have no more information about this process or influence on it than any other local resident.

But it’s my view that the scale and nature of Sutton Bookshare makes it a useful supplement for the library service but not a substitute for any part of it. My hope is that Bookshare becomes a useful thing in its own right. It’s more like a club than a public service, albeit one that’s organised by the council rather than independently. I also hope that the open data work on this project will make libraries more accessibile than they are at the moment.

A postscript for Amanda Craig

I’ve just listened to the discussion on BBC Radio 4′s PM programme with Sutton Council’s Daniel Ratchford and the author Amanda Craig.

Amanda seems to hold some odd views about books.

The first is that books are far too precious to lend. While I’d agree that books are definitely valuable in the sense that they’re useful and enjoyable, they don’t do any good sitting on your shelves. So I’ve listed 130 of my own books and while I’d definitely like them back, if I lose the occasional one then I can stand the loss. I offer things to share because I know that most people are honest and responsible. If you believed otherwise you probably wouldn’t engage in almost any kind of relationship, personal or commercial.

Amanda also thinks that sharing books is tantamount to stealing from authors. This is because when you borrow a book from a library the author gets a small payment (the Public Lending Right) but when you share a book with a friend the author gets nothing.

I think this is terribly narrow-minded.

Sharing books on a relatively small scale doesn’t threaten authors. People stopping reading books threatens authors. Sutton Bookshare is a small project that in its own way will help people discover and read new books. Authors will benefit because those same people will be far more likely to visit a public library or buy books subsequently. It’s not too much of a stretch of the imagination to think that someone might borrow one book by an author on Bookshare and then buy another.

When you’re looking at a book page on Sutton Bookshare you’re also three clicks away from buying that book on Amazon. Bookshare links directly to Amazon’s search for that book.

The threat to authors comes largely from other things. It comes from the time people choose to spend doing things other than reading books because they now have more options. Watch YouTube or read? Fool around on Facebook or read? Play computer games or read? Listen to internet radio or read?

A project like Sutton Bookshare and Sutton Open Library is the wrong target. We’re getting people hooked on books not taking money out of authors’ pockets. Authors, libraries and of course readers will benefit.

I’ve always spent a lot of money on books. I probably always will because it’s very unlikely to be convenient or even possible for me to get all the books I want through borrowing from people or libraries. Authors should be scared of Facebook and World of Warcraft not book sharing.

Feb 11 official crime maps — there should be a law against it

It’s always good when open data makes the headlines, albeit slightly for the wrong reasons today. Nonetheless, too much traffic to our website is a problem we’d all like to have. It shows public interest if nothing else. After all, who wouldn’t want an easy way to find out how much crime is on their street and in their neighbourhood?

But before we fall over ourselves to be grateful for this latest attempt at transparency we should exercise more than a little caution.

This won’t be news to anyone who thinks seriously about data, but a map is a visualisation, not the data itself. It’s one way of representing the underlying data. In as much as the data is accurate, complete and relevant, the website is simply giving us a single way to look at it that’s already been decided for us. No matter how often we’re reminded that the map is not the territory (and let’s be honest, most people have never heard that saying, let alone considered the issues in any depth), if you’ve only got the map it might as well be the territory. Psychologically, the two become conflated.

Perhaps apocryphally, Stalin said that it’s not who votes that counts but who counts the votes. Likewise, we should be hugely cautious about giving too much weight to official visualisations of data. As the policing minister Nick Herbert wrote today (my emphasis):

We live in the age of accountability and transparency. The public deserve to know what is happening on their streets, and they want action. By opening up this information, and allowing the public to elect Police and Crime Commissioners, we are giving people real power – and strengthening the fight against crime.

So what we’re looking at here isn’t a value-neutral scientific exercise in helping people to live their daily lives a little more easily, it’s an explicitly political attempt to shape the terms of a debate around the most fundamental changes in British policing in our lifetimes.

Transparency isn’t wrong. It’s absolutely vital to make a meaningful contribution to public debate, but we need to distinguish pseudo-transparency from the real thing. Spatial visualisation and analysis is enormously difficult to get right and even thoughtfully-designed visualisations require a fair bit of understanding to interpret correctly. Slap it on a map works fine when you just want to see where your local recycling centres are, but as soon as you start to classify crimes by type and bound them into streets and neighbourhoods you’re into the realm of professional spatial analysis. You need to know what you’re doing and have access to tools that enable you to shift category and spatial boundaries to account for anomalous effects. The newspapers that have run lists of the most crime-ridden streets in the country today might want to consider the fact that longer streets will on average have more crime than shorter streets, just to take one simple example of a relevant factor that’s not accounted for if you want to visualise this data in that way.

Whether is trying to pull a fast one on us or is simply naive about the possibilities for doing something meaningful for a general audience with this data, the result is the same: plenty of heat and very little light. Mark Monmonier’s How to lie with maps provides a good starter text for the myriad ways in which maps can deceive, intentionally and otherwise.

On a more positive note, we’re also getting the data itself to use. This is a good thing, in as much as the data itself is, as stated above, accurate, complete and relevant. Unfortunately, it’s not. It’s derived data that’s already been classified, rounded and lumped together in various ways, with a bit of location anonymising thrown in for good measure. I haven’t had a detailed look at it yet but I would caution against trying to use it for anything serious. A whole set of decisions have already transformed the raw source data (individual crime reports) into this derived dataset and you can’t undo them. You’ll just have to work within those decisions and stay extremely conscious that everything you produce with it will be prefixed, “as far as we can tell”.

£300K for this? There ought to be a law against it. Worse than useless, it’s thoroughly misleading. In future, we need fine-grained datasets for these kinds of applications and a big head start (six months?) between publishing official data and the commissioning of official expensive projects around it to ensure that everyone really understands what can and should be done with it.

Jan 11

Open data for all

There are five types of potential users for open data and data-driven apps:

  1. data experts and computer scientists who can use semantic web technologies;
  2. software developers who can use XML, JSON, etc.;
  3. power users who can use CSV, spreadsheets, RSS, KML/Google Earth, perhaps Yahoo Pipes;
  4. general users who can use a web browser;
  5. offliners who need printed materials, ambient displays, public screens etc.

Most of the focus seems to be on providing data for data experts and developers so they can build apps for general users and power users. We need more data suitable for power users to use directly and more apps for offliners. We’re all offline sometimes.

My own app for offliners is QR Code Posters which will print a poster from any RSS feed. See how it can be used here.

Sutton Open Maps caters for general users, power users and developers by showing draggable Google Maps of local features along with KML (Google Earth), XML and JSON downloads on the same page. Whether you want to just find a local recycling centre, download the data into Google Earth for a school project or build your own app from the data, you’re covered. (It’s open source, too.)

This post started life as a comment.

Jan 11

TfL’s information doesn’t want to be free

I’m a big fan of London’s Barclays Cycle Hire scheme. I praised it when it was introduced, I created a free API service for developers to help them get live data about bike availability to make useful apps for people, I built a realtime 3D visualisation of bike availability and I even wrote a simulator to help me better understand bike movement patterns. I still think it’s a great system and I’m keen to do what I can to help people use it and to make it work better.

So when Boris announced that the scheme had just passed its one millionth journey milestone it seemed like a good time to ask Transport for London for the journey data. It’s an easy enough job: Just a single database query to fetch the times, origin and destination of each trip. If I could load this data into my simulator I might be able to see where extra bikes and docking stations might be needed. I put in a Freedom of Information Act request, confident that I’d have the data within the 20 working days limit required by law.

That was three months ago on 8 October. I’m still waiting.

The good news is that the data has just been made available in TfL’s developers’ area and some people are already starting to do interesting and useful things with it. But behind that happy fact is another example of a public body deciding to completely ignore their Freedom of Information Act responsibilities and the rights of an applicant in pursuit of its own perceived interests.

Data delayed is data denied

Under the law, public bodies have got 20 working days to reply either with the information requested or to claim an exemption. The time limit is there for a good and obvious reason: Without it, public bodies can string an applicant along indefinitely, and with many requests being time-sensitive this can often past the point where the information would be useful.

Fortunately I didn’t have a specific deadline for using this data but it certainly would have been more useful to me sooner rather than later. I could have been working on it for two months by now. And if TfL had been keen for other developers to use it, they could have had it too. Some developers were keen to get hold of it for the Open Data Hackday on 4 December last year but that came and went without any sign of the data.

So why was the data delayed? I estimate that there would have been less than two hours work to produce it and send it to me, or to put it on an open website where anyone could download the file.

“Your free information is in this locked box. Sign this contract and if we like what you’re doing you can have it.”

The answer lies in TfL’s desire to wrap the data in a complicated contract rather than make it available to me or anyone else directly and legally unencumbered. This might make sense in the context of some data and some data users but it’s directly inimical to the aims and indeed the law of freedom of information. The data in TfL’s developers’ area isn’t open data and it’s not available to everyone. As the site says:

Please complete the registration form below to use our syndication feeds. Before we give permission to use any feeds, we need to know how they will be used, where they will be used and how many people are likely to view them.

So why should anyone have to apply for permission to get access to their freedom of information answer? Why not just send it to the applicant?

The Information Commissioner, who regulates public bodies’ compliance with the Freedom of Information Act is quite clear that information must be supplied regardless of the identity and motives of the applicant. His guidance (PDF) states:

A request therefore has to be considered on the basis that it could have been made by any person; the identity of that person is not a material consideration when deciding whether or not to release information. It is for this reason that we do recommend as good practice that requests under obvious pseudonyms should normally be considered unless there is reason to think that any of the matters below need to be taken into account.

There follows some general exceptions regarding vexatious requests, people requesting their own personal information and costs issues, none of which apply in this case.

On the issue of the applicant’s motives:

There is also no specific reference in the FOIA to the principle that requests for information must be considered without reference to the motives of the requester.

However, there are no references in the Act indicating that anyone can be asked to provide a reason for requesting information and it is from this absence that the principle [of disregarding the applicant's motives] is drawn.

The Information Commissioner then quotes the Lord Chancellor’s code of practice on freedom of information:

Authorities should be aware that the aim of providing assistance is to clarify the nature of the information sought, not to determine the aims or motivation of the applicant. Care should be taken not to give the applicant the impression that he or she is obliged to disclose the nature of his or her interest as a precondition to exercising the rights of access, or that he or she will be treated differently if he or she does (or does not).

But if I want to get a response to my FOI request from TfL I am asked to enter into a contract with them whose terms include:

2.1.2 [You shall] only use the Transport Data in accordance with these Terms and Conditions and the Syndication Developer Guidelines, and not use such information in any way that causes detriment to TfL or brings TfL into disrepute. The rights granted to You under these Terms and Conditions are limited to accessing and displaying or otherwise making available the Transport Data for the purposes stated by You in Your registration.

So not only is TfL’s contract explicitly asking me to state my motive as a precondition of access, it also constrains me from using the information for any other purpose and arguably prevents me from using that information to criticise TfL, thereby causing it “detriment” or bringing it into “disrepute”. If I don’t agree to this they can deny access altogether and if I subsequently break the agreement in their view they can revoke access. This is a funny kind of free information.

The Freedom of Information Act is designed to enable scrutiny of government. It’s inevitable that some information requested may cause embarrassment to the public body providing it or even bring it into disrepute. If the law is going to be workable at all, public bodies must consider each application on its merits alone without concerning themselves with the applicant or their motives. To do otherwise would allow public bodies to effectively pick and choose which requests they answered. TfL’s decision to require me to enter into an extremely restrictive contract with them to get a response to my freedom of information request is applicant and motive discrimination by the back door. It’s not something that should be tolerated from TfL much less adopted by other public bodies as a way to weaken FOI applicants’ rights. Free information should not come wrapped in a restrictive contract wall. That’s why I won’t be accepting TfL’s terms and I’ll simply have to leave the analysis of this Cycle Hire data in the very capable hands of others.