Oct 09

Great council websites aren’t enough. We need 1% for open data.

BBC News has run a government open data story today featuring Mash the State, Openly Local, Pic and Mix and Socitm. There’s probably not much there that will be news to avid open data followers familiar with these projects but by all means go and have a read. While there has been much talk and a fair bit of action on open data lately (not least Sir Tim Berners-Lee’s appointment to advise the government) the situation on the ground is still that most councils aren’t embracing open data and show little signs of interest.

Media reports often feature cutting-edge projects such as Kent County Council’s Pic and Mix and can distort the public perception away from the reality that the great independent civic websites using public data are mostly having to scrape and steal it. Very few councils will even acknowledge them, let alone co-operate with them.

Our campaign to get councils to create RSS feeds intentionally gives them the easiest possible first step into the open data world — just put up an RSS feed of your news and we’ll be smiley. Yet only 27% of councils even manage that.

The relatively greater movement of councils towards social media sites like Facebook and Twitter, while not to be confused with genuine open data initiatives, is encouraging because it shows many councils’ willingness to engage users online in the places where they want to go rather than insisting that the only council web activity will happen on the council’s own site.

Despite Socitm’s Better Connected process and laudable efforts among many councils to improve their websites, the unavoidable conclusion is that councils often aren’t the best organisations to create e-government services for the public. Councils’ own websites will decline in importance accordingly.

As more people get connected and want to do more with their councils online, having a great council website won’t be enough. That’s why today Mash the State is calling for councils to dedicate 1% of their web budgets for open data projects.

Why is this?

1. People want choice. As we see with social media and RSS users, many people want and expect to be able to interact with their councils online using the websites and tools which they prefer and with which they’re familiar. And why shouldn’t they? For any council serious about engaging the public, saying “It’s our way or the highway” will no longer do.

2. Boundaries are barriers. Council boundaries and departmental responsibilities are inevitable artifacts of organising collective work but users often don’t know who’s responsible for dealing with their issue, nor do they particularly care. Why should they? IT can give citizens the ability to navigate civic life in ways that are often agnostic to the bureaucratic structures of government and to work with multiple government organisations and departments simultaneously and seamlessly. Fix My Street provides a great user experience because users don’t need to know which council is responsible for the fault they’re reporting nor find the relevant contact point within that council. It works the same way for everyone, everywhere. No single council could match that level of ease and utility. The only way to improve on Fix My Street would be to create a similar system with national scope.

3. Computing is growing, expectations are rising, budgets are shrinking. The relentless pace of technological development leaves councils running (and losing) a frantic race to catch up. Things will only get worse for councils that make it hard for people to reuse their content in ways that suit them and create third-party interfaces to their services. Just as the council gets its new website (which may have to last fundamentally unchanged for 3-5 years) along comes the next big thing and already it’s outdated. People want videos and podcasts, to be able to follow various council activities across a range of social media sites, they want iPhone apps and mobile web capability. We don’t know what they’ll want tomorrow but we can reasonably assume that councils often won’t have the skills in house to satisfy those expectations nor the budgets to hire specialists to do it for them.

4. Councils don’t have a monopoly on great ideas. With the best will in the world, when councils create web services for the public those services will reflect the ways in which the council wants to interact with the public. When the public create web services they reflect the ways in which they want to interact with the council. Which is likely to provide a better user experience? There is an enormous inertia within councils that stifles innovation, often because of genuine risks or opportunity costs. Usually it takes outsiders to rock the boat. When it goes wrong, some individuals may have wasted their time but the public has lost nothing. When it succeeds, everyone benefits.

The age of technological catch-up for councils is rapidly drawing to a close. Technologies to move content and data between applications and websites are mature and widely-implemented. Councils can no longer bury their heads in the sand and continue to act as if they believe that they have all — or even most — of the answers. The future of local e-government is letting people “pick and mix” the content and services they require using the tools and systems they prefer rather than insisting that everything must happen on councils’ own websites.

For those councils already moving strategically down this path — keep doing what you’re doing. For those that have dipped their toes in the water, take a look at the bigger picture and see where the web culture is heading. How many individuals now have their own websites as opposed to social media presences, remotely hosted blogs and Flickr accounts? How many new and recent technologies have emerged in which you see potential but don’t have the resources even to investigate, let alone implement?

Councils need stand-alone open data projects with their own resources and budgets. Lumping it in with general website work has demonstrably failed to give open data the priority it deserves.

Councils may spend that 1% in various ways. For some it will be an opportunity to create feeds and build inbound and outbound APIs, to integrate with third-party websites and services that have proven track records that provide real value. Councils could build relationships with local developers that are interested in government projects, supporting civic hacking groups where they exist and helping to create them where they don’t. “Hack the Council Day” could be a regular feature in the borough calendar. Or they could simply donate some of that money to new or existing projects that inspire them.

For the 74% of councils that still don’t have a single RSS feed, getting that done might be a good place to start.

Apr 09

Why I’m throwing down the gauntlet to our councils over RSS feeds

You’re free to republish this article under the Creative Commons Attribution 2.0 UK licence with credit and a link to Adrian Short / Mash the State

Today I connected 66 councils to their citizens by making it easy to subscribe to their news by email. It took me around ten minutes. I’d say this was a fairly good use of my time in terms of the ratio of effort to value produced, but I can’t claim to have done it single handed. What made it possible is that all 66 of these councils serve an RSS feed from their websites — and they’re the only ones in the country that do. Hooking those feeds up to FeedMyInbox through the council pages at Mash the State was a simple matter of dropping a single web link into a template and pushing it to the live site. Job done.

RSS is a simple way of getting data out of a website and into another program. The technology is ten years old and RSS feeds are ubiquitous on blogs, on mainstream news media websites and in Web 2.0 applications. The three leading web browsers — Internet Explorer, Firefox and Safari — all contain built-in RSS readers. Yet despite running websites costing tens of thousands of pounds annually each, only 15% of UK councils bother with RSS. Nothing could be more symbolic of large parts of government’s unwillingness to think beyond the confines of their own websites than making it practically impossible to receive basic local council information like news and events except by taking a trip to to do it on the council’s own terms.

The ten minutes it took to emailify those 66 councils compare quite unfavourably with probably a similar number of hours I’ve spent trying to scrape Sutton Council’s news into a database, and from there through Delicious into RSS and Twitter. Writing screen scrapers — programs which extract text from web pages and turn them into structured, reusable data — is sometimes tricky but Sutton’s news is trickier than most. The news archive serves inconsistent page structures and even dynamically changing URLs to compete with. I vowed never to write another scraper, though as we’ll see, that’s a promise I soon had to break.

Screen scraping and copyright infringement are the dirty not-so-secrets of the civic hacking world. Show me a useful, innovative third-party civic website and I’ll most probably be able to show you the terms and conditions that were ignored and the data that was taken and repurposed without permission or legal licence. Similar behaviour is not unknown in the public sector itself, in some cases because government organisations are recycling that very same stolen data from third party applications into their own websites. The recent Rewired State National Hack the Government Day saw some incredibly inspiring, innovative and useful projects produced in very short order. How many of these projects didn’t involve citizens jailbreaking their own government to get the data they’ve paid for? What kind of society not only massively impedes but actually criminalises — in principle if not in practice — citizens devoting their own time, skills and money to write software to improve democracy and public services? Our society, it seems.

This has to stop. Hackers have shown their ability and willingness to surmount technical obstacles and run legal risks to get the data they need but less technical citizens simply cannot. No-one should have to. A rich, technologically-advanced and supposedly forward-thinking society such as ours should make citizens’ access to government data so commonplace that it doesn’t deserve comment. No technical wizardry required. No legal minefields to navigate. Just all the data served through common protocols with open licences that permit, well, anything. Then we can focus our time and energy on the considerably more interesting higher-order opportunities that come from actually using government data, not just getting hold of it.

Last week I launched Mash the State, a national campaign to get government data to the people. It’s not a new idea but our method is. We’ll be setting up a series of challenges to the public sector, asking one group of public bodies at a time to release one specific set of data. Our first challenge asks all local councils to serve up an RSS news feed by Christmas. I wouldn’t have bet good money in 2003 that by 2009 370 councils would still be without RSS, but here we are. I’ve thrown the gauntlet down and I’m pleased to see that a couple of hundred people have signed up to our website or followed us on Twitter to help make this happen. The councils have got over eight months to do what in most cases will not be more than half a day’s work to serve RSS from their websites. Others less fortunate will have to persuade their content management system suppliers to enable this feature for them. All have got plenty of time to perform this technically trivial task in time to give the public a small but highly symbolic Christmas present that shows that government in this country is prepared to trust its citizens with their own data.

As for my promise never to write another scraper, it didn’t last long. The very first task to build Mash the State was an hour spent writing a scraper to tease a list of councils from a government website. Join us and help to hasten the day when no-one will ever have to do anything like that again.