I've recently had the pleasure of taking part in two hacker events organised within the context of healthcare and, specifically, the UK's National Health Service (note for Americans who missed the opening of the Olympics: it's the state sponsored monopoly of "death panels" we use to decide which of our senior citizens are to be terminated). To be clear (and more seriously), I am using the term "hacker" in the way it is used within Information Technology circles: a hacker is a person with a passion for exploring and solving problems through writing and sharing software.

Over last weekend I was in Liverpool to attend the second NHS Hackday - a gathering of doctors, designers and developers. Here's my buddy and the originator of the event, Carl, providing more details about the aims and objectives of NHS Hackday:

After this, on the Monday and Tuesday, I was invited to take part in a hackathon organised by the UK's Department of Health. Officials and big-wigs from the DoH provided a diverse group of ten developers and designers with a problem: to build an "innovation platform" in 48 hours. Said big-wigs were on hand to answer questions, provide guidance and give feedback throughout Monday. On Tuesday afternoon we presented our work to a couple of them.

Both events were positive, energetic and fun. Unfortunately, they also led me to the conclusion that Information Technology (IT) provision within the National Health Service (NHS) is Fucked Up Beyond All Recognition (FUBAR). While I concede this assertion is intentionally couched in a provocative way, I doubt the sentiment will come as a surprise to those working within the NHS.

Taking each event in turn, I'd like to explain why I've reached this conclusion since it's important to back up such negative assertions with concrete evidence (unfortunately, these two events provide plenty of material).

NHS Hackday - The ePortfolio Data Liberation Front

The modus operandi of the weekend was simple: anyone could pitch an idea as a lightning talk on the Saturday morning (lightning talks are strictly time bound - in this case, two minutes), people came together to investigate and work on the most interesting things presented and, on the Sunday afternoon, presented the fruits of their labours to a panel of experts and big-wigs from the worlds of health and technology. If they were lucky, they got a prize.

Somehow, I got involved in a group called the ePortfolio Data Liberation Front...

Every junior doctor working as a physician within the UK's hospitals has to demonstrate that they're at least competent given their level of training. As a result the Royal Colleges require trainees to undergo observation and assessment.

This is essential for ensuring patient safety (I, for one, wouldn't want to be treated by a doctor who couldn't show the required competencies in order to practice). It is also an important pillar of a hospital doctor's continuing professional development: a mechanism for feedback, advice and noting areas for improvement.

The results of this process, the evidence the doctor needs in order to progress to the next level of training, are currently stored in one mandatory location: the nhs-eportfolio. Trainees are obliged to pay £18 a year to use this service, usually as part of their annual membership fee for whichever of the Royal Colleges they happen to belong to. The ePortfolio's about Page states that they have over 35,000 trainees registered. So at a conservative guess (35000 times 18) their annual income is at least £630,000.

One would hope that such an important and widely used service would be great value for money and the beneficiary of continued investment, improvement and updates.

Unfortunately, this is not the case.

It appears that ePortfolio is loathed by its users to the extent that there are even blogs about how awful it is. Users complain that it's unintuitive, disorganised, ugly to look at and, at times of peak usage, slow. Results from a quick straw poll of doctors and consultants at the hack day were unanimously negative (and sometimes venomous).

I happened to meet the author of the afore mention blog, LJ and another doctor, Marcus Baw, on the first morning of the hack day. We discussed the various problems with the current offering and quickly came to the conclusion that helping trainee's to get to their data was a fundamental issue. Unfortunately, there appears to be a conflict between what is in the ePortfolio's best interest and the interests of their users.

If a trainee can easily download their data (after all, who else's data is it?) in a widely used and open format then there's nothing to stop them from taking their data to a different service. This is obviously bad news for ePortfolio who are sitting pretty as a mandatory monopoly. They are not exposed to the pressure of dealing with a cheaper, better, faster competitor.

We asked ourselves, how hard would it be to extract data from the ePortfolio and dump it in an industry standard structured data format like, for example, JSON? The answer is that it's both easy and hard.

Given a doctor's credentials it's very easy to fool ePortfolio into letting you in so you're in a position to automate the "scraping" (a colloquial term to mean downloading, parsing and cleaning) of data. Here's how I did it using Python (it would have been a lot shorter had their log-in form not been such a mess):

import requests
import lxml.html
import getpass

LOGIN = 'https://nhseportfolios.org/Anon/Login/Login.aspx'

def getCredentials():
    """
    Asks for user's credentials and returns a (username, password) tuple.
    """
    username = raw_input("Username: ").strip()
    password = getpass.getpass("Password: ").strip()
    return (username, password)


def signIn(username, password):
    """
    Given the username and password of a user will return a correctly
    configured requests.session object to use for subsequent requests.

    This is a slight faff because they include a token in the form to avoid
    naive automated POSTing attacks so we have to HTTP GET the form, extract
    the token and POST back for the session cookies. I'm being lazy and
    returning *everything* they expect in the request (but I guess they're
    just checking the token).
    """
    # Grab the login page to get the form values we need.
    raw = requests.get(LOGIN)
    session_left_slice = raw.headers['set-cookie'].find('=') + 1
    session_right_slice = raw.headers['set-cookie'].find(';')
    session_id = raw.headers['set-cookie'][session_left_slice:session_right_slice]
    html = lxml.html.fromstring(raw.text)
    db_viewstate = html.cssselect("input#__DATABASE_VIEWSTATE")[0].value
    ev_validation = html.cssselect("input#__EVENTVALIDATION")[0].value
    # Create the form payload, many of these fields are empty, but using them
    # so we look *exactly* like a browser.
    username_key = 'ctl00$ContentPlaceHolderMain$ucUserLoginBox$ePortfolioLogin$UserName'
    password_key = 'ctl00$ContentPlaceHolderMain$ucUserLoginBox$ePortfolioLogin$Password'
    login_button = 'ctl00$ContentPlaceHolderMain$ucUserLoginBox$ePortfolioLogin$LoginButton'
    form_payload = {
        'ctl00_ScriptManager1_HiddenField': '',
        '__LASTFOCUS': '',
        '__EVENTTARGET': '',
        '__EVENTARGUMENT': '',
        '__DATABASE_VIEWSTATE': db_viewstate,
        '__VIEWSTATE': '',
        '__SCROLLPOSITIONX': '',
        '__SCROLLPOSITIONY': '',
        '__EVENTVALIDATION': ev_validation,
        username_key: username,
        password_key: password,
        login_button: 'Please wait...',
        'ctl00$ContentPlaceHolderMain$ucCodeLoginBox$txtLoginCode': ''
    }
    session = requests.session()
    session.post(PATHS['login'], data = form_payload)
    return session

if __name__ == '__main__':
    print "Enter user credentials"
    username, password = getCredentials()
    print "Validating user credentials"
    session = signIn(username, password)

So far, so good. Given the session object created above, it's simple to download the HTML as if you were a logged in user:

# Get the user's home page.
home = session.get('https://nhseportfolios.org/Auth/SitePages/Trainee/Default.aspx')

Unfortunately, the HTML that ePortfolio returns is terrible.

It is inconsistent, un-idiomatic (I saw at least one nested table), contains little or no semantic stylesheet or id attributes and is a veritable tag soup of awfulness. As a result, extracting the important data from the raw HTML was a tale of woe. It was the cause of most of my frustrations over the weekend. I'd have had more fun sticking forks in my eyes.

My immediate reaction was a feeling of fremdschämen. Surely the developers at ePortfolio were aware that from both a non-technical "external" point of view and a technical "internal" view (of their HTML) the site sucked harder than a black hole? How could who-ever is in charge of ePortfolio allow this to happen?

Then I remembered that the reason ePortfolio is allowed to continue in this state is that they have no external pressure for them to change. Whoever decided to make the system a mandatory monopoly was, not to put too finer point on it, hugely misinformed, unconcerned with trainee doctor's requirements and foolish.

Happily, working with such a smart, motivated and jolly couple of doctors was the highlight of the weekend because they are a beacon of hope. LJ and Marcus understand technology and care passionately that it becomes a force for good: improving healthcare in the UK rather than continuing to be the root of so much frustration. Obviously, people like them should be in charge of making IT precurement decisions.

I'm also pleased that we were awarded second place by the judges. A better prize would be to open-source the ePortfolio and give trainee doctors free access to their own data.

For LJ's perspective on the weekend check out her blog post and the resulting exchange of comments with one of ePortfolio's developers.

Sonar - Surfacing Innovation from within the NHS

Arriving home, hot foot from Liverpool, I was immediately travelling down to London for, well, I wasn't sure what...

Ten developers and designers gathered at the Design Council in London under the auspices of the Department of Health. We also met managers from the health care sector. The managers had decided that the NHS needed an "innovation platform to surface best practice" (or something like that). Once we'd parsed the buzzword bingo it became clear that they wanted something akin to Hacker News (the premier website where hackers and tech start-ups share innovation) or Reddit (a news website containing content submitted by its members). In both instances, attention is given to those stories that the site's members vote as most interesting.

Given the amount of "surfacing" jargon going on in the room we (the hackers) decided to create NHSonar, a site for surfacing innovation within the NHS.

It was a lot of fun and we managed to build something close to the manager's original vision (you should be able to submit innovations upon which other users could vote and comment with the most popular innovations on the front page). All this was done within two working days.

So we come to the crux of this blog post: expensive, badly written software that's foisted on healthcare professionals via a top-down style of management results in the terrible state we're in right now. Happily, despite having their collective legs pulled about the over use of business buzzwords, enlightened healthcare managers understand that "surfacing innovation" is an essential part of stopping the rot.

If a handful of hacker-types working in an agile manner with open-source tools can create an innovation site in 48 hours, can you imagine what might be achieved if the government, senior healthcare professionals and managers engaged with the free software community?

One positive outcome would be managers that don't speak "buzzword" - they'd be fluent in "techno-babble" instead. Perhaps, on second thoughts...

:-)