Monthly Archives: April 2014

#FHIR DevDays, Amsterdam in November

in November, I’ll be at the FHIR Dev Days:

The International HL7 FHIR Developer Days 2014 will be held on November 24th – 26th in Amsterdam. The HL7 FHIR Developer Days are an intensive hands-on education opportunity aimed at those involved in the design, development, implementation, deployment and support of systems that use (or will use) the HL7 FHIR standard.

The FHIR Developer Days combines three opportunities:

FHIR connectathon/hackathon
A place for implementers, whether new to FHIR or with previous experience, to test and develop their FHIR implementations in a collaborative environment. It’s a chance to work with the specification surrounded by others doing the same thing, side by side with experts to answer any questions.

Education
The three project leads of FHIR, Lloyd McKenzie, Grahame Grieve and Ewout Kramer as well as other FHIR experts will be present to deliver training, guide the connectathon and work one on one with attendees to discuss issues. There is also the possibility to follow the HL7 FHIR Hands on-training course facilitated by René Spronk.

Networking
This meeting is also about building a community of peers – enabling connections with other implementers that one can use after the meeting itself.

Download the complete program here: Program International FHIR Developer Days 2014

Furore are a deeply committed participant in the FHIR development process – Ewout Kramer, from Furure, is one of the core project team, and directly responsible for the C# reference implementation, a number of key resources including Patient and Questionnaire, and makes many other contributions to the specification.

I’m looking forward to this, and I hope to see you there.

Update: check out this interview with Ewout and Lloyd about the DevDays

Setting the Hay on Fire

Quoting from Australia’s Birthstain: The Startling Legacy of the Convict Era, by Babette Smith (Amazon Kindle link):

In 1847, John Hobbs set fire to a haystack as a means to change his life.

In a statement to the police, Hobbs said that he and his mates, Tom Webster and Robert Lewer, had to beg the money for the toll to cross Kew Bridge on London’s outskirts that morning. They spent the day wandering around the farmland on the other side. Hobbs did not say whether they were looking for work. Nor did mention whether they had spent the balance of their begging on a pint at the put, although he was to claim later that he was drunk that evening. Whatever their real purpose, as dusk fell they converged on a farm they had passed earlier in the afternoon. There, to the puzzlement of the farmer, Frederick Piggot, and others who testified to what happened, they laid some faggots under a large haystack, then used Hobbs’ pipe to light three lucifers so each could participate in in setting it ablaze.There was no personal grudge involved. They were unknown to Farmer Piggot, and neither he nor any of the witness had ever seen them in the parish of Richmond before.

The local police sergeant who had been called to the fire found the trio waiting at the police station for his return. “We’ve come to give ourselves up for setting fire to the hayrick’, they told him. “I’m willing to hear anything you might have to say’, he replied, but cautioned that it could be used in evidence. Nevertheless, they insisted. So, as the sergeant put it, ‘They were all three together – as one made a statement, I asked the others if they agreed with it, and they said “Yes”.’

‘We were the whole day in Richmond without anything to eat’, said the three young men, after first describing how they begged for a bridge toll to cross the river. ‘Our lives for some time past have been spent in such misey and poverty we were determined, when we left London this morning, to do something to alter it.’

They were shipped to Van Dieman’s Land (Tasmania) – twenty years for Webster, fifteen for Hobbs, and ten for Lewer. There, John Hobbs would meet and marry an Irish girl shipped for the same crime. in fact, many of the convicts shipped to Australia were transported for the same grounds. Check out, for instance, the passenger list from the Dalhousie, a convict ship from 1863, where an unexpectedly high number of convicts were shipped for Arson, or “Firing a stack”. (Some, in fact, had to investigate themselves since the local police were too lazy).

Btw, this has nothing to do with Healthcare Interoperability – I posted this to provide more information relating to the picture that David Hay just posted, and his terse comment in his first blog post. Meanwhile, I spent yesterday afternoon having fun setting Serrated Tussock (not quite hay, but close) on fire on a farm in King Valley:

Serrated Tussock

p.s: Most of the convicts were real criminals (as you can see from the crimes listed for the Dalhousie), but often many were trapped by poverty and unemployment. Initially, being shipped to Australia was a truly terrible punishment, but by the time Gold was found near Ballarat, the only downside was dislocation from family. You got hard labour either way, but in Australia, after your sentence, then you could be free in a land of full employment…

p.p.s: For my English friends – after visiting England, my question became “Your predecessors shipped the criminals out here”?

New CDA Stylesheet

From Keith’s blog:

A few weeks ago, Josh Mandel identified problems with the sample CDA Stylesheet that is released with C-CDA Release 1.1 and several other editions of the CDA Standard and various implementation guides. There are some 30 variants of the stylesheet in various HL7 standards and guides.  A patched version of the file has been created and is now available from HL7’s GForge site here.

Note for Australian users: if you use the NEHTA authored stylesheet, this release doesn’t affect you.

#FHIR: Subscribing to a server

This question has come up several times lately, so I thought I’d show how to subscribe to a server.

Below is pseudo code for subscribing to a server. It assumes that you have [base] – the address of the server, and a local store for settings (could be an ini file or a database), a client that can do http, and a bundle class that matches the semantics of a Bundle (e.g. an atom feed or it’s JSON equivalent), and load from the http outcome. The logic is pretty simple:

  • get the latest _history from the server
  • follow the next links till they run out, aggregating the content
  • store the feed.updated from the first response to use for next time you do it
procedure Subscribe(base)
  client = connectToServer(base) // and maybe get the conformance statement
  loop
    sleep(however long you want)
    feed = downloadUpdates(client)
    foreach entry in feed.entries *working backwards* // see note below
      process entry however you want
  until it's time to stop
procedure downloadUpdates(client)
  lasttime = store.getProp('LastTime')
  master = new Bundle()
  next = null
  i = 1
  do
    log('Downloading Updates (page('+i+')')
    if (next != null)
      feed = new Bundle(client.fetch(next))
    else if (lasttime != null)
      feed = new Bundle(client.fetch(base+'_history?_since='+lasttime))
    else
      feed = new Bundle(client.fetch(base+'_history')
    master.entries.addAll(feed.entries)
    if (next == null)
      store.setProp('lastTime', feed.updated)
    next = feed.link['next']
    i++
  until next == null
  return master

Notes:

  • for bonus marks, you could add a _count parameter and ask for as many resources per page from the server as you can get – that reduces the load on the network a little, and the lag time some
  • The outcome of the fetching loop is a single bundle that lists all the updates on the server since the last time in reverse chronological order. Whether you have to process in the correct order depends on what you are doing
  • If you want to see a real implementation of this, look at the class org.hl7.fhir.sentinel.SentinelWorker in the FHIR SVN (see under build/implementations/java)

#FHIR FAQs and Knowledge Base articles

HL7 has published a set of FAQs and knowledge base articles about FHIR, that cover questions about the following aspects of the FHIR specification:

  • Scope and Relationships
  • Specification
  • Architecture
  • Tooling and Support
  • Resource Design
  • Implementation Approach
  • Codes and Terminology
  • Implementation Details
  • Domain Questions
  • Security

The questions were mined from the FHIR email list, the implementer’s skype channel, the various blogs about FHIR, and the questions that get asked of the project leads by email and in person.

This is a very good source of information about FHIR, and highly recommended for FHIR users of all kinds to read. Note that access to this information resource is a member benefit – you can only get to it if you have a member login on the HL7 website.

 

 

Question: HL7 Open source libraries

Question:

I work on EHR, and I want to use HL7 to communicate with different systems in a medical environment, for that I found apis (Hapi, nHapi, javasig, Everest,) and I don’t know what is the best and what are the selection criteria

Answer:

Well, what are your selection criteria? The first question is whether you are doing HL7 v2 or v3. Almost certainly, it’s v2. What language are you using? (or want to use).

Here’s some information about the libraries:

  • HAPI: an open-source, object-oriented HL7 2.x parser for Java – it also includes a number of other utilities for using HL7 v2
  • NHAPI: NHapi is a port of the original project HAPI. NHapi allows Microsoft .NET developers to easily use an HL7 2.x object model
  • Javasig: A library of code written to process V3 messages based on the MIF definitions
  • Everest:  Designed to ease the creation, formatting, and transmission of HL7v3 structures with remote systems. Supports CDA R2, and canadian messages

My evaluations:

  • HAPI – a solid well established community, with runs on the board and reliable code. I’d be happy to use this
  • nHAPI – as far as I can tell, not well established, and given it’s for an HIE, I question the long term viability of the community. Unlike HAPI, I never hear of people actually using this
  • Javasig: this is dead – note that the only link I found was to a web archive. You’d have to be desperate to try to use it, though the maintainers might get interested if you did
  • Everest: this has a community of users in Canada, but I’ve not heard of any use outside there. I’m not sure to what degree Mohawk are supporting it (I’ll ask)

You should consider one of the paid libraries – given the amount of hours you’re going to invest in the work (1000s, I bet), a few thousand for a software library and related tools is peanuts. There’s a lot of good choices there (btw, I have one of my own, which doesn’t cost that much).

 

Questions about Questionnaire

One of the themes of the connectathon that will be held in a few weeks in Phoenix is using the questionnaire resource. That’s generated lots of attention to it, and a number of questions about how to use a questionnaire to drive a form to collect answers from a person. For the purposes of the connectathon, the answers are here:

At the upcoming May HL7 Workgroup Meeting in Phoenix, we will organize the 6th FHIR Connectathon. Track 2 will focus on Questionnaires, so both client and server developers can test their skills at creating and supporting the Questionnaire resource

SQL Injection attacks against HL7 interfaces

One of the questions that several people have asked me in the discussions triggered by the report of CDA vulnerabilities is whether there is any concern about SQL injection attacks in CDA usage, or other HL7 exchange protocols.

The answer is both yes and no.

Healthcare Applications are subject to SQL injection attacks.

An SQL injection attack is always possible anywhere that a programmer takes input from a user, and constructs an SQL statement by appending it into a string, like this:

connection.sql = “select * from a-table where key = ‘”+value_user_entered+”‘”;

There is any number of equivalents in programming languages, all variations on this theme. If the value the user entered includes the character ‘, then this will cause an error. It may also cause additional sql to run, as memorably captured in this XKCD cartoon:

 Exploits of a Mum

Exploits of a Mum

There’s several ways to prevent SQL injection attacks:

  • check (sanitise) the inputs so that SQL can’t be embedded in the string
  • use parameterised SQL statements
  • escape the SQL parameters (connection.sql = “select * from a-table where key = ‘”+sql_escape(value_user_entered)+”‘”);
  • ensure that the underlying db layer will only execute a single sql command

Doing all of these is best, and the last is the least reliable (cause it’s the easiest). Because all of these actions are not as convenient as the vulnerable approach, SQL injection attacks continue to be depressingly common. And that’s on web sites.

My experience is that healthcare interfaces tend to be written very differently to websites that are going to run in public. They are not written expecting hostile users – by and large, they don’t, either. And they are always written to a budget, based on estimates that don’t include any allowance for security concerns. That’s right – the only time I have seen security feature in interface costings is for the PCEHR.

So I assume that the default state is that healthcare interfaces are potentially subject to SQL injection attacks due to programmer efficiency (anyone who calls it ‘laziness’ is someone who’s never had to pay for programming to be done).

Healthcare Applications are not so subject to SQL injection attacks.

However, in practice, it turns out that it’s not quite such a concern as I just made it show. That’s for several reasons.

I was specifically asked about CDA documents and SQL injection. Well, CDA contents are rarely processed into databases. However the XDS headers that are used when CDA documents are exchanged are often processed into databases, and these are usually extracted from the CDA contents. So any XDS implementation is a concern. The PCEHR is an XDS implementation, though programmers never needed to wonder whether it sanitized it’s inputs. A side lesson from the PCEHR story is that name fields are not generally subject to SQL injection attacks, since it’s not too long before a name with an apostrophe trips them over if they are (and that’s true on more than just healthcare interfaces).

Really, the context where SQL Injection attacks are most likely is in the context of an HL7 v2 interface.

Some healthcare applications sanitize their inputs. But validation has it’s challenges.

However the real reason that healthcare applications are not so subject to SQL injection attacks is operational. An SQL injection attack is iterative. I think this is one of the best demonstrations of how to do one:

This an ASP.NET error and other frameworks have similar paradigms but the important thing is that the error message is disclosing information about the internal implementation, namely that there is no column called “x”. Why is this important? It’s fundamentally important because once you establish that an app is leaking SQL exceptions

The key thing here is that a general pre-requisite for enabling SQL injection attack is to leak interesting information in errors, and then the attacker can exploit that iteratively. Well, we’re safe there, because interfaces rarely return coherent errors 😉

Actually, that’s not really true. It’s easy, if you can craft inputs to the interface, and capture outputs, to iterate, even if you don’t get good information. Mostly, typical users don’t get such access; they can enter exploit strings into an application, but the error messages they might generate downstream will go into some log somewhere.

That means that the interface error logs need protecting, as does access to the interfaces (in fact, this is the most common security approach for internal institutional interfaces – to restrict the IP addresses from which they can be connected to)

Generally, if you have access to the system logs and the interfaces, you’re very likely to have access to the databases anyway. Hence, SQL injection attacks aren’t such a problem.

But really, that’s pretty cold comfort, specially given that many attacks are made by insiders with time and existing access on their side. I’m sure that it’s only a matter of time till there’s a real world exploit from SQL injection.

Security cases for EHR systems

Well, security is the flavor of the week. And one thing we can say for sure is that many application authors and many healthcare users do not care about security on the grounds that malicious behaviour is not expected behaviour. One example that sticks in my mind is one of the major teaching hospitals in Australia that constantly had a few keys on the keyboard wear out early: the doctors had discovered that the password “963.” was valid, and could be entered by running your finger down the numeric keypad, so they all used this password.

So here’s some examples of real malicious behaviour:

Retrospectively Altering Patient Notes

From Doctors in the Dock:

A Queensland-based GP retrospectively changed his notes to falsely claim that a patient refused to go to hospital – then submitted those notes as evidence in an inquest, has been found to have committed professional misconduct and been suspended from practising medicine for 12 months including six months plus another six if he breached conditions within three years – and banned him from conducting most surgical procedures except under supervision for at least a year.

This is very similar to a case here in Melbourne more than a decade ago, where a doctor missed a pathology report that indicated cancer was re-occurring in a patient in remission. By the time the patient returned for another check-up, it was too late, and she died a few weeks later. The doctor paid an IT student to go into the database and alter the report to support his attempt to place blame on the pathology service. Unfortunately for him, the student missed the audit trail, and the lab was able to show that it wasn’t their fault. (I couldn’t find any web references to it, but it was big news at the time).

Both these involve highly motivated insider attacks. As does the next one, which is my own personal story.

Altering Diagnostic Reports

Late one evening, in maybe about 2001, I had a phone call from the owner of the company I used to work, and he told me that the clinical records application for which I was lead programmer had distributed some laboratory reports with the wrong patients data on them. This, of course, would be a significant disaster and we had considerable defense in depth against this occurring. But since it had, the customer – a major teaching hospital laboratory service – had a crisis of confidence in the system, and I was to drop everything and make my way immediately to their site, where I would be shown the erroneous reports.

Indeed, I was shown a set of reports – photocopies of originals, which had been collected out of a research trial record set, where what was printed on the paper did not match what was currently in the system. For nearly 48 hours, I investigated – prowled the history records, the system logs, the databases, the various different code bases, did joint code reviews, anything. I hardly slept… and I came up with nothing: I had no idea how this could be. We had other people in the hospital doing spot reviews – nothing. Eventually I made my way to the professor running the trial from which the records came, and summarised my findings: nothing. Fail. As I was leaving his office, he asked me casually, “had I considered fraud”? Well, no…

Once I was awake again, I sat and looked afresh at the reports, and I noticed that all the wrong data on the set of reports I had were all internally shuffled. There was no data from other reports. So then I asked for information about the trial: it was a multi-center prospective interventional study of an anti-hyperlipideamic drug. And it wasn’t double-blind, because the side-effects were too obvious to ignore. Once I had the patient list, and I compared real and reported results, I had a very clear pattern: the fake result showed that patients on the drug showed a decrease in cholesterol readings, and patients that were on placebo didn’t. The real results in the lab system showed no significant change in either direction for the cohort, but significant changes up and down for individual patients). For other centers in the trial, there was no significant effect of the drug at the cohort level.

So there evidence was clear: someone who knew which patient was on drug or placebo had been altering the results to meet their own expectations of a successful treatment (btw, this medication is a successful one, in routine clinical use now, so maybe it was the cohort, the protocol or dosage, but it wasn’t effective in this study). How were the reports being altered? Well, the process was that they’d print the reports off the system, and then fax them to the central site coordinating the trial for collation and storage. Obviously, then, the altering happened between printing and faxing, and once I got to this point, I was able to show faint crease marks on the faxes, and even pixel offsets, where the various reports had literally been cut and pasted together.

I was enormously relieved for myself and my company, since that meant that we were off the hook, but I felt sorry for the other researchers involved in the trial, since the professor in charge cancelled the whole trial (inevitable, given that the protocol had been violated by an over eager but dishonest trial nurse).

What lessons are there to be learnt from all this? I know this a tiny case size (3), but:

  • Many/most attacks on a medical record system will come from insiders, who may be very highly motivated indeed, and will typically have legitimate access to the system
  • In all cases, the audit trail was critical. Products need a good, solid, reliable, and well armoured audit log. Digital signatures where the vendor holds the private key are a good idea
  • Digital signatures on content and other IT controls can easily be subverted by going with manual work arounds. Until paper goes away, this will continue to be a challenge

If any readers have other cases of actual real world attacks on medical record systems, please contribute them in the comments, or to me directly. I’m happy to publish them anonymously