Interaction Conformance and Versioning - Implementation Issues

At a high level of abstraction the question is how we can ensure that the receiver of a message is aware (in as precise a way as possible) what the semantic intent of the sender was. This includes the static model and vocabulary, the dynamic model, and constraints.

If we were to specify a checklist of what a receiver should look for in order to establish the semantic intent of the sender the checklist would look as follows:

• Message(or Batch).InteractionId: the interactionID identifies the static models (wrappers and payload(s)) and constraints placed upon them. The interactionId for normative and DSTU artifacts includes a version component which in turn refers to specific versions of the static models.
  • Note: in the "old" HL7 v3 dynamic model the interactionID also identifies the Receiver Responsibilities. In the new dynamic model interactionId doesn't have dynamic model aspects associated with it anymore.
• Message(or Batch).realmCode: this attribute signals the imposition of Realm-specific (contextual) constraints/substitutions. The value of this attribute identifies the context in question. The constraints/substitutions defined by a realm may not conflict with the constraints as specified by the InteractionId. The realmCode manages the bindings/substitutions of Valuesets and CMETs (and templates and datatype specialisations if and when substitution rules for these are agreed). Each realm has a realm owner: the owner must be an HL7 Affiliate.
  • Note that in the Dutch context this attribute won’t be used anywhere else in an interaction – if the realmCode is used at the root class of the interaction it applies to all classes in the interaction. However, in some cases, realmCode may switch within an instance. E.g. A Canadian claim message referencing a U.S. realm clinical act.
  • (Lloyd) I think there needs to be a rule that realmCode is mandatory on the root data element.
  • If a message instance contains an unknown realm, then the receiver wont be able to determine whether or not vocabulary or other parts of the instance are valid against realm constraints.
  • If an application rejects a message because because the instance asserts a realm the application supports but in a place the application didn't expect it to, that application would be considered non-conformant.
• Message(or Batch).profileId: states the constraints (or potentially the extensions) from the standard static model (Composite Message Type) as well as the dynamic model (Communication Process Model) definition that the instance complies to and implements. When multiple profiles are specified, the message instance must be valid against all of them. However, a receiver may choose to validate against only the first one recognized. For this reason, 'preferred' or more-rigorous profiles should be listed first. The constraints defined by a Profile may not conflict with the constraints as specified by the InteractionId or the Realm.
  • Note: for v3, profiles can’t be (fully) expressed in a computable fashion yet; a MIF based mechanism has been defined but not yet confirmed and there is no tooling which yet supports it
  • A conformance profile definitely can indicate support for one or more realms. The open question is whether it should allow restricting where realm can transition.
• Message(or Batch).versionCode: infrastructure versioning (CMETs, datatypes, vocabulary -particularly structural codes-, transports) is communicated using the versionCode attribute and has values such as "2006NormativeEdition". The versionCode indicates "all infrastructure stuff that was normative as of the point and time the normative edition was released". For early adopters, it could be one of the development editions or an equivalent "packaged release" by a particular realm.
  • See Interaction Versioning for details.
  • In the case of vocabulary, some domains may be marked as "fixed at the time of publication" while others may be marked as "mutable", meaning that they change over time. E.g. The set of drug codes expected to be supported are those published by the local regulatory agency at a given time, rather than those which happened to exist at the time the publication was issued. The specific mechanism by which "fixed at the time of publication" vs. "mutable" will be documented in the vocabulary specifications remains to be resolved.
  • Note: the UK MIM uses this attribute to identify their publication edition, e.g. their equivalent of “2005B”. In the UK a reference to a HL7 Edition doesn’t make sense given that artifacts are inspired by, but not constrained versions of, universal artifacts as contained in an HL7 Edition.
• InfrastructureRoot.templateID: states the constraints (or potentially the extensions) from the standard static artefact model definition that the instance complies to and implements. The constraint applies from the "point of occurrence" in the model. A sender can assert any templateID anywhere they like. It can be ignored by receivers.
  • Note: Profile ids always apply from the root of the message, templateId apply to the "point of occurence".
  • There's no reason to prohibit the declaration of any templates at all, because the declaration of non-recognized templates has no impact on the receiver. Prohibiting custom templates would be similar to prohibiting local extensions, which you're also not allowed to do.
  • If an application rejects a message because that message contains a template the application doesn't recognize, that application would be considered non-conformant.
• InfrastructureRoot.typeId: This identifies the actual message type (wrapper, CMET, etc.) used by the sender in constructing the message. The message type applies from the "point of occurrence" in the model. A sender can assert any typeID anywhere they like. It can be ignored by receivers.

(Charlie McKay) Can a conformance profile mandate that particular values of these attributes are present. For example: If you send a weight, you will always include the profiles chosen "weight" templateId. All instances for an interaction will include the "UK" realmCode. TypeId will be populated for all payloads in the profile. My answer to this would be that the profile must be able to express such constraints, but I do not think that we have a balloted or even committee voted resolution to that effect.

Open Issues

Schema versions of an interaction

Note: for the purpose of this discussion, the word "schema" is used for XML schema as well as associated schematron scripts.

Consider the following scenario:

• The 2005B publication of AORTA contains the description of an interaction ABCD_IN000001NL and the constraints placed upon it. A schema ABCD_IN000001NL.xsd is published as well, and is published jointly with the 2005B publication. The schema is not normative, vendors are free to use other schema as long as their messages also validate against the published schema.
• The schema may not enforce (all of) the constraints as defined in the 2005B publication (e.g. the schema is not able to enfore everything, the universal model used to generate the schema had not been localized, there is no constraints language). After publication of the 2005B cycle some changes are made to the schema to enforce some of the constraints. Note: the constraints as present in the documentation haven't changed - only the schema have been changed.

Question: do we need versioning of the schema, and to inform the receiver of the interaction what version of the schema was used?

Answer: No. The sender defines conformance against abstract models. If the 2005B specification specified that attribute A shall not be used, and schema version X allows it, a sender is not allowed to populate the attribute. Creating a schema version Y which disallows the use of attribute A serves as a technical aid to enforce the constraint, a change in use from schema X to Y doesn’t have any impact on systems which conform to the 2005B specification, and therefore identification of the schema version is of no relevance.

It is certainly important to have schema version information in the schema, and it may be useful to include this in the instance in a comment, maybe only in diagnostic mode -- senders using the wrong specification and/or old weaker schemas is a likely source of trouble in an implementation - so telling the receiver what you are using in a comment does at least provide reassurance / the opportunity to be told that there is something better that you could do. However this should certainly NOT be done using xsi:schemalocation as this may be dereferenced by receiving processors which then try to access the schema file and cause all sorts of problems (file not found errors / processing delays / extra network activity / ...) Charliemccay 05:33, 2 Jun 2006 (CDT)

Another use case is when the older version of the schema contained a bug, which is fixed in the newer version. This will not change the interaction (assuming the bug was purely in the schema). However, receivers might want to check the version of the schema used by sender, in some cases they might even refuse instances produced with the older, buggy schema. Of course having the schema version in a comment has the disadvantage that comments are harder to parse. The argument that schemas are non-normative and therefore need no versioning is in itself not valid; it is common practice in software development to do versioning of all artefacts, requirements, design documents, source code, test scripts etc. Versioning doesn't just apply to normative artefacts, but may apply to all artefacts. Marc de Graauw 9 Jun 2006

I don't think the issue is with schema versioning - any party that releases multiple versions of an object should consider a versionioning system. Question is a) do we have to communicate schema versions in interaction instaces? b) if there is a need to communicate schema versions in a manner other than as an XML-comment, what element/attribute should they be in, taking into account that there may be MT-schema versions as well as IN-schema versions. Rene spronk 06:58, 9 Jun 2006 (CDT)

As for a), I strongly believe it should be communicated, for the reasons Charlie and I gave. If you do not, and you run into trouble in production, there is no quick way to rule out the use of a different (weak or buggy) schema as the source of trouble. If there is no schema version attribute in the instance, the only way to rule out schema version problems is use the phone, get some implementer at the other side, have him or her dive into their sources and find out which schemas were used. Theoretically you are right in stating conformance in HL7 is against abstract models (and communicated with versionId), but this stance is also highly unpractical. There is no automated way to check conformance against the HL7 models, nor are the HL7 models expressed in such a way that a HL7 parser can take an HL7 model, and an HL7 instance, and report (non-)conformance. In the absence of such mechanisms there are two possibilities i) abstain from automated conformance checking at all, ii) delegate those parts of conformance checking which can be done by XML Schema to XML Schema (maybe Schematron too). For users of the XML ITS, ii) seems the obvious choice to me. Why have humans dig up documents and hand-check instances against written specs when there is an automated possibility for most of this tedious work? Now if we do ii) (which we all do) then the schema versions in practice become quite important, even if the final word in conformance still lies with the HL7 models, not the schema's. With a schema version attribute it is all of a sudden possible for a receiver to return: "ERROR: You used an old schema version which I cannot accept" and "ERROR: Your instance does not validate against your indicated schema version.". Without the schema version atrribute in the instance the only possibility on this level is: "ERROR: Your instance does not validate against any of the schema versions which I accept", which is significantly less expressive. Besides, as we all know it is quite common for an instance to validate with one schema parser where validation fails with another. If both sides know directly the same schema was used, it is much easier to go straight to the heart of the problem. To quote your example: "Creating a schema version Y which disallows the use of attribute A serves as a technical aid to enforce the constraint, a change in use from schema X to Y doesn’t have any impact on systems which conform to the 2005B specification, and therefore identification of the schema version is of no relevance." You are right, when both systems conform to the 2005B spec, schema version is not relevant. But the problems occur when one of the systems involved does not conform to the 2005B spec. In practice, the receiver will find the instance does not validate against receivers' schema and return an error. The receiver will not inspect the non-validating instance to report "ERROR: You used attribute A, which is not allowed." Consequently the sender gets an error message which does not help much in locating the problem. Software is full of bugs, even in production, even in medical care, so software interactions should handle buggy situations as gracefully as possible. So yes, I do think schema version should be communicated in the instance, not just diagnostic, but always. As for b), I think putting information in a XML-comment which can and should be used by an application is bad XML design, so it should be in an XML element or attribute. I'm not so sure which one, and I don't consider myself enough of an HL7 expert to say which one. I guess distributing versioned schemas in named and versioned schemasets, and communicating the schemaset version in the instance should be enough, so an instance would claim to validate against the relevant schemas in this schemaset. Marc de Graauw 8 Jul 2006

Adding the schemaversion as a processable item would require a change to the XML ITS (versioning of schema isn't an abstract model issue, it is only relevant to the XML ITS). Pre-adoption would mean adding a new namespace and adding an attribute for schemaVersion.

Alternatively, to avoid having to do this, if we were to only release a schemaset with each publication release (e.g. 2005A, 2005B): given that profileId identifies the publication release, it would also identify the schemaset contained therein. Each instance contains a profileID (it's mandatory in AORTA) and therefore the schemaversion. Rene spronk 02:03, 8 Jul 2006 (CDT)

If there are fixes/improvements in the schemas without changes to the model, so there relationship between publication and schemaset could be 1:n. I see the point with the processable item, so it seems there are two solutions:

• use profileId to indicate the publication + schemaset, so a new schemaset (within the same publication) would mean a new profileId
• take schemaversion out of the interaction into a SOAP Header. Looks nice, this SOAP Header actually describes the schema used in making it's body. Marc de Graauw 8 Jul 2006

You're missing the point - if one only ever releases schemasets in conjunction with a publication, then there is a 1:1 relationship between the profileId and the schemaset. Its means one would never release a schemaset in between publication cycles. Rene spronk 03:50, 8 Jul 2006 (CDT)

I get the point I think, and I've been re-contemplating it for a while, and I appreciate the simplicity of the approach, but I still have doubts whether it will work in practice. There's no doubt sooner or later there will be an (urgent) fix in the schemas without a corresponding fix in the publications. A 1:1 approach means a new publication. Even when the publication is affected, and a single (small but important) aspect of the models is changed, with a single schema, rolling out an entire new publication seems overkill. What I'm looking for is a versioning policy with finer granularity, especially for patches. I seems to me profileId is perfect for this, since multiple can occur, with strongest first. So if we make a new and better schema, why could't instances claim conformance to it in a profileId: 'I conform to publication 2005B with improved schemaset 2005B-2' and next claim 'I conform to publication 2005B with the default schemaset'? Marc de Graauw 9 Jul 2006, 15:00 CDT

Simple - conformance is never expressed in terms of schema. Yes, an instance can conform to schema, but that's not what conformance is all about in HL7. Conformance is ITS independent. Therefore we either need an ITS-level solution, or put it in a SOAP header, or we have to use the above approach where the schema version can be implied by the publication version. Rene spronk 10:16, 10 Jul 2006 (CDT)

Interaction versioning vs using a new Interaction

Question: when should a new interaction be created, and when should an interaction just be published using a new version number?

• If between publication 2005B and 2006B a non Semantically Backward Compatible change is made to an abstract model or constraints thereupon then per the Interaction Versioning rules a new identifier has to be assigned to the artifact.
• When dependent models of the Composite Message model (e.g. CMETs, Vocabulary, Datatypes) have been assigned a new version, the interaction must be assigned a new version as well.

Note: The UK MIM specification appears to use the version part of interaction identifiers (SSDD_INnnnnnnRRvv) to identify versions of the original artefact – irrespective of whether these changes where Semantically Backward Compatible.

Schema version information in the schema is a must in the following scenario: 2005B contains interaction X, in 2006A a semantically backward compatible change is made to X, per the interaction versioning rules the interactionId of X does not change, so the name of the schema stays the same, and now there is no way to distinguish the schemas of X 2005B and 2006A, though the schemas can be very different. So I think version information in the schema is obligatory.Marc de Graauw 8 Jul 2006

Although Interaction Versioning doesn't state it explicitely, a semantically backward compatible change of XXXXUV01 (2005B) leads to a new version of the artefact, in this case to XXXXUV02 (2006A). These two artefracts have different root elements and different schema, and there is no need for schema versioning. A NON-backwards compatible change leads to an entirely different identifier, e.g. YYYYUV01. Rene spronk 03:56, 8 Jul 2006 (CDT)

If that is the case, then every change leads to a new root element and a new schema, which in turn means there never is structural back- or forward compatibility since always both parties will need to update their software to use the new schemas and root. Is this correct? If so, I think an important chance for structural compatibility is lost. Marc de Graauw 9 Jul 2006

Any change to the models results in a new schema. A system need not depend on schema by the way, depending on how it has been build. The schema for a new version of an interaction will be able to parse (with the exception of the root node) instances that are compliant to the old schema (for the older version of the interaction) (~my assumption - we'll have to check this with the ITS experts). XPath expressions with a . rootnode will remain valid between interaction versions. As such structural compatibility is maintained. Rene spronk 23:31, 9 Jul 2006 (CDT)

We've had exactly the same issues in exchanges in the Dutch Criminal Justice Chain recently. The problem is when a change to the model affects only a subset of users - say adding an optional information item which is important to some but not relevant to others. Even the slightest change such as a new root tag means the non-users of the change will have to update their software to reflect the change. The pain is not so much in the small software change, but larger companies often have a rigourous test- and release policy which requires them to fully test the new interface. This means a very significant cost without any benefit for non-users of the change. In this case, the cost was not acceptable (nor the change enforceable) so we designed the schemas and versioning so that for a subclass of changes a minor release can be made which results in no changes to existing instances in cases where the new information item is not used - in other words, existing instances remain valid without any change. I guess the situation is not much different in medicare - no-one wants to test and release for changes they do not use. In the Dutch situation the pain might be mitigated by the central messaging point, which could support both interfaces and translate, but the general principle should apply to n:m exchanges too. If so, then this would constitute a use case for a subclass of version changes which do not affect existing instances.Marc de Graauw 9 Jul 2006, 9:55 CDT

This problem in the AORTA began with the fact that the (optional) versionCode was not updated release after release, while the specifications and the schemas kept changing. I'm all for a 1:1 tie in of a schema version with a release, but the AORTA specs lacked, by not making it clear what was what. The September 2006 specs corrected that by making versionCode and profileId mandatory with values per release. This still does not make schemas mandatory, although I do agree with Marc de Grauw that schemas (possibly ammended by schematron) are all you have for validation unless you want to do it by hand. Alexander Henket 28 Aug 2006, 22:22 CET

schemas (possibly ammended by schematron) are all you have for validation unless you want to do it by hand - very true. The nature of the XML ITS means that schema&schematron are (by definition) never sufficient to check if an instance is valid. So validation "by hand" is a must. The published schema aren't normative, because you as a vendor may want to use a different version of the schema (transformed, simplified). The instances that are produced have to validate against the published schema, but that doesn't necessarily make the published schema normative. Schema can be transformed without losing any constraint that may be contained within them. Both for theoretical reasons (the XML ITS doesnt contain all constraints that are present in the abstract model) as well as for implementation reasons (a schema and all its losless transformations are equally valid) I'm hesitant to recommend that one particular schema instance be normative. Making schema normative sounds like a simplification, but it causes problems as well. Rene spronk 05:47, 29 August 2006 (CDT)

Digital Signature requirement

Question: what is the impact on versioning if we add the requirement (at some point in time) that all messages need to have a digital signature?

Answer:

• This is likely to be introduced as part of a specific publication cycle (let's call it 200XA). If an interaction references 200XA (or a later version) in its profileId, then it also contains a digital signature. From the moment publication 200XA comes into effect all receiving applications shall reject interactions that claim conformance to publications prior to 200XA.
• The only alternative is to regard this requirement in the same manner as a change in the dependent models of the Composite Message model (e.g. CMETs, Vocabulary, Datatypes), but that would be stretching the definition of dependent models too far.