HTC Position document: MIF, Repositories & HL7 Work practices

Version 0.1 (Grahame Grieve 3 November 2006).

Version 0.2 (Grahame Grieve & Lloyd Mckenzie 8 November 2006).

Version 0.3 (Grahame Grieve, Lloyd Mckenzie & Laura Sato 21 November 2006).

Version 0.4 (additional editing by Woody Beeler, 14 Dec 2006 and 17 Jan 2007).

Introduction

HTC is working with the HL7 Tooling TC to develop tools to support the development, maintenance and implementation of the HL7 V3 standard. Much of the HTC work is focused around the Model Interchange Format (MIF) and the artifact repository.

There has been considerable confusion about these two concepts, and some lack of clarity and debate about how these two things, with their associated tools, will impact on HL7, and in particular, the work processes by which the V3 standard is developed.

This document is a position paper from the HTC directorate which discusses these concepts, and proposes a future workflow paradigm for HL7. The HTC tools will be developed to lead HL7 towards this vision for more efficient standards development.

MIF - Its intended purposes and non-purposes

The Model Interchange Format is an XML file format developed by the Tooling Committee to allow for the storage, management and exchange of V3 models. The definition and use of MIF has created some confusion, partly because there is a lack of clarity about what MIF is intended for, and what it is not intended for.

Purpose - serve as an "internal" exchange format

The MIF is designed for "internal" HL7 uses. It is defined to support processes internal to HL7's standards development and internal to the tasks of implementing those standards. It is used to store and distribute HL7 V3 model artifacts; it enables tracking of HL7 content and packaging; and it is used to produce the various forms of standards documentation. The innards of a MIF document should be opaque to all but the tool developers creating tools to support these internal processes. The tool developers should be the only ones who ever think about the MIF. The tools that these developers provide allow review and management of the content in "user-friendly" environments, and should provide export facilities to other formats for other uses of HL7 Model definitions.

Purpose - represent "HL7 V3 meta-model"

The MIF XML format is defined in a set of schemas that are the definitive source of meta-data definitions for the HL7 methodology (the “HL7 v3 meta-model”). As a functional meta-model, the MIF schemas are used to validate the content of MIF documents, and also to generate code that is used in some of the HL7 V3 tools. However due to the nature of schema, the MIF does not serve as a a human readable version of the meta-model. Creating such a human-readable version is of high importance. MnM and the Tooling Committee should look at the options for generating such a rendition.

Non-purpose -- document the methodology rules and processes

Since some of the MIF schemas represent the meta-model of new methodology that is not yet documented and approved, an impression has arisen that MIF is also being used to document the processes and rules for creating HL7 artifacts and their contents. However, the MIF, even as meta-model documentation, is only the documentation of the information that supports modeling processes; it is not intended to document the processes themselves.

Rather than removing this meta-data from the MIF, MnM should undertake to document the processes they support in an appropriate fashion. In transition, while these documents are being developed, there will still be concepts that are only documented in the MIF.

Non-purpose -- a hands-on model for standards developers

Standards Developers should never have any reason to look at the XML format of a particular MIF document – the Tooling TC and HTC should provide some form of structured editor to edit each kind of concept. Providing such tools will mean that: it is much easier to learn how to contribute; contributions will be made faster; and the quality of the contributions will be higher, since the tools will enforce most validation and quality rules as the content is generated.

However this is certainly not going to happen for all artifact kinds in the short term, and maybe even in the long term some MIF artifact kinds may have such a low volume of complex editing that it is just not worth creating a structured editor. In transition, standards developers will still need to look in the MIF and edit it occasionally, but we wish to remove the need and temptation for this as much as possible.

Non-purpose -- a hands-on model for implementers

Implementers should have no reason at all to use MIF documents. HL7 should be able to deliver industry standard methods of presenting metadata to describe the instance models, so that implementers can use these industry standard formats instead of MIF. Currently implementers, including the JavaSIG, use the MIF as their source of metadata when working with instances.

Right now there is a powerful driver for this, which is that the MIF is the only reliable complete source of metadata about the instance, and many of the industry standard formats, such as XML schema and UML models, lack the features to completely represent the HL7 content. In the short to medium term, HL7 can resolve this by making more use of meta-model extension features such as those built into MOF, and in the longer term, HL7 should work with the OMG and W3C to make some improvements in the industry meta-data formats so this can be done in a standard way. In transition, Implementers will find the MIF an attractive form, but we must work to reduce their need to rely upon it, even though the MIF will remain available for those who want to use it.

Recommended Positioning

To summarize, HL7 should position the MIF as an internal format used to support HL7 development and publication processes. Although the MIF may be the near-term distribution format for use with HL7-specific tools, HL7 should commit to developing alternative standards-based approaches to support V3 implementers. Nonetheless, we recognize that there will be a transition period of unknown length while some of these problems – which are quite intransigent – are resolved.

Artifact Repository

The Tooling TC has characterized the high level functional and non-functional requirements for the artifact repository (these are appended below in Appendix 1). However there is still some confusion about what exactly a repository is, and a variety of names have been used to describe different concepts of what it is and how it is used.

The fundamental questions relate to the scope of the repository – how much should be a in a single repository, whether there should be local and/or central repository, and what that process for committing to central repositories is.

We propose that HL7 adopts the following solution for a repository:

• The repository is stored and usable locally on the PC
• Each HL7 artifact should correspond to a single MIF document
• The repository should store each MIF document as an individual unencrypted file in a fixed directory structure [Note also items 3, 6, & 15 of Repository Requirements Table]
• Package-level artifacts may be definitional packages that contain artifact definitions, or "publishing" packages that contain header information and references to artifacts from other packages to be published as part of the "publishing" package.
• The repository should make rules about the names and locations of the files within the directory structure, and validate the content before accepting it for storage
• HL7 should create a Subversion repository to store the repository, and this is the primary vehicle by which the repositories are shared
• All the source MIFs for the HL7 V3 standard should be stored in the one repository (though separate respositories may be supported by HL7 affiliates)
• There should a standard structure for the contents of the repository. Each MIF artifact should have a defined location based on properties such as it’s realm, domain, artifact kind, etc. The structure should assist users to work with only the parts of a repository they need (a full repository could be rather large). The HTC is working on a proposed structure.
• The repository should validate the entire MIF content of the repository so that any broken links or other illegal MIF content is shown in eclipse as validation errors.

In Appendix 2 we provide some technical details for how an eclipse Eclipse implementation of this solution meets the stated requirements from Appendix 1, including the searching facilities. The Eclipse community has extensive experience on how to manage source files, and this simple solution is based directly on the concepts used in other eclipse tools, in particular the java development environment. We know that this approach provides simplicity, transparency, flexibility and robustness, while being quite scalable and easy to use.

In this model, HL7 does not have a special repository on a server anywhere, we just re-use the Subversion or the currently available CVS version control repository.

HL7 Workflow

One of the key aspects of the simple solution proposed here is that each repository as a full copy of the entire HL7 standard source, as MIF or whatever other file types are appropriate.

If the HL7 publication tools were re-engineered to work directly off the repository (which is part of the HTC plans), then every single HL7 standards developer who was granted access to the repository would be able to produce complete or partial publications on demand.

We believe that this would make a significant difference to the HL7 publication process. No longer would a single central point (usually Don Lloyd and Woody Beeler at this time) receive all the different publication databases, and have to perform last minute troubleshooting and harmonization. Instead, everyone would always know what errors existed – and would receive direct notification as soon as they created them. The role of the central function could be reduced to overseeing the process, verifying the completeness of content, and providing for changes in the overall packaging as individual domains alter their content.

HTC will be working towards demonstrating a full repository with working validation at the San Diego meeting.

Appendix 1: Functional Requirements for the Repository

The following are the list of functional requirements that the Repository should support:

1. The local repository must be able to store and retrieve HL7 artifacts.
2. The local repository and its associated tools should support non-technical users without requiring them to learn underlying technologies used to meet the various requirements for the repository (such as CVS for example)
3. The local repository and its associated tools should assist and preferably enforce the user to maintain a correctly structured repository while they work.
4. The local repository and it’s associated tools should make it easy for users to synchronize their work with master repositories, and to control the scope of the synchronization so that incomplete work is not accidentally committed to the master repository
5. The local repository and it’s associated tools will cooperate with master repositories to apply business rules to prevent corruption of the master repository and ensure correct positioning of the data within the master repository
6. The local repository infrastructure must support multiple different concurrently active HL7 Projects / repositories on a desktop, including multiple different versions
7. The underlying technologies used to provide the solution must be free for use by HL7 members in developing HL7 standards
8. The local repository must be able to run on a desktop with no network connections using locally stored content
9. The local repository must support moving projects quickly and repositories or subsets thereof between different systems using CDs, memory sticks, etc
10. The local repository must provide search mechanisms to allow users to search by HL7 specific concepts or relationships. Further searching requirements are required.
    1. Be able to perform common queries very efficiently
       1. Search by artifact id
       2. Find “current version” and “historical versions” of artifacts
       3. Find an artifact and all dependencies (derivations, imports)
    2. Be capable of performing less common queries
       1. It should be possible for ‘advanced’ users to create their own queries against the repository without writing programming code. (Writing XPath or SQL is fine).
       2. E.g. Perform “ad-hoc” query based on “contains word”, “all stuff created by the UK in the last 6 months”, “what models use this vocabulary domain or code or datatype”, “find all of the content for this domain”
11. The interface (including integrity, searching, etc.) of the local repository must be available to developers of Eclipse plug-ins and unrelated products such as .Net and COM based tooling
12. The repository access software must support the following operating systems:
    • XP, Vista
    • Mac OS X +
    • Linux
13. It should be possible to install and run the repository tools without special access authority not typically granted to most users (i.e. It should not require 'Admin'/'root' authority to install and/or run)
14. <hardware requirements & scalability> - not yet clarified
15. Changes must have transactional integrity
16. The local repository installation should be simple, preferably part of a larger install of HL7 tools

Appendix 2: Implementation Map for Requirements

Table: Maps requirements above against the capabilities offered using Eclipse with Subversion for version control.

1	The local repository must be able to store and retrieve HL7 artifacts	Stored as files
2	The local repository and its associated tools should support non-technical users without requiring them to learn underlying technologies used to meet the various requirements for the repository (such as CVS for example)	Users need only use the Eclipse UI and it’s standard team interface. (Users need to understand team development concepts such as update and commit, but not arcane version control tools or commands)
3	The local repository and its associated tools should assist and preferably enforce the user to maintain a correctly structured repository while they work	UI will enforce it; validation will pick up any errors user makes directly to the directory structures
4	The local repository and it’s associated tools should make it easy for users to synchronize their work with master repositories, and to control the scope of the synchronization so that incomplete work is not accidentally committed to the master repository	Will use Subversion for the master repository. The Eclipse client support for Subversion comes with all appropriate tools for this
5	The local repository and it’s associated tools will cooperate with master repositories to apply business rules to prevent corruption of the master repository and ensure correct positioning of the data within the master repository	Using the Eclipse validation infrastructure
6	The local repository infrastructure must support multiple different concurrently active HL7 Projects / repositories on a desktop, including multiple different versions	Different workspaces may have different repositories. Users an have as many different repositories as they require (and have space for)
7	The underlying technologies used to provide the solution must be free for use by HL7 members in developing HL7 standards	All these technologies are open source with appropriate licenses for unfettered use, or will be developed under the HTC with appropriate licenses
8	The local repository must be able to run on a desktop with no network connections using locally stored content	No network connections required except for interaction with the Subversion repository
9	The local repository must support moving projects quickly and repositories or subsets thereof between different systems using CDs, memory sticks, etc	Can either copy directories or files, or use specific Eclipse actions as they become available, or possibly other utilities could be provided
10	The local repository must provide search mechanisms to allow users to search by HL7 specific concepts or relationships. Further searching requirements are required	Will be implemented using the same techniques as the java search in the Eclipse tooling, which serves as a prototypical domain specific smart search. (Search will allow searching across multiple repositories)
11	The interface (including integrity, searching, etc.) of the local repository must be available to developers of Eclipse plug-ins and unrelated products such as .Net and COM based tooling	MIF stored as files in directories will be available. Indexes might not be available in other unrelated products.
12	The repository access software must support the following operating systems: XP, Vista Mac OS X + Linux	Eclipse supports these operating systems
13	It should be possible to install and run the repository tools without special access authority not typically granted to most users (i.e. It should not require 'Admin'/'root' authority to install and/or run)	No special permissions required to install Eclipse
14	<hardware requirements & scalability>	HL7 has not yet clarified the exact requirements, but the requirements will be less than for a standard Eclipse install with the Java Development Tools.
15	Changes must have transactional integrity	May need further research, but Subversion has "atomic commits" as one of its primary objectives, and therefore meets much of this requirement.
16	The local repository installation should be simple, preferably part of a larger install of HL7 tools	No installation required – just create a repository using the appropriate Eclipse action