Invalid x Value Sets in NE2008

Background

On February 11, 2009 Keith Boone identified an error in value set "x_ActMoodIntent" as it was represented in the vocabulary (and therefore in the vocabulary schema) published as part of Normative Edition 2008 (NE2008). This page is being built to document the problem, its analysis and intended resolution.

The simplest step, looking at the value set in any way (ballot, MIF2 file, data base, RoseTree) showed, as Keith had found, that this value set contains only one code - EVN (event) - and the intent codes that had previously been part of the value set were all missing.

The remainder of this document provides:

• report or analysis as to how this happened both so as to avoid a repetition and to identify other value sets that were similarly affected
• a list of the five value sets that were affected and how they were affected,
• a summary of the impact of this error on packages such as Normative Edition 2008, and
• the resolution of this issue in the form of new releases and patches to NE 2008

Analysis

GW Beeler reviewed the history of the vocabulary content and found the error was introduced between the release of design repository "hl7_rimrepos-2.18.2.zip" in December 2007, and "hl7_rimrepos-2.18.3.zip" in February 2008. (See releases onn gForge:

Found the error in logic.In January/February, we progressed through several stages (each with multiple steps).

The first stage created code system hierarchies in all code systems based upon the value set hierarchies that previously existed.

The second stage involved re-aligning the value sets of the the CS RIM attributes (ActClass, ActMood, etc., but not ActCode, RoleCode and so forth). The task was to add or edit value sets that would align with each of the codes in the affected code system. This included "correcting" the value set constructors (elements that include value sets in other value sets) so that they matched the hierarchy in the code system. This is the stage that introduced the error Keith Boone found.

For each of these stages, we created a source "table" (in a data base) that "drove" the process, and an program process and/or set of queries to execute it. All of these have been retained, along with textual documentation of the whole process in the "MasterVocabularyQuery" data base.

Stage 2 logic

The detailed logic summary for the second stage process, written at the time of the changes in February of 2008, says:

Executes function "VS_ProcessVsChangesForStrucuralCs" which does the following:

1. Opens record sets on tables: "zVS_VsChangesForStructCs" (the source), "VOC_value_set", "VOC_code_reference", and "VOC_value_set_constructor". (It also has a query for deleting selected rows from "VOC_value_set_property").
2. Steps through the source data one row (concept code) at a time. For EACH code, it:
   • a) Determines whether there is an existing Value Set defined for that code. If there is, it:
     • i) Steps through all Voc_value_set_constructor records for which the existing VS is the "containing" VS. For each:
       • 1) if the contained VS is non-taxonomic, reset the containing VS to the "root" VS for this code system; if the contained VS is taxonomic, the constructor entry is deleted.
     • ii) Steps through all Voc_value_set_constructor records for which the existing VS is the "contained" VS and deletes these entries. (This could also be done with a simpler Delete query.)
     • iii) Deletes all Voc_Code_reference entries for which the existing VS is the "usedToBuildVs" (including the head code)
     • iv) Deletes all Voc_Value_set_property entries for which this VS is the "usedToBuildVs"
     • v) Determines whether the existing VS name matches the "style" for VS names of structural attributes. If it does not, it computes a corrected name and should (but does not yet) document that the old name is being "retired."
     • vi) Edits the Voc_value_set record to assert the correct name, definingExpression (null), allCodes(false), isTaxonomicSet (true), and basedOnCodeSystem for this VS
     • x)If there is no existing VS, the routine creates a new VS entry in Voc_value_set using the naming style and the concept's print name to arrive at a name for the VS.
     • xi) In either case, the value set id is stored in a "collection" keyed by the head code internalId (to allow later construction of the hierarchy.
   • b) Creates (or recreates) an entry in Voc_code_reference for the head code of the VS
3. Steps through each row (concept code) of the source table a second time. It uses the parent head code internalId to look up the parent value set id in the collection created in step 2-a-xi. Then adds a row in Voc_value_set_constructor to show this relation ship IF the child VS has children of its own, but not if it is a leaf term.
4. invokes "UpdateVocabularyVersionNumber" to add a new release version for these changes.

Logic Error

The problem is in step 2a-ii. The deletions documented here were fine for the "taxonomic" value sets because the constructors were added back in in step 3. However step 2a-ii should have excluded the cases where the "containing" value set (the "usedToBuildVs") was non-taxonomic (an "x_" value set), because these might not align with the code hierarchy and were not added back in as part of step 3. The algorithm had no such check and therefore executed the deletions that caused the problem Keith Boone found.

This analysis allowed the construction of queries to identify other value sets that might also have been affected. Specifically a query on the DB from a year ago that showed which "non-taxonomic" value sets included a taxonomic value set in their construction. This query revealed the list of value sets ion the following section, and inspection of all five from the current vocabulary content verify that they were indeed affected in a similar way.

Resulting Affected Value Sets

In all, it affected five "x_" value sets:

• x_ActMoodIntentEvent - Deleted inclusion of the ActMoodIntent value set, which is based on the "INT" hierarchy in the ActMood code system

• x_ActOrderableOrBillable - Deleted inclusion of the "ActClassProcedure" and "ActClassObservation" value sets, which are based on the "PROC" and "OBS" hierarchies in the ActClass code system.

• x_EntityClassDocumentReceiving - Deleted inclusion of of the "EntityClassOrganization" value set which is based on the "ORG" hierarchy in the EntityClass code system.

• x_EntityClassPersonOrOrgReceiving - Deleted inclusion of of the "EntityClassOrganization" value set which is based on the "ORG" hierarchy in the EntityClass code system.

• x_RoleClassCredentialedEntity - Deleted Inclusion of "LicensedEntityRole" value set (renamed to "RoleClassLicensedEntity" by step 2a-v above), which is based on the "LIC" hierarchy in the RoleClass code system.

Impact on Published Package

The impact of such an error on the content of a package, such as the Normative Edition, is, fortunately, limited. Because the value set remained defined and available for hyper-links, none of the static model designs (RMIMs), their documentation, or the documentation of other vocabulary artifacts were misrepresented. The specific elements that need to be patched are:

• the Design Repository delivered in support of the package, as this is the "primary" source for RIM and Vocabulary content;
• the Vocabulary MIF files (two forms) that are used by tools such as the V3 Generator to build schemas and views for static model;
• the Vocabulary.xml file that is used by the V3 Generator as the basis for the vocabulary schema that is referenced by all static model and data type schemas;
• the voc.xsd schema that is referenced by all static model and data type schemas is created by the V3 Generator;
• four html publication files that hold the definitions for value sets based on the ActClass, ActMood, RoleClass and EntityClass code systems, and display the affected value sets when browsing the content.

The most critical of these are the Vocabulary.xml file and the voc.xsd schema which are used to build and validate version 3 documents and messages (voc.xsd) and are used to re-generate sets of schemas (vocabulary.xml).

Planned Corrective Actions

In order to address this error, two near-term steps are planned:

1. An updated release of the following is being prepared for immediate release:
   • New release Design repository, which includes a vocabulary core mif file;
   • Copies of the updated "vocabulary.xml" and "voc.xsd" will be added to the Design Repository package for this release;
   • New release of "CoreMif Instances" to include the updated MIF2 file;
   • E-mail notification of these releases, including hyper-links to the new releases.
2. A "patch" for Normative Edition 2008 that will include all affected files, packaged in the directories where the Normative Edition web and CD versions install them. (This will permit "unzipping" the archive in a way that will replace the originals from the NE2008 distributions.)

Additional patches will not be prepared, but if members organization need patches for a particular design repository release they will be invited to rtequest those by contacting the V3 Publishing Work Group.

Note: These patches will include another technical correction for EnityNameUse value sets that also affects the vocabulary schema and was reported two days ago. The problems that necessitated the two corrections are unrelated.