Conformance Request for ONC RFP

Attached is the draft report that Karen mentioned. Section 5 is the pertinent information that led to today’s call. It contains our proposed approach, and along with that, the open issues which Nathan and the CGIT Work Group will help resolve. So instead of reading the entire 50 pages of the report, you only need to read the 5 pages that is Section 5. In fact, Karen and I reviewed that information after the call today and added a couple of items to Nathan’s research work (items 6 and 7 below).

File:HL7 ONC Testing Report.pdf

Provide an estimate to complete each of the following:

Question 1

How long it would take CGIT and InM to create the requirements for the Conformance Guidance Document.

Answer from Conformance: We assume this question is about the amount of elapsed calendar time needed to create requirements for the Conformance Guidance Document. We are unable to estimate that yet as we still don't have a good idea of what the consultant will be asked to do.

Question 2

Provide an estimate of the effort (hours) for the RFP work to be completed by the person that wins the RFP

The RFP work will include both creating the Conformance Guidance document and education/training materials for testing facilitators and the general HL7 Working Group Members

Answer from Conformance: This is difficult to estimate because we are still not clear on exactly what they will be tasked to do.

Question 3

Recommendations on the Conformance Guidance Document should be separate from the V2 standard (currently it’s kind of Chapter 2B; provide recommendation of what to do with Chapter 2B)

Answer from Conformance: Yes the document should be separated. This is a trivial task that was already planned for HL7 2+. This is good to know this is happening in context of this task but itself will not take time from the consultant.

Question 4

Whether we need Conformance Tool Facilitators and if we do need them, how to recruit them

(Perhaps the obvious plan would be to ask if Conformance Tooling Vendors would like to fill those roles initially)

Answer from Conformance: Yes, we need conformance tool facilitators. If we can define better what they will be doing and how conformance will be verified in the future then we will be in better shape to know how to organize the facilitators.

Question 5

Recommendation on whether to ballot conformance statements by general membership or just Conformance Tooling Facilitators

Answer from Conformance: The term "Conformance Statements" is an ambiguous term as these can be implicit or explicit. For the RFP we would recommend a term such as "Consistent and unambiguous expression of requirements".

The group is not in favor of limiting review of certain requirements to tooling experts. All implementation guide requirements should be open to all to review.

Question 6

Certifying Tooling Vendors and their tools –provide requirements for being a certified tooling vendor (how much effort (in hours) is it going to be to write the requirements; how much effort (in hours) to actually develop and implement the program).

Provide recommendation on whether writing the requirements should be done by CGIT or should an RFP be created to address the need. Provide an estimate of duration of the work based on whom you recommend performing the work.

Need to discuss this more.

Question 7

Provide recommendation on whether developing the program should be done by CGIT or should be an RFP be created to address the need. Provide an estimate of duration of the work based on whom you recommend performing the work.

Answer from Conformance: It is not clear exactly what work would be completed. More discussion needs to be done to identify those areas that a consultant could help in. Conformance is already moving forward with improvements to Chapter 2B and creating a Conformance tutorial. We definitely need assistance to move this forward but we are still not clear on what the scope of this RFP would be and how this project might assist our current work.

Future State of HL7

• Develop a policy that requires conformance statements in all implementation guides
  • Comment from Conformance: Conformance is an overloaded term that used in different ways. Can be implicit or explicit. Don't use "conformance statement" use something else "expression of requirements" or something more specific
• Develop a guidance document that defines how to develop robust conformance statement
  • Comment from Conformance: Consistent and unambiguous expression of requirements instead of "conformance statements". Representation needs to be consistent
• Train work groups to develop the conformance statements
  • Comment from Conformance: Term "conformance statement" needs to be changed
• Recruit conformance facilitators
• Develop a plan for creating conformance statements for all or some subset of extant implementation guides for which there is not conformance tooling.
  • Comment from Conformance: Makes sense for new versions. Don't necessarily need to go back to old guides. Definitely going forward. Needs to be sufficient value in work to do it again. Not for re-affirmations.
• Develop a certification program to certify conformance tools and/or tooling vendors
  • Comment from Conformance: Can we can have criteria that specifies: context-free testing, context-based testing, scope based on the number of test cases, incorporation of results sender/receiver. Address all the conformance constucts and conformance requirements. What is their coverage. Should go through the same training as conformance facilitators
• Develop and maintain a tooling catalog
  • Comment from Conformance: Very useful
• Continue efforts to create examples (CDA) and models (CIMI) for representing data
• Identify method to collect and report implementation/usage metrics annually

Questions/Comments from Nathan

1. What is a conformance statement exactly? I thought I understood but now I think I've not understood it correctly. Isn't any requirement in a profile essentially a conformance statement? Or are they only referring to the human text requirements that exist in guides that have not been fully expressed in a computable form?

1. What does having better conformance statements get for us? I see a big push for this but don't understand what the basic challenge they are trying to solve. Are they hoping that if they are written better then vendors can create tools for them.

1. Need to ask document authors to clarify statement "tooling vendors" is too restrictive. It should be "tooling implementers" as not all conformance tooling is sold as a product.

1. Changes to guides should not be done retroactively. But rather as new guides are created then new standards are used to create them. Going back to improve old guides will only cause problems because doing this will bring up issues that need to fixed in the old guide and will require a release and clarification of the old one. Old guides are water under the bridge, but new guides can be better.

1. One of the ideas that CGIT has mentioned in the last couple of years is the idea that testing resources should be available for guides when they are being created. In this way proposed standards can be tested when the guide is being balloted. Could this approach be a better process to tackle this issue? If there is a standard way to encode conformance statements and these can be loaded into a conformance tool then the group can propose examples that both pass and fail conformance. These can be then included in the balloted standard. In this way there is confirmation that the conformance statements have been written correctly before the standard is complete.

Comments from Meeting

• Consistency across different base-standards
• Consistency within a family/product
• Implementation guides having a certain level of quality
• Testable/Computable criteria
• Check list of items that would identify? This could be a developed by work on RFP.
• Need to tell them what to do.

• Need a consistencies on granularity
• Usage, cardinality, etc, unravel to a conformance statement
• Suggest that we need a preferred approach across product families, CDA through a huge monkey wrench in understand. FHIR is more like V2.
• Need to have more of a concern of how you are constraining and extending your standard rather than seeing implementation guide as a set of constraints. Get away from Implementation Guides of conformance statements.
• Everyone in a project that creates an implementation guide needs to have everyone understand how to make an implementation guide. Conformance needs to create criteria for what someone needs to know to be a conformance facilitator. Need to have sessions with Conformance to show what they are doing in their guide.
• Changing the focus of what an implementation guide is.
• They can build up scenarios and examples but their tester should be someone else.

Excerpts from Proposed Approach document

5. Proposed Approach

HL7’s proposed approach at this point in time is to require that its implementation guides provide relevant conformance statements that are machine computable and robust enough to enable conformance tooling vendors to develop conformance tooling concurrent with the development of the implementation guide. HL7 feels very strongly that it should not develop conformance testing tools internally but should provide robust conformance statements that allow tooling experts to do so.

5.2 Develop a Guidance Document that Defines How to Develop Robust Conformance Statement

HL7 will need to undertake development of a guidance document, probably authored by an external consultant with input from HL7’s Technical Steering Committee, our Conformance & Guidance for Implementation/Testing work group and the HL7 Standards Governance Board, that defines how to develop these conformance statements and the required content/format, etc. The HL7 Technical Steering Committee recommends that this guide be written by a hired consultant who will be tasked with researching best practices, other industry literature and work with the HL7 groups noted above to ensure that the guidance reflects that knowledge and is suitable for HL7 standards.

Additionally,

• Identify which internal HL7 bodies will have input on this document and oversee its development.
• Identify which internal HL7 body (e.g. the Technical Steering Committee, the entire HL7 Community, etc) will

approve this document and how that will be achieved.

5.4 Recruit Conformance Facilitators

Requiring a conformance facilitator for each implementation guide, much like the current requirement for vocabulary, modeling and publishing facilitators for other projects, will ensure development of cadre of volunteers well versed in the development of conformance statements and that this skill becomes ingrained in all work groups. This role is currently defined as follows: “The conformance Facilitator, recommended for Implementation Guide projects, is a member of the integration team with in-depth knowledge of interoperability standards and conformance/constraint rules. This individual advises the project team on the use of HL7 standards and the constraint(s) requested to support the stakeholder requirements”. This above statement would need to change “recommendation” to a “requirement”.

5.6 Develop Certification Program to Certify Conformance Tools and/or Tooling Vendors

Much like programs that certify MU tools, HL7 needs a program whereby we certify various tools that adhere to criteria that we establish. Work needed to create this program would include development of a practical guide for conformance tooling vendors on expected format and output for tools. We could develop these to be aligned to the ONC Health IT Certification Program test methods.