Datatypes R2 Issue 103
Contents
NullFlavor considerations
Introduction
There is a series of issues around the nullFlavor definition and the way it is used. These all come together around the CD datatype.
The definition of nullFlavor:
Indicates that a value is an exceptional value, or a NULL-value. A null value means that the information does not exist, is not available or cannot be expressed in the data type's normal value set. Every data element has either a proper value or it is considered NULL. If (and only if) it is NULL, the isNull provides more detail as to in what way or why no proper value is supplied.
The definition of NI:
No information whatsoever can be inferred from this exceptional value. This is the most general exceptional value. It is also the default exceptional value
The definition of OTH:
The actual value is not an element in the value domain of a variable. (e.g., concept not provided by required code system)
The instruction regarding the use of OTH in CD:
a CD value without a value for the code property, or with a value that is not from the cited coding system is an exceptional value (NULL of flavor other)
The definition of mandatory:
if an attribute or association is described as mandatory in a model, then in all message instances that are derived from this model, the element must be present and non-null
Some problems arise from these definitions.
- The definition of nullFlavor is internally inconsistent
- There is confusion between "null" and "incomplete"
- The CD use of OTH is not consistent with the definition of OTH
- linking conformance to nullFlavor has unexpected interactions with the above issues.
Each of these will be explored below. There is some overlap between the different issues; fixing one may fix others, but it is useful to consider each independently.
Internal inconsistencies in definition
The definition for nullFlavor itself says: "A null value means that the information does not exist, is not available or cannot be expressed in the data type's normal value set.", but the definition for the root nullFlavor is "No information whatsoever can be inferred from this exceptional value."
The problem may arise from the term "this exceptional value" - does it mean that you have no information about the reason this is exceptional, or do you have no information about the value that is exceptional. The second interpretation - which would appear to be the technically correct interpretation - is problematic because there is many situations where quite a bit of information is actually provided.
The addendum to the definition muddies the waters a little more, when it says "Every data element has either a proper value or it is considered NULL." The problem is that there is gulf between "no information" and "not a proper value" - these are very different things. But perhaps the biggest problem here is that the question of what constitutes a proper value is left ambiguous - is it proper in respect of the rules laid out in the datatypes specification itself, or proper in regard to the constraints upon the attribute where the datatype is used? (In general, this is a trivial distinction, but there is a few cases where this turns out not to be trivial at all, and these cases are common and important).
More generally, there is confusion between the notion of "null" - a value where there is no information provided, and the notion of exceptional value - a value that falls outside the scope of the definition of the datatype (or it's constraints, depending on the meaning of 'proper'. The confusion may be in the specification, or in the readers, depending on perspective, but it is there, and significant.
confusion between "null" and "incomplete"
In classical computing, a null value is special value that has no other information. It is sometimes referred to as a "sentinel value", and more often by the less descriptive "special value".
HL7 defines the concept of nullFlavor, which extends the concept of null to specify why a concept is null. In fact, the technical definition is "The null flavors are a general domain extension of all normal data types" - in other words any data value may have either a nullFlavor value, or a value defined in the domain of the type of that value.
This is a departure from classical computing practices, with attendant implementation issues, but these are just a price which HL7 chose to accept.
The problem comes when data values that are labelled as null carry some information that illuminates why they are null. Now in the datatypes as defined at this time, this is of critical semantic importance - we need to be able to use AD.use, for instance, on an AD that is null, to explain that we didn't ask what the home address was. But this use of nullFlavor - where we call it null, but still provide more information - is not only not conformant to the definition given above (by the strict technical interpretation of the meaning of domain values), it violates the general semantic definition of null: a partially populated information object is not null.
And in the case of CD, there can be quite a bit of information provided on a null CD.
CD use of OTH inconsistent with the definition of OTH
The definition of OTH mentions CD specifically, and says that CD takes a nullFlavor of OTH is it doesn't provide a concept from the required code System. but CD says, in rather more detail, that a CD has the nullFlavor of OTH if the code is blank and the codeSystem is provided. In other words, if you can specify a codeSystem but not a code (or if you make your own code up), then the nullFlavor is OTH.
There is a subtle but important distinction between the two - over the use of the word "required". The CD definition does not make any reference to "required", only to the specified codeSystem. If the value is not in the specified codeSystem, then the nullFlavor is OTH. This has nothing to do with what is the required codeSystem.
This slight difference is magnified because of conformance
conformance is linked to nullFlavor
Mandatory is explicitly linked to nullFlavor: The requirement that an attribute is mandatory is met by the attribute containing a non-null value. In other words, that the attribute contains a "proper" value.
At first glance, this seems like a reasonable definition. However by this point it should be clear that there is some differences between what is a null value and what is an improper value. There is cases where a proper value has a nullFlavor, and cases where an improper value does not have a nullFlavor.
As things stand at present, "proper" is abandoned in favor of use of null, but in several regards we are not consistent in our practice. Part of the problem is that "proper" has no formal definition - and it would probably not be appropriate anyway.
CD
All these problems meet at CD. Imagine the following scenario:
The model specifies that the code is mandatory, and must come from a SNOMED value set. The originating system actually stores the diagnosis as plain text, and a medical records coder encodes the diagnosis in ICD-10, and SNOMED, in order to send the code. The text is "right arm wound on upper arm", which is encoded in ICD-10 as S41.1, and the SNOMED expression 283369005<272741003:24028007>.
R1
In this case, the XML ITS fragment for the act.code (say), based on Datatypes R1, is
< code nullFlavor="OTH" codeSystem="[oid for ?]">
<originalText>right arm wound on upper arm</originalText>
<translation code="S41.1" codeSystem="2.16.840.1.113883.6.3"
codeSystemName="ICD-10" displayName="Open wound of forearm, part unspecified"/>
<translation code="283369005" codeSystem="2.16.840.1.113883.6.96"
codeSystemName="SNOMED-CT" displayName="Laceration of upper arm (disorder)">
<qualifier>
<value code="272741003" codeSystem="2.16.840.1.113883.6.96"
codeSystemName="SNOMED-CT" displayName="Laterality (attribute)"/>
</qualifier>
<qualifier>
<value code="24028007" codeSystem="2.16.840.1.113883.6.96"
codeSystemName="SNOMED-CT" displayName="Right (qualifier value)"/>
</qualifier>
<translation>
</ code>
So, the first problem is whether the root carries a nullFlavor or not.
In this case, it was given a nullFlavor, since the model expects something
from snomed, and this is not it. So we expect to have to give it a nullFlavor
of OTH. And because we did that, we must give it a codeSystem - that's
a constraint in the abstract spec:
invariant(CD x)
where x.other {
x.code.other;
x.codeSystem.nonNull;
};
But that's not right - we don't even have a coding system. So we can't use nullFlavor OTH. We could try for a non-nullFlavor. In this case it might work - no nulFlavor and no codeSystem. But whoops, there's this constraint:
invariant(CD x)
where x.nonNull {
x.code.nonNull;
};
I guess we'll have to stick to nullFlavor OTH and make a codeSystem up. If we change the use case so that the system is using it's own internal coding system, then this works a little better - at least we're not making a codeSystem up that we didn't use.
Now we turn to conformance. The model says that the code is mandatory and must be encoded from SNOMED, but the CD says it is OTH, and provides some other oid which is not SNOMED, which by some advanced logic (poor implementors!), from the valueSet, we can tell is required.
So this instance is not conformant, and cannot be made to be conformant. So this is definitely not what we anticipated when we set out to allow translations and require a code from a particular valueSet.
R2
In datatypes R2, we have made an important change, by changing the rule that the translation nesting must reflect the sequence of translation order, to a new rule that says that the "" See [[]]. Under this rule, the XML ITS would look like this:
< code id="x1" code="283369005" codeSystem="2.16.840.1.113883.6.96"
codeSystemName="SNOMED-CT" displayName="Laceration of upper arm (disorder)">
<originalText>right arm wound on upper arm</originalText>
<translation code="S41.1" codeSystem="2.16.840.1.113883.6.3" source="x1"
codeSystemName="ICD-10" displayName="Open wound of forearm, part unspecified"/>
<qualifier>
<value code="272741003" codeSystem="2.16.840.1.113883.6.96"
codeSystemName="SNOMED-CT" displayName="Laterality (attribute)"/>
</qualifier>
<qualifier>
<value code="24028007" codeSystem="2.16.840.1.113883.6.96"
codeSystemName="SNOMED-CT" displayName="Right (qualifier value)"/>
</qualifier>
</ code>
actually, there's a little bit of getting ahead of ourselves here, in that we haven't discussed how to do the source references, and I have used ID/IDREF to illustrate the concept - but let's put that aside
Anyhow, it's obvious that the fix we propose to make for CE also makes CD work - for basically the same reason, it's related to conformance. This fix simplifies the task of assessing conformance, and finding the [?] code - but the key question is, what code goes in the root?
The sender choses, and the sender's choice mightn't align with the receivers choice. What if no valueSet is specified? what if there is more than one code given that is valid in the valueSet (for instance, if there was multiple snomed renderings). So receivers may have to prowl through translations anyawy. And because of valueSets, there is some complexity relating the codeSystem (which is the most important basis for checking which translation to use) to what you want. The point of this is that this is complicated, and the practical outcome of saying to put a code of the senders choice, partly guided by conformance, might not be as big as we'd like to the implementors.
Multiple codeSystems
So R2 works, if and only if there is only one codeSystem specified. But this not always the case. There is a variety of reasons why one might specify that the code might be sent in more than one codingSystem or valueSet, and a multitude of ways to do it.
A typical example of this is in Australia, with regard to sending lab results. Each lab system has it's own internal terminology for describing lab tests, and has been sending out reports based on this terminology for years. Some 5 years ago or so, the Australian standards body recommended that everything also be LOINC encoded, and standardised the use of CE4-6 for this in V2. But a new decision is to standardize on SNOMED and add new pathology tests to SNOMED. So Australia would like to require all three coding systems for the moment (and has discussed asking for 3 new components on v2 CE for this purposes). There's a number of other reasons for this as well.
--LLoyd - stick a Canadian use case in here. Charlie, is there a NHS use case? --
There's a number of ways to specify multiple valuesets. It could be done by using the realm machinery, though it probably doesn't really give you fine enough control to pull that off. It could be done by specifying templates or a profile with a mandatory valueset different to the underlying model. or it may be done directly in the model itself - to specify multiple required valueSets.
Going back to the use case, and adding to it that the combination of models (however done) specify that this code must contain both a SNOMED and a ICD-10 code, then how does this look in the XML?
The problem is that the concept of "root" code no longer really applies - there is multiple codes provided to meet the conformance requirements, and there can no longer be a single nullFlavor to deal with meeting conformance in this regard.
One possible response to this problem is to say that the nullFlavor relates to all the requirements - you are null unless they are all satisfied. If this is the case, what does the XML look like for this CD use case? whichever code you put in the root, what are the nullFlavors on them? There is two main possibilities, and they are both broken in regard to conformance.
Another possible response is to say that we cannot support more than one constraint on a CD (or any type, as this is a specific case of a general problem). But Given that we allow multiple encodings to be sent, and given the importance of multiple encodings, it would not be an acceptable outcome if the consequence of requiring more than one encoding was that an instance couldn't be conformant.
Another solution is to redesign CD. You could go for a model like this:
type ConceptDescriptor alias CD specializes ANY {
ED originalText;
SET(CODE) encodings;
};
type CodedConcept alias CODE specializes ANY {
ST code;
ST displayName;
UID codeSystem;
ST codeSystemName;
ST codeSystemVersion;
LIST<CR> qualifier;
SET<CS> rationale;
CD source;
}
But this gives you a different (but related) problem - what is the nullFlavor on the root? is it some combination of the nullFlavors of the encodings? So this is just moving the deckchairs on the titanic (though it does have some merit in it's own right from other perspectives).
Other Examples
Earlier I said that there are cases where "proper" != null. I've illustrated some good ones. Here's some others:
- nullFlavor QS - say, add enough to fill the beaker. If you are dealing with a formula, you specify that the value : PQ is mandatory, and require a proper value. the user says, a bit of this, a bit of that, and fill it out with Normal Saline. Is this a proper value? Should it be conformant? Interesting question, probably with a not simple answer.
- A classic case where non-null is not proper is an address with a nonnull flavor and a single del. This should fail conformance in most cases, but doesn't, since it's not technically null.
- if a coded attribute is CWE mandatory, it shouldn't be given a nullFlavor just cause the codingSystem is extended, but by the rules of CD, it must be. So it would be non-conformant by the rules of conformance even though it isn't.
Solution
Though there are some minor internal problems with the nullFlavor definitions, it's only when you introduce the notion of conformance that the real problems start. And though the problems have mostly been illustrated by CD, they don't actually only apply to CD.
I propose that the solution to this problem is to separate conformance and nullness. Nullness should be a property of the datatypes themselves - and in fact, only a few things here and there need to be changed to make this happen, such as the removal of the word "required" from the definition of OTH.
This complicates conformance somewhat, and this will be discussed below. But it does clearly define the scope of "proper" - it's a proper value if it can be described by the unconstrained domain of the datatype itself.
Implications
To discuss:
- not using OTH if code is not in codeSystem
- how to represent content in instances if it fails to conform?
- which attributes can be populated if the value is null?
- how to make conformance work properly without null (ii.root case)
Skype thread archive - Grahame / Gunther / Lloyd
[2:51:23 PM] Lloyd McKenzie says: Would the rules be clear that this behavior would be allowed on non-mandatory codes, but not on mandatory ones? [9:50:04 PM] Lloyd McKenzie says: Effectively, the code behaves as "null" because it has no real value. [9:50:21 PM] Grahame Grieve says: well, it's still OTH - you're just saying why [9:50:35 PM] Lloyd McKenzie says: Technically, it's not OTH [9:50:55 PM] Grahame Grieve says: why not? [9:51:21 PM] Lloyd McKenzie says: OTH says "the value is not expressible in the allowed constraints" [9:51:26 PM] Lloyd McKenzie says: That's not the case here [9:51:40 PM] Lloyd McKenzie says: It's expressible, it just hasn't actually been expressed . . . [9:51:56 PM] Grahame Grieve says: oh that [9:52:07 PM] Lloyd McKenzie says: :) [9:52:15 PM] Grahame Grieve says: how do you know it's expressible? [9:52:57 PM] Lloyd McKenzie says: Ok, it's "potentially expressible" [9:53:06 PM] Lloyd McKenzie says: As opposed to "definitely not expressible" for OTH [9:55:10 PM] Lloyd McKenzie says: I think we have the head of a different problem here too [9:55:30 PM] Lloyd McKenzie says: If I have a code that's CNE, mandatory, I expect to find a code from my valueset. [9:56:04 PM] Lloyd McKenzie says: If I declare 2 templates that are also effective, and the attribute is CNE, mandatory but they have different value-sets, I expect them to be there too. [9:53:44 PM] Grahame Grieve says: I don't think that the OTH definition was written with that in mind. I think it should be "not expressed" [9:54:01 PM] Grahame Grieve says: and "expressibility unknown" and "not expressible" should be specialisations [9:54:52 PM] Grahame Grieve says: specially since what we did approve as a staging solution for R1 was that [whatever the code is] is subsumed by OTH [9:56:31 PM] Lloyd McKenzie says: If I send an instance that satisfies some of the valuesets, but not others, is the attribute null or not? [9:56:57 PM] Grahame Grieve says: yes, last time we discussed this we agreed that conformance was utterly broken on CD [9:57:01 PM] Lloyd McKenzie says: I could go for the specializations [9:57:07 PM] Lloyd McKenzie says: I'm not sure it's just CD [9:57:46 PM] Lloyd McKenzie says: Imagine an instance with three different template constraints on GTS. [9:58:00 PM] Lloyd McKenzie says: Is a violation of a constraint considered to make something null? [9:59:03 PM] Lloyd McKenzie says: Only 380 e-mails. I got off easy :) [10:05:30 PM] Lloyd McKenzie says: 335 when I toast spam . . . [10:11:07 PM] Grahame Grieve says: Is a violation of a constraint considered to make something null? - no [10:11:30 PM] Lloyd McKenzie says: Except possibly for OTH [10:11:42 PM] Lloyd McKenzie says: OTH says "I can't express this within the constraints provided" [10:11:57 PM] Lloyd McKenzie says: For CD, that's usually a valueset + CNE [10:12:19 PM] Lloyd McKenzie says: For GTS, it might be a constraint that says "this needs to be a continuous interval", but perhaps in some situation it isn't [10:12:31 PM] Lloyd McKenzie says: For II, maybe I've got the extension, but no root [10:27:45 PM] Grahame Grieve says: OTH says "I can't express this within the constraints provided" - that's not the definition [10:29:03 PM] Lloyd McKenzie says: It says it in other words [10:29:11 PM] Grahame Grieve says: what words? [10:29:26 PM] Lloyd McKenzie says: "Not in the value domain of a variable" [10:29:44 PM] Lloyd McKenzie says: The value domain is effectively the allowed set of values, given the constraints applied to the variable [10:30:38 PM] Grahame Grieve says: ok. so now we have a situation where we can express full information in the instance, but we still claim no information, because some model makes some rule somewhere? [10:30:53 PM] Grahame Grieve says: even though we do express full information [10:31:28 PM] Lloyd McKenzie says: The challenge is that "full information" is dependent on compliance with the value domain. [10:31:39 PM] Grahame Grieve says: where is this defined? [10:31:42 PM] Lloyd McKenzie says: If I send you a valid code that's outside the domain, you can't process with it. [10:34:24 PM] Grahame Grieve says: what's that got to do with the definition or meaning of No Information [10:34:43 PM] Grahame Grieve says: I think we really need a different flag - one for nullness and one for compliance [10:35:09 PM] Lloyd McKenzie says: They're intrinsically linked [10:35:34 PM] Grahame Grieve says: linked? or related? [10:35:37 PM] Lloyd McKenzie says: Linked. [10:35:54 PM] Grahame Grieve says: I have a concept. What I have is the coding system - mine - in which the original concept was encoded [10:35:54 PM] Lloyd McKenzie says: You can't execute many operations on an attribute that is null [10:36:07 PM] Lloyd McKenzie says: You can't test if it's a specialization. [10:36:29 PM] Grahame Grieve says: I translate it to a different coding system, but it doesn't have a concept for my concept [10:36:56 PM] Grahame Grieve says: so I end up with two CD's, one which is fully encoded, and one which is nullFlavor OTH with a codingSystem and originalText [10:37:08 PM] Grahame Grieve says: agree? [10:37:15 PM] Lloyd McKenzie says: (having a codingSystem is dumb, but yes) [10:37:40 PM] Lloyd McKenzie says: And the one that's OTH is null, even though you have information [10:37:51 PM] Grahame Grieve says: ok. So then when I send it, whether it is null or not depends on what the constraints from the model are [10:38:28 PM] Lloyd McKenzie says: Yup. [10:38:43 PM] Lloyd McKenzie says: If it were CWE, it wouldn't need to be OTH [10:38:45 PM] Grahame Grieve says: so sometimes I am claiming that "No information whatsoever can be inferred from this value", and sometimes I am not. Even though the value itself is unchanged. [10:38:56 PM] Lloyd McKenzie says: You're misreading NI [10:39:00 PM] Grahame Grieve says: it would still need to be OTH, whether it was CWE or not. [10:39:09 PM] Lloyd McKenzie says: NI says "you can't tell anything about the reason why this is null" [10:39:17 PM] Grahame Grieve says: I just quoted the definition [10:39:25 PM] Lloyd McKenzie says: No information whatsoever can be inferred from this exceptional value. This is the most general exceptional value. It is also the default exceptional value. [10:39:39 PM] Lloyd McKenzie says: You can't infer information *from the null value* [10:39:48 PM] Lloyd McKenzie says: It doesn't say anything about the other properties [10:40:39 PM] Grahame Grieve says: I thnk you are wrong [10:40:40 PM] Grahame Grieve says: and right [10:40:46 PM] Grahame Grieve says: you are wrong because the exceptional value is the whole value [10:40:52 PM] Grahame Grieve says: for instance: a CD value without a value for the code property, or with a value that is not from the cited coding system is an exceptional value [10:40:58 PM] Grahame Grieve says: proves the other thing wrong. [10:41:07 PM] Grahame Grieve says: so reading the spec, you are wrong. BUt you are making sense [10:41:41 PM] Lloyd McKenzie says: Always good when I make sense ;) [10:42:54 PM] Lloyd McKenzie says: Fundamentally, a null value is not processable the way a non-null value is processable [10:43:11 PM] Lloyd McKenzie says: If you make something mandatory, you're saying "this needs to be processable" [10:43:26 PM] Lloyd McKenzie says: (where processability is based on the type of data you're dealing with) [10:43:26 PM] Grahame Grieve says: I think you are building an edifice on a deck of cards [10:43:37 PM] Grahame Grieve says: a null value is not processable the way a non-null value is processable - where is this stated? [10:43:53 PM] Grahame Grieve says: the datatype spec rather blurs this concept in places [10:43:58 PM] Lloyd McKenzie says: I'm saying what I think makes sense and is useful. [10:44:27 PM] Grahame Grieve says: oh? really? me, I'm just working with V3 - I lost the link to sensible and useful long ago ;) [10:44:33 PM] Lloyd McKenzie says: If I get an II, and it's mandatory, then I expect the value I receive to have certain characteristics and to be able to do things with it. [10:44:43 PM] Lloyd McKenzie says: :P [10:45:10 PM] Lloyd McKenzie says: I'm not necessarily pushing sensible, but at least I want useful. (And sensible if possible ;)) [10:45:35 PM] Grahame Grieve says: There's some real problems in that regard [10:45:42 PM] Lloyd McKenzie says: Certainly the use of OTH fits what I'm describing. [10:45:56 PM] Grahame Grieve says: I cannot think of any other specification or context where null doesn't mean completely null [10:46:23 PM] Grahame Grieve says: the idea that you can have something that is null, and then actually have it present data - this has many troublesome connotations. [10:46:30 PM] Lloyd McKenzie says: I'm happy to say that there are two different axis here. [10:46:46 PM] Lloyd McKenzie says: Basically you can have: [10:46:51 PM] Lloyd McKenzie says: Complete information [10:46:53 PM] Lloyd McKenzie says: Sufficient information [10:46:56 PM] Lloyd McKenzie says: Insufficient information [10:46:59 PM] Lloyd McKenzie says: No information [10:47:13 PM] Lloyd McKenzie says: If we make something mandatory, it means we need at least "sufficient information" [10:47:43 PM] Lloyd McKenzie says: Most of the null flavors talk about reasons for having "no information" [10:48:14 PM] Lloyd McKenzie says: OTH talks about why we have insufficient information [10:49:19 PM] Grahame Grieve says: actually, but your wider definition, it could be "inappropriate information" rather than insufficient [10:49:34 PM] Grahame Grieve says: but maybe this is "insufficient allowable information" [10:49:50 PM] Lloyd McKenzie says: Fair enough [10:50:01 PM] Lloyd McKenzie says: Shall I add Gunther? [10:50:03 PM] Grahame Grieve says: there's a fine line - what information do I leave out because of constraints before what is left becomes insufficient? [10:50:09 PM] Grahame Grieve says: I guess. [10:50:12 PM] Grahame Grieve says: he [10:50:15 PM] Grahame Grieve says: he's got nothing to do anyway [10:50:23 PM] Lloyd McKenzie added Gunther Schadow to this chat [10:50:36 PM] Grahame Grieve says: I'll copy and paste where we have been [2:51:41 PM] Grahame Grieve says: we are arguing about the definition of null and it's relationship to conformance [2:52:02 PM] Grahame Grieve says: huh. well, read that. We've all got nothing to do [3:05:39 PM] Grahame Grieve says: I guess I don't care what I have to leave off to meet the constraints, as long as I can meet the constraints. [3:06:15 PM] Grahame Grieve says: But what about this CWE/OTH issue - the datatypes spec says CD must be OTH if it doesn't have a code, but you don't think it should be null in this case [3:06:58 PM] Lloyd McKenzie says: If it has to have a code whether it's CWE or CNE, then what's the point of coding strength? [3:08:42 PM] Gunther Schadow says: Can u summarize the issue in one sentence? Is it about OTH or NULL in general? [3:09:17 PM] Grahame Grieve says: yes, that's a good question [3:09:21 PM] Lloyd McKenzie says: Null tackles two different issues: [3:09:35 PM] Lloyd McKenzie says: "No information" and "Insufficient appropriate information" [3:09:53 PM] Lloyd McKenzie says: MSK, NASK deal with the former. OTH deals with the latter [3:09:54 PM] Grahame Grieve says: and OTH sits on the fence between the two [3:10:38 PM] Gunther Schadow says: Why is that fine distinction important? [3:11:09 PM] Lloyd McKenzie says: "Insufficient appropriate information" causes grief because "what's appropriate" changes based on the spec. If I've got an attribute asserting 2 templates, I might have 3 valuesets, but only comply with one. [3:11:42 PM] Lloyd McKenzie says: Thus I have "insufficient appropriate information" according to 2 specs, but am fully valid against the 3rd. [3:13:11 PM] Grahame Grieve says: this also runs into translations - the root is null, but the concept is correctly provided within - that's another problem - cause the concept isn't actually null. we sort of fixed that by adopting your (G) recommended solution to the CE problem, but Lloyd keeps positing multiple constraints, and we just made a mess of that [3:14:01 PM] Gunther Schadow says: But how is that different from "temporarily unavailable" or "masked" (your thing Lloyd?). Even "unknown" and hence "no information"? It is all bound to change as the world evolves. [3:15:33 PM] Lloyd McKenzie says: The difference is that for a single INSTANCE, if it's UNK, it's UNK against all specs. [3:15:47 PM] Lloyd McKenzie says: For OTH, it might be null against some specs and not null against others [3:16:17 PM] Lloyd McKenzie says: And we don't have the ability to say "are you null against spec A?", we can only ask "are you null?" [3:17:33 PM] Gunther Schadow says: Who cares about the spec? MSK is different depending on purpose. What does "single instance" mean? Why is that important? You are always null because of something. Who is asking for being able to know why? [3:17:55 PM] Lloyd McKenzie says: Single instance matters because it's the basis of conformance [3:17:57 PM] Gunther Schadow says: If you spec does not include the last menstural period, then that is unknown to me unless you send something else. [3:18:43 PM] Lloyd McKenzie says: If null is based on the spec, we need to expose that. [3:18:54 PM] Lloyd McKenzie says: If it's not based on a spec, then we can't use OTH as a null flavor [3:19:06 PM] Gunther Schadow says: I hear Grahame doing his spiel about v3 not being sensible, but how is this distinction going to make it more reasonable? [3:19:24 PM] Lloyd McKenzie says: Right now, it's broken [3:19:24 PM] Gunther Schadow says: If null is based on the weather we don't need to expose that either? [3:19:29 PM] Gunther Schadow says: What is broken? [3:20:01 PM] Lloyd McKenzie says: I can have a single instance that is null according to some specs and not null according to others, but the test to say "is this null" doesn't allow me to test the spec [3:20:24 PM] Grahame Grieve says: I think the real problem here is that nullness should only be linked to the data itself, not to anything about conformance to some model [3:20:36 PM] Grahame Grieve says: CD OTH links the 2. [3:20:55 PM] Grahame Grieve says: if we broke those two things apart, then we would get a clearer picture [3:20:55 PM] Lloyd McKenzie says: Except that there's a need for that link [3:20:59 PM] Grahame Grieve says: why? [3:21:23 PM] Lloyd McKenzie says: Because you want to be able to say "This attribute is mandatory, CNE" and be able to count on it being a valid code from the value-set [3:21:36 PM] Grahame Grieve says: what has that got to do with nullflavor? [3:22:10 PM] Lloyd McKenzie says: Mandatory says "can't be null". If OTH is a null flavor, then "incomplete" information is still considered null [3:22:11 PM] Grahame Grieve says: already I must ignore nullFlavor to evaluate that in my validator [3:22:23 PM] Grahame Grieve says: so we need to change the definitions in conformance, and make something that works [3:22:38 PM] Grahame Grieve says: Mandatory should mean, must be properly populated [3:22:51 PM] Lloyd McKenzie says: If OTH is not a null flavor and null only means no data at all, then I can't use mandatory to ensure I have complete data [3:22:50 PM] Gunther Schadow says: Again, what is "instance" here? You have two meanings of it and that confuses you. There may be an object and you create different views of it for different purposes. One purpose only wants the age if less than 18 years old. So you conform to it. The object is the same, the view may be different. But the view is always different, because sometimes it allows you to send some extra info other times it doesn't. What you send depends on the spec. Yes. The spec might require you to mask out patient name (MSK). Fine, I don't see a problem. [3:23:03 PM] Lloyd McKenzie says: Instance means "what goes over the wire" [3:23:15 PM] Lloyd McKenzie says: Object is irrelevant to 95% of the HL7 world [3:23:35 PM] Lloyd McKenzie says: Objects don't conform, only XML instances (or other wire formats) conform [3:23:42 PM] Gunther Schadow says: but it's not in reality. If HL7 doesn't care about reality, I can't help it. [3:23:44 PM] Lloyd McKenzie says: HL7 is a data exchange standard [3:23:55 PM] Lloyd McKenzie says: All we care about is what's exchanged when we talk conformance [3:23:57 PM] Grahame Grieve says: Gunther starting to sound like me a little [3:24:05 PM] Gunther Schadow says: And you send what the spec allows. You can know more. What you send is not all you have. [3:24:36 PM] Lloyd McKenzie says: So question 1 is: [3:24:51 PM] Lloyd McKenzie says: Is mandatory == nullFlavor exists? [3:24:58 PM] Gunther Schadow says: ? [3:25:13 PM] Gunther Schadow says: mandatory means not null, doesn't it? [3:25:19 PM] Lloyd McKenzie says: Sorry, yes [3:25:27 PM] Grahame Grieve says: that's the standing definition, but it's broken [3:25:34 PM] Lloyd McKenzie says: So mandatory means null flavor doesn't exist [3:25:47 PM] Grahame Grieve says: not for CD, at least, not in practice. [3:25:51 PM] Lloyd McKenzie says: Ok, so you're saying that mandatory means more than just null flavor doesn't exist? [3:26:06 PM] Grahame Grieve says: it must. try validating CD. [3:26:12 PM] Lloyd McKenzie says: CD is the prime example, but there are others (e.g. an II where I've got an extension but no clue what the root is) [3:26:17 PM] Grahame Grieve says: you have to look in translations and ignore the root nullFlavor [3:26:27 PM] Lloyd McKenzie says: Null flavor applies to the whole type [3:26:33 PM] Grahame Grieve says: I reject you II case. it's going to be non-conformant some how [3:26:35 PM] Lloyd McKenzie says: Translations are irrelevant. [3:26:48 PM] Grahame Grieve says: maybe NullFlavor applies to the whole type, but that means that CD is done wrong [3:27:16 PM] Lloyd McKenzie says: CD is definitely done wrong in terms of differentiating root from translations. However, that's a side issue [3:27:35 PM] Grahame Grieve says: if the root nullFlavor is the nullFlavor for the whole concept, not just the root concept, then it's designed wrong [3:27:55 PM] Grahame Grieve says: because they are conflated in the current design. [3:28:17 PM] Lloyd McKenzie says: At the moment, OTH seems to allow for the null flavor being present to work for "mandatory", until you dig beneath the covers and think about multiple specs. [3:29:03 PM] Grahame Grieve says: well, that too. But it doesn't even work with a single spec - unless you account for our R2 CE fix. [3:29:06 PM] Grahame Grieve says: which you reject anyway [3:29:28 PM] Grahame Grieve says: anyhow, as soon as you have multiple specs, linking mandatory to nullFlavor breaks down [3:29:46 PM] Grahame Grieve says: the assessment has to be more sophisticated. [3:29:48 PM] Lloyd McKenzie says: Agree. [3:30:59 PM] Grahame Grieve says: we could make some fixes here. [3:31:01 PM] Grahame Grieve says: The actual value is not an element in the value domain of a variable. (e.g., concept not provided by required code system). [3:31:02 PM] Grahame Grieve says: make it [3:31:06 PM] Grahame Grieve says: The actual value is not an element in the value domain of a variable. (e.g., concept not provided by specified code system). [3:31:15 PM] Grahame Grieve says: bingo - broken the link. [3:31:45 PM] Grahame Grieve says: leaves mandatory hanging in the wind, but we have to do something about that anyway [3:31:51 PM] Gunther Schadow says: Why is mandatory for CD not mean not null? [3:31:56 PM] Lloyd McKenzie says: Um, I don't see a difference in the sentences . . . [3:32:09 PM] Grahame Grieve says: no. right. [3:32:25 PM] Grahame Grieve says: The actual value is not an element in the value domain of a variable. (e.g., concept not provided by the codeSystem property). [3:32:31 PM] Grahame Grieve says: that's what I meant to say [3:32:52 PM] Gunther Schadow says: Why does CD make an exception for you? It doesn't. [3:33:03 PM] Grahame Grieve says: Gunther, mandatory for CD does not mean not null because the translation that meets the conformance requirement may not be the root [3:33:10 PM] Grahame Grieve says: or the one that fails to meet it [3:33:12 PM] Gunther Schadow says: If you make some CD attribute mandatory, it can't be null. It can't be OTH. [3:33:37 PM] Grahame Grieve says: but if the originalCode is not coded, and then a translation is the correct code... [3:33:41 PM] Lloyd McKenzie says: Right, but what happens if an attribute is valid against some specs, but not others - do you assert OTH or not? [3:33:47 PM] Gunther Schadow says: Then you can't send the message. [3:33:58 PM] Grahame Grieve says: Lloyd, let's stick to a single spec for the moment [3:33:59 PM] Lloyd McKenzie says: Pretend it's not mandatory [3:34:04 PM] Grahame Grieve says: you can't send the message? [3:34:04 PM] Gunther Schadow says: Mandatory means: if you don't send me this bit of info, I just can't process anything. [3:34:14 PM] Lloyd McKenzie says: If we've got a single spec, there's no problem. [3:34:19 PM] Grahame Grieve says: yes, but you do send it, only in one of the translations [3:34:35 PM] Lloyd McKenzie says: Null flavor at the root applies to the root and all translations against the spec being asserted against [3:34:37 PM] Gunther Schadow says: Then it's not null. [3:34:52 PM] Grahame Grieve says: but it is null, because of this rule in the CD definition: [3:35:02 PM] Lloyd McKenzie says: That rule is crap [3:35:10 PM] Lloyd McKenzie says: It didn't take into account translations [3:35:12 PM] Gunther Schadow says: It is not null. [3:35:14 PM] Grahame Grieve says: Conversely, a CD value without a value for the code propertyis an exceptional value (NULL of flavor other) [3:35:21 PM] Lloyd McKenzie says: It's similar to the rule that you have to identify a code system when you say OTH [3:35:32 PM] Lloyd McKenzie says: Which doesn't take into account multi-code system valuesets [3:35:36 PM] Gunther Schadow says: But aren't we going to redo CD so you put the expected thing on the outside anyway? [3:35:55 PM] Gunther Schadow says: Either way, it is not null. Tranlations need to be taken into account here. [3:35:57 PM] Grahame Grieve says: I have a root CD which has no code in some arbitrary codeSystem. I have a translation which has a code in the right codeSystem [3:36:11 PM] Lloyd McKenzie says: Which is why we need to talk about multiple specs (i.e. templates) where you may have different expected codes [3:36:19 PM] Lloyd McKenzie says: (Which means you can't posisbly always put it on the outside) [3:36:23 PM] Gunther Schadow says: Then the CD as a whole is not null. [3:36:29 PM] Grahame Grieve says: if we redo CD like that, it works for a single spec. Then Lloyd invokdes multiple specs and we are back to where we started [3:36:54 PM] Grahame Grieve says: either we redo CD so we can express a nullFlavor for the concept at the root, not just the root attempt to encode the concept [3:36:59 PM] Lloyd McKenzie says: If it's valid against one spec and not against another (it's OTH), is it null as a whole or not? [3:37:01 PM] Grahame Grieve says: or we come up with some other solution [3:37:15 PM] Grahame Grieve says: but Lloyd's multiple spec conformance case is worse than that [3:37:18 PM] Gunther Schadow says: Not really. The rule would be: put the expected codeSystem/code on the outside depending on what the spec says. [3:37:28 PM] Grahame Grieve says: but there's more than one.... [3:37:42 PM] Lloyd McKenzie says: Exactly - you've got 3 specs applying simultaneously [3:37:54 PM] Gunther Schadow says: More than one, of course, but different specs, different messages. [3:38:06 PM] Lloyd McKenzie says: 1 message - multiple templates [3:38:11 PM] Gunther Schadow says: then they'd be inconsistent. [3:38:15 PM] Grahame Grieve says: this is why I want to break null and conformance apart. Define null semantics clearly based on the data, and tell conformance to go and sort itself out based on the properly expressed data and the models [3:38:34 PM] Lloyd McKenzie says: Templates are allowed to be inconsistent. I can have 3 different valuesets - that's what translations gives me [3:39:01 PM] Lloyd McKenzie says: I think I agree with the split, and mandatory needs to tie to both concepts [3:39:39 PM] Grahame Grieve says: Lloyd - what do you think you agree with? [3:39:50 PM] Gunther Schadow says: The rule to put preferred code on the outside can only work with everyone agreeing. You can't have one spec saying you neen LOINC here and the other saying you need SNOMED. They can't apply simultaneously if they're inconsistent. [3:40:15 PM] Lloyd McKenzie says: Which means "preferred code on the outside" is a really dumb idea [3:40:27 PM] Gunther Schadow says: no its not. [3:40:42 PM] Lloyd McKenzie says: It is. Because it makes you say one spec is more important than any other. [3:40:45 PM] Gunther Schadow says: It's a really dumb idea to use two templates simultaneously that disagree. [3:40:47 PM] Grahame Grieve says: but it seems to me at a semantic level we should be able to make that work - we can send multiple encoding - so we should be able to ask for more than one [3:41:11 PM] Lloyd McKenzie says: Requiring that SNOMED and ICD10 codes both be present is perfectly reasonable. [3:41:18 PM] Gunther Schadow says: You have to choose as an implementer. I use spec A or spec B, but not both. Only both if they are consistent. [3:41:22 PM] Lloyd McKenzie says: No reason to care which is the root and which is not [3:41:35 PM] Lloyd McKenzie says: As an implementer I can easily send both codes [3:41:47 PM] Gunther Schadow says: It is reasonable to require both. But noone would in practice do that. [3:41:55 PM] Lloyd McKenzie says: You bet they would [3:42:03 PM] Gunther Schadow says: Who would? [3:42:14 PM] Gunther Schadow says: Who is actually now asking for it? [3:42:25 PM] Lloyd McKenzie says: We'll have some jurisdictions in Canada that require SNOMED, others that require ICD10 and applications that want to create single instances that comply with both. [3:42:38 PM] Lloyd McKenzie says: I.e. I don't care which I'm sending to, I produce the same instance [3:42:57 PM] Lloyd McKenzie says: Forcing me to swap the codes I can send around just based on recipient is dumb. [3:43:19 PM] Grahame Grieve says: we're doing this in Australia with Lab results- we currently require every lab to send it's own internal code system, LOINC [3:43:21 PM] Gunther Schadow says: In that case, none of your folks can benefit from the preferred-code-outside trick. Tough luck. But you're not hurt by it. [3:43:28 PM] Gunther Schadow says: You just have to process all translations. [3:43:49 PM] Grahame Grieve says: and we'd like to ask for snomed too, if we weren't stuck with V2. though we might go for more components in CE yet [3:43:51 PM] Lloyd McKenzie says: I don't think anyone should expect their code in the root [3:44:02 PM] Lloyd McKenzie says: I think it's dangerous and it hurts interoperability [3:44:17 PM] Gunther Schadow says: Still, if the spec says LOINC is mandatory (and null not allowed) then you better have a LOINC code there. If it says local code mandatory you better have that. If it says nothing mandatory then there is no problem. [3:44:28 PM] Grahame Grieve says: so, are we in agreement that it's reasonable to asked to send moer than encoding at once? [3:44:36 PM] Grahame Grieve says: so, are we in agreement that it's reasonable to asked to send more than one encoding at once? [3:45:03 PM] Gunther Schadow says: It's reasonable to suggest more codes, but only one will work for actual interoperability. [3:45:35 PM] Grahame Grieve says: why will only one work? [3:45:36 PM] Lloyd McKenzie says: Yes, but if the spec says "send LOINC" and the template says "send SNOMED" and you have a root code of SNOMED and no translation (because the concept doesn't exist in LOINC), do you send a null flavor of OTH for the root or not? [3:46:00 PM] Grahame Grieve says: which do you send in the root? let's figure that out first [3:47:24 PM] Lloyd McKenzie says: I don't think it should matter. I think either should be valid in the root. Or even neither - just send originalText in the root and have two translations. [3:47:46 PM] Grahame Grieve says: semantically I agree. So the problem is with conformance, not interoperability [3:48:35 PM] Lloyd McKenzie says: If we agree that they can go either place, then the interoperability issue goes away [3:49:25 PM] Grahame Grieve says: it seems reasonable to me - instead of picking the root, you scan the root and translations looking for the right codeSystem. [3:49:44 PM] Lloyd McKenzie says: Right. [3:49:55 PM] Lloyd McKenzie says: Or more particularly, for a code from the desired valueset [3:50:06 PM] Grahame Grieve says: but how do you know that? [3:50:13 PM] Grahame Grieve says: all you have is codeSystem. [3:50:47 PM] Lloyd McKenzie says: You have code + codeSystem. You can ask CTS whether that combo fits within a given value-set [3:51:18 PM] Grahame Grieve says: oh. All of a sudden this is not so fun and easy. The performance hit for that operation is nasty [3:51:19 PM] Gunther Schadow says: "Yes, but if the spec says "send LOINC" and the template says "send SNOMED" and you have a root code of SNOMED and no translation (because the concept doesn't exist in LOINC), do you send a null flavor of OTH for the root or not?" only one can be mandatory. [3:51:36 PM] Grahame Grieve says: why? is that for interoperability or conformance? [3:51:40 PM] Gunther Schadow says: It's mandatory with respect to either SNOMED or LOINC. Likely neither of them in practice. [3:51:58 PM] Gunther Schadow says: If conformance is not for interoperability then I don't care. [3:52:07 PM] Lloyd McKenzie says: Pretend it's not mandatory for a second [3:52:16 PM] Lloyd McKenzie says: (in either the base spec or the template) [3:52:28 PM] Gunther Schadow says: Mandatory with CD doesn't mean squad if you don't say what code system. I can always send you a made up code. [3:52:36 PM] Gunther Schadow says: O.K. not mandatory. [3:52:54 PM] Gunther Schadow says: If it's not mandatory your OTH problem goes away. [3:52:56 PM] Lloyd McKenzie says: It's OTH against the template and not OTH against the base spec [3:53:03 PM] Lloyd McKenzie says: Do I send the null flavor or not? [3:53:09 PM] Grahame Grieve says: following this argument, btw, a single model should be able to say mandatory in two valueSets [3:53:45 PM] Grahame Grieve says: what's it? I've gotten lost [3:54:15 PM] Grahame Grieve says: Gunther, did you have any comment about the practicalities of plowing through the translations looking for the actual code you want to use? [3:55:22 PM] Lloyd McKenzie says: I have an instance that declares support for a template. Base spec requires LOINC, template requires SNOMED. I send just a SNOMED code, because no LOINC translation exists. As far as the base spec is concerned, the instance is OTH, as far as the template is concerned, the instance is not null. So do I send OTH as a null flavor or not? [3:56:05 PM] Grahame Grieve says: Gunther is claiming that you cannot meet those constraints. [3:56:27 PM] Grahame Grieve says: I think that this is very disappointing, to accept such a limitation because we cannot define conformance away from the properties of the data [3:57:01 PM] Lloyd McKenzie says: I think it's really bad to say we can't - we could even do it in v2 with the CE, CWE and CNE datatypes [3:57:14 PM] Lloyd McKenzie says: That's the whole point of having translations [4:00:23 PM] Grahame Grieve says: back to the definition of OTH - it's wrong to link it with required. The definition of CD itself is quite specific - CD acquires a nullFlavor of OTH if there is no code but a codeSystem is provided. [4:00:28 PM] Grahame Grieve says: nothing to do with required there. [4:01:24 PM] Lloyd McKenzie says: Not sure what you mean by required. If you mean "Requred" in the conformance sense, then yes, that's orthogonal. Where I used the term 'required', read "draws from a valueset based on" [4:01:41 PM] Lloyd McKenzie says: The definition sucks [4:01:57 PM] Lloyd McKenzie says: codeSystem is useless with a multi-codesystem valueset [4:02:22 PM] Lloyd McKenzie says: And not having a root code shouldn't be a problem if you've got the desired code in a translation [4:02:44 PM] Lloyd McKenzie says: I'm not terribly concerned with how things are - I'm interested in how they should be and what it'll take to get them there :) [4:02:55 PM] Grahame Grieve says: yeah, well we agreed to add valueSet - but you can only use it with OTH [4:03:08 PM] Grahame Grieve says: maybe we should split CD and things up and do it differently [4:03:23 PM] Grahame Grieve says: hang on [4:03:48 PM] Gunther Schadow says: It's not OTH against anything. You send LOINC or SNOMED. If you can't send either one it's OTH for that one. [4:04:42 PM] Gunther Schadow says: "Gunther, did you have any comment about the practicalities of plowing through the translations looking for the actual code you want to use?" - few would do it in practice. You get shot for that by the practitioners. [4:04:46 PM] Lloyd McKenzie says: Against the base spec, I haven't sent a LOINC code - because I can't. That sounds like OTH to me . . . [4:04:58 PM] Grahame Grieve says: type ConceptDescriptor alias CD specializes ANY {
ED originalText; SET(CODE) encodings;
};
type CodedConcept alias CODE specializes ANY {
ST code;
ST displayName;
UID codeSystem;
ST codeSystemName;
ST codeSystemVersion;
LIST<CR> qualifier;
} [4:05:12 PM] Grahame Grieve says: need to add rationale to CODE [4:05:26 PM] Lloyd McKenzie says: BEAUTIFUL :) [4:05:47 PM] Lloyd McKenzie says: Throw valueset as a set on codedConcept [4:05:55 PM] Grahame Grieve says: glad you like it. Doesn't actually solve your multiple thingy yet [4:05:59 PM] Gunther Schadow says: ??? [4:06:09 PM] Gunther Schadow says: What does that do? [4:06:21 PM] Lloyd McKenzie says: It gets rid of the silly concept of "root code" [4:06:30 PM] Grahame Grieve says: means that we clearly have a nullFlavor for the concept itself, and also one for each encoding [4:06:34 PM] Lloyd McKenzie says: And removes originalText from all the translations [4:06:37 PM] Gunther Schadow says: it's not silly. People wanted to know what is translated from what. [4:06:48 PM] Grahame Grieve says: oh. yes. hang on [4:06:48 PM] Lloyd McKenzie says: Root code doesn't give you that [4:06:49 PM] Gunther Schadow says: If you can go down a list you can go inside. [4:06:54 PM] Gunther Schadow says: There is no value in the difference. [4:07:30 PM] Lloyd McKenzie says: (any more - it used to, but they took the whole tree structure away). However, Grahame has a new funky way to handle the what was translated from what aspect - not shown here. [4:07:53 PM] Grahame Grieve says: type ConceptDescriptor alias CD specializes ANY {
ED originalText; SET(CODE) encodings;
};
type CodedConcept alias CODE specializes ANY {
ST code;
ST displayName;
UID codeSystem;
ST codeSystemName;
ST codeSystemVersion;
LIST<CR> qualifier;
SET<CS> rationale;
CD source;
} [4:07:57 PM] Gunther Schadow says: root used to. I agree it does not if things go according to Grahame and me. But the set of CV does not help it. [4:08:06 PM] Gunther Schadow says: "but they took the whole tree structure away" who is they? [4:08:10 PM] Gunther Schadow says: And how took they? [4:08:16 PM] Grahame Grieve says: what set of CV? [4:08:34 PM] Grahame Grieve says: source points you back to the one you were translated from, if you care [4:08:47 PM] Lloyd McKenzie says: They = vocab committee [4:08:52 PM] Gunther Schadow says: When? [4:09:00 PM] Lloyd McKenzie says: 2 WGMs ago at least [4:09:10 PM] Gunther Schadow says: Ask again you get another opinion. [4:09:21 PM] Gunther Schadow says: Doesn't mean anything. [4:09:32 PM] Grahame Grieve says: this also allows for more options in regard to CE, because you no longer have the root problem. You can make a constraint that any encodings that have rationale != O have no qualifiers [4:10:01 PM] Grahame Grieve says: this meets Lloyd's other problem and still enables proper translations [4:10:04 PM] Lloyd McKenzie says: Motion passed - everything is subject to change, but at the moment it's policy. Believe it was a joint InM Vocab meeting [4:10:30 PM] Gunther Schadow says: whatever [4:11:32 PM] Lloyd McKenzie says: Anyhow, I can accept that people are scared of trees and that the complexity of exposing the history of translations was more than many care about. The new structure gives the ability to convey it when necessary. while being simpler. [4:11:37 PM] Grahame Grieve says: Lloyd, you liked my proposal, but it doesn't solve your problem [4:12:08 PM] Lloyd McKenzie says: No, it still doesn't tell me whether the attribute should be OTH or not [4:12:17 PM] Grahame Grieve says: I like it, because it enables me to say clearly that the concept as a whole is encoded or not [4:12:28 PM] Grahame Grieve says: so we still have to separate conformance and nullness [4:12:47 PM] Grahame Grieve says: unless we go for SET<CS> for nullFlavor [4:12:53 PM] Grahame Grieve says: man, that's a good idea. I'm on fire today [4:13:15 PM] Lloyd McKenzie says: I'm trying to come up with the appropriate way to curse you . . . ;) [4:13:50 PM] Grahame Grieve says: but to be serious, if you entertain the concept of multiple models to conform to, then you must entertain the concept of separating nullFlavor from mandatory [4:14:15 PM] Lloyd McKenzie says: The issue isn't merely one of compliance. If something is CNE, not mandatory and I send originalText, I'm compliant, it's just not a valid value [4:14:28 PM] Grahame Grieve says: huh? [4:14:30 PM] Lloyd McKenzie says: I'm ok with separating nullFlavor from mandatory [4:14:41 PM] Grahame Grieve says: how can you be compliant in that case/ [4:14:42 PM] Grahame Grieve says: how can you be compliant in that case? [4:14:46 PM] Lloyd McKenzie says: It's only non-conformant if it's CNE and mandatory [4:14:58 PM] Lloyd McKenzie says: If it's not mandatory, I can send nulls [4:15:01 PM] Lloyd McKenzie says: Including OTH [4:15:05 PM] Grahame Grieve says: it's also non conformant if it's not valid [4:15:13 PM] Lloyd McKenzie says: Or more specifically "not null, but not fully specified" [4:15:39 PM] Lloyd McKenzie says: I can use OTH with CNE [4:16:28 PM] Grahame Grieve says: really? pl [4:16:30 PM] Grahame Grieve says: really? ok [4:19:26 PM] Lloyd McKenzie says: that's the only circumstance where OTH is needed [4:19:57 PM] Grahame Grieve says: no, you use it anytime you don't have a code. CNE/CWE is irrelevant [4:19:59 PM] Lloyd McKenzie says: With CWE, you're saying "this is perfectly valid even if no code from the preferred value set exists, so long as you send a local code or original text" [4:20:13 PM] Lloyd McKenzie says: Then what does CWE/CNE do? [4:20:59 PM] Grahame Grieve says: tells you whether you can define your own thing, so far as I can tell [4:21:09 PM] Grahame Grieve says: your own code, I mean [4:21:16 PM] Lloyd McKenzie says: Or send original text [4:21:26 PM] Lloyd McKenzie says: CWE *definitely* says you can send originalText [4:21:50 PM] Grahame Grieve says: so you can with CNE unless it is mandatory, right? [4:22:09 PM] Grahame Grieve says: and you can send originalText anyway, with CWE, mandatory or not? [4:22:22 PM] Lloyd McKenzie says: That's my understanding [4:22:55 PM] Lloyd McKenzie says: Which means that you can't have to send OTH if it's CWE [4:23:01 PM] Grahame Grieve says: but you have to use the nullFlavor OTH if you don't provide a code. [4:23:04 PM] Grahame Grieve says: so you do. [4:23:13 PM] Lloyd McKenzie says: But that rule is broken [4:23:17 PM] Grahame Grieve says: yeap.' [4:26:40 PM] Grahame Grieve says: not sure that my design is better, really. if you have more than one encoding, and they have different nullFlavors, what is the nullFlavor on the concept? [4:27:37 PM] Lloyd McKenzie says: I think we move away from null flavor. [4:27:44 PM] Lloyd McKenzie says: If you've got any properties, it's not null [4:27:52 PM] Lloyd McKenzie says: However, it may not be "complete" [4:28:11 PM] Lloyd McKenzie says: Complete means "meets the requirements of the specification" [4:28:23 PM] Grahame Grieve says: This is what I've been working towards for a long time: If you've got any properties, it's not null [4:28:41 PM] Lloyd McKenzie says: Mandatory says "must not be null and must be complete" [4:28:53 PM] Grahame Grieve says: but, well, we designed around nullFlavor with propertoes so strongly, I don't think it can be done [4:29:21 PM] Lloyd McKenzie says: Anything can be done :). What's so strong about null flavor with properties? [4:29:38 PM] Lloyd McKenzie says: I don't think we see it many places (yet) except CD [4:29:41 PM] Grahame Grieve says: huh [4:29:44 PM] Lloyd McKenzie says: And we're gutting and rebuilding it anyhow [4:29:47 PM] Grahame Grieve says: unknown home address [4:29:52 PM] Grahame Grieve says: unknown birth name [4:30:03 PM] Grahame Grieve says: unknown home phone number [4:30:07 PM] Lloyd McKenzie says: Ahh. Those are recent, but true. [4:30:19 PM] Grahame Grieve says: some of the things in PQ [4:30:26 PM] Grahame Grieve says: TRC. But still with units [4:30:35 PM] Lloyd McKenzie says: Can't have TRC with units [4:31:03 PM] Grahame Grieve says: yeah, that's what the theorists would say, but I know what anyone in my company would do. [4:31:06 PM] Lloyd McKenzie says: Or more specifically, "I can't think of any reason why units would be needed or relevant with TRC" [4:31:48 PM] Grahame Grieve says: actually, not strictly true. TRC mg/mL would be interpreted rather differently to TRC pg/L [4:31:52 PM] Lloyd McKenzie says: The uses are really narrowing down the "type" of data [4:31:57 PM] Grahame Grieve says: though I agree there's no rational grounds for doing so [4:32:15 PM] Lloyd McKenzie says: No rational grounds and misleading convention [4:32:37 PM] Grahame Grieve says: well, I can tell you that working lab guys won't let go of units on TRC [4:32:39 PM] Lloyd McKenzie says: If you want to express units, then do an IVL with an upper bound [4:32:51 PM] Grahame Grieve says: yeah, that's why TRC shouldn't exist at all [4:33:04 PM] Grahame Grieve says: it's wrong. Wrong wrong wrong, unless it's used in a CO [4:33:08 PM] Lloyd McKenzie says: It should, because you don't want to talk about units [4:33:12 PM] Grahame Grieve says: like for urinalusis [4:33:27 PM] Lloyd McKenzie says: TRC is about something like peanut oil [4:33:36 PM] Lloyd McKenzie says: "may contain a trace" [4:33:46 PM] Lloyd McKenzie says: Don't know how much, don't care. Know it's not a lot [4:33:58 PM] Grahame Grieve says: yeah, but what does it mean - how much is a trace? you shouldn't be able to say something like that without putting an upper bound on what a trace is [4:34:02 PM] Lloyd McKenzie says: Lab would probably never use it. [4:34:08 PM] Grahame Grieve says: never use what? [4:34:12 PM] Lloyd McKenzie says: TRC [4:34:19 PM] Grahame Grieve says: So who would? [4:34:24 PM] Lloyd McKenzie says: Pharmacy [4:34:29 PM] Grahame Grieve says: what's the use case? [4:34:35 PM] Lloyd McKenzie says: Talking about drug ingredients [4:34:55 PM] Lloyd McKenzie says: Specifically, non-active ingredients [4:35:09 PM] Grahame Grieve says: what - we are telling you to add a trace? [4:35:42 PM] Lloyd McKenzie says: When we want to say it contains red die #4, we don't know how much, we don't care. It's certainly not a significant constituent from a mass perspective. [4:36:08 PM] Lloyd McKenzie says: Not generally for ordering, but for drug masterfiles and talking about contraindications, allergic reactions, etc. [4:36:55 PM] Grahame Grieve says: well, I still reject the notion personally. No one should be allowed to be so careless. They should always give an upper bound [4:37:01 PM] Grahame Grieve says: but I know I won't win that one [4:38:36 PM] Lloyd McKenzie says: Upper bound doesn't tell you anything. [4:38:50 PM] Lloyd McKenzie says: (in this use case) [4:38:58 PM] Grahame Grieve says: maybe I just feel strongly about this because of experience. my daughter is allergic to cashew nuts. I know the amount that I need to be concerned about - it's not super small, like the kids that die from allergic shock, but it's not that high either, 1 whole cashew nut will be touch and go for her. So I'd like to know what "traces of nuts" really means [4:40:34 PM] Grahame Grieve says: > It's certainly not a significant constituent from a mass perspective. [4:40:42 PM] Grahame Grieve says: I'd like to know what you think is signficant [4:41:50 PM] Lloyd McKenzie says: It means that if I'm weighing ingredients and mixing them, is going to be less than 1% of overall mass. I.e. Not sufficient to be considered a base or something. [4:42:04 PM] Lloyd McKenzie says: Anyhow - this is a tangent. [4:42:10 PM] Grahame Grieve says: oh - 1%? so we do have an upper bound? [4:42:20 PM] Grahame Grieve says: but yes, it's a tangent. [4:42:34 PM] Lloyd McKenzie says: That's the one in my head, but it may be different for others. [4:42:49 PM] Lloyd McKenzie says: You've given one use-case for NULL with properties [4:43:01 PM] Lloyd McKenzie says: Where the properties are really refining the "type" of thing we're talking about [4:43:07 PM] Lloyd McKenzie says: Almost like structural attributes [4:43:51 PM] Lloyd McKenzie says: Could do the same thing for telecom on type - telephone or e-mail [4:44:07 PM] Lloyd McKenzie says: You said that already, never mind [4:44:40 PM] Lloyd McKenzie says: Could you do it for ED - Non-available JPEG? [4:45:00 PM] Grahame Grieve says: I guess that's another use case. [4:45:05 PM] Grahame Grieve says: hadn't thought of that one [4:45:38 PM] Grahame Grieve says: probably the new II properties are the same [4:45:55 PM] Lloyd McKenzie says: Except they're not supposed to be constrained . . . [4:46:12 PM] Lloyd McKenzie says: (I had thought of it and shivers started running up and down my spine) [4:46:34 PM] Grahame Grieve says: like nullFlavor : SET<CS>? I thought I'd get a response for that one [4:47:08 PM] Lloyd McKenzie says: That was more like a strong urge to retch ;) [4:47:22 PM] Grahame Grieve says: The verified identifier is MSK, but I can tell what I'm using... [4:47:46 PM] Lloyd McKenzie says: Ah. That new one. I thought you were talking "type" [4:48:02 PM] Lloyd McKenzie says: Possible, I suppose [4:48:09 PM] Grahame Grieve says: ouch, really, but there's a number of categorising properties like that where you need to be able to populate them when the thing is null [4:48:29 PM] Grahame Grieve says: II.type might be gone. I've been hounding Tom to give me a convincing use case. He's running out of time. [4:48:33 PM] Lloyd McKenzie says: Right. So we might want to declare that some properties are "categorizers" [4:48:46 PM] Lloyd McKenzie says: :) :) [4:49:19 PM] Lloyd McKenzie says: And categorizors can be present on null attributes just as structural attributes can be present on null associations. [4:50:04 PM] Lloyd McKenzie says: The tricky bit is that structural attributes *must* be present on associations. However, mandatory properties such as mime type don't have to be present on attributes. [4:51:06 PM] Grahame Grieve says: candidates are: II.scope AD.use EN.use TEL.use ED.lang ED.mediatype [4:51:25 PM] Grahame Grieve says: though what were we going to do with CD? [4:51:41 PM] Lloyd McKenzie says: Don't think there are categorizers for CD [4:51:49 PM] Grahame Grieve says: so, there's no OTH on CD? [4:52:10 PM] Lloyd McKenzie says: There's two separate use-cases for data on "null" things: [4:52:14 PM] Lloyd McKenzie says: - categorizors [4:52:25 PM] Lloyd McKenzie says: - supplemental (non-computable) information [4:52:37 PM] Grahame Grieve says: you mean updateMode etc? [4:52:42 PM] Lloyd McKenzie says: The latter is things like valueSet id and originalText on CD [4:52:48 PM] Lloyd McKenzie says: Or just an extension on an II [4:52:55 PM] Grahame Grieve says: yuck [4:52:57 PM] Lloyd McKenzie says: You've got data, just not enough [4:53:13 PM] Lloyd McKenzie says: It exists. Someone gives me a california drivers license number [4:53:24 PM] Lloyd McKenzie says: I stick it in as a search criteria and it brings back possible matches [4:53:53 PM] Lloyd McKenzie says: Not enough for a definitive match, but good enough to give me possibles [4:54:39 PM] Grahame Grieve says: oh right, this is that use case were root is null and extension is populated. [4:55:09 PM] Grahame Grieve says: but if that is a valid concept, why should it be "not enough"? either it is or it isn't. [4:55:25 PM] Grahame Grieve says: If we allow that use case, then there's no reason to call the II null; [4:57:22 PM] Lloyd McKenzie says: The II is no more null than the CD with originalText is [4:57:47 PM] Grahame Grieve says: well, I was getting to that ;) [4:58:06 PM] Lloyd McKenzie says: However, it's not a 'complete' II, in the sense that if II is mandatory, I'm going to be mad if you omit the root [4:59:01 PM] Lloyd McKenzie says: I think every type should have an "isComplete" property. I some cases, it might take an argument of a particular spec (e.g. CD where it depends on the value-set asserted by the spec) [4:59:09 PM] Grahame Grieve says: oh. we're talking about conformance again? [4:59:32 PM] Grahame Grieve says: why are you going to be mad if omit the root? not for that use case, for others, of course [4:59:33 PM] Lloyd McKenzie says: Conformance comes in where you add mandatory [4:59:51 PM] Grahame Grieve says: but you're back to null and conformance [5:00:28 PM] Lloyd McKenzie says: Mandatory is related to conformance and to null [5:01:09 PM] Grahame Grieve says: we want to break the second part [5:01:40 PM] Grahame Grieve says: null is related to the data itself (the instance, the object, the view), and not to anyone's expectations of it (the constraints, models, templates) [5:02:03 PM] Lloyd McKenzie says: I think we want to say "mandatory = not null and isComplete" [5:02:15 PM] Lloyd McKenzie says: I.e. Mandatory is about two things [5:02:21 PM] Grahame Grieve says: I think you're getting ahead of yourself. [5:02:38 PM] Lloyd McKenzie says: Though I guess the not null pieces is redundant because you couldn't be complete if you were null . . . [5:03:17 PM] Grahame Grieve says: isComplete has to be a set - complete with regard to more than one specification [5:03:43 PM] Lloyd McKenzie says: It does for CD for sure, not sure about others [5:03:53 PM] Grahame Grieve says: or, we actually require validator to figure it out directly based on what is present, not based on a short cut called nullFlavor, or a new shortcut called isComplete [5:04:13 PM] Lloyd McKenzie says: Or have it take an argument, such as a valueset id [5:04:42 PM] Grahame Grieve says: well, it needs a series of arguments, one of which is domain/valueSet/codeSystem what ever [5:04:52 PM] Lloyd McKenzie says: The datatypes need to be clear what the rules are to constitute a "complete" attribute value [5:06:15 PM] Grahame Grieve says: why is it the datatypes problem? [5:06:25 PM] Grahame Grieve says: it's the model's problem. [5:07:07 PM] Lloyd McKenzie says: Don't think so. The rules for what constitutes a "complete" II, CD, TEL, etc. should be consistent. [5:07:46 PM] Grahame Grieve says: no. you have a case where complete means it has an extension and no root [5:07:49 PM] Lloyd McKenzie says: If I say "You must give me an X", I need to be confident in what I'm getting and shouldn't have to say any more than "You must give me" [5:07:57 PM] Grahame Grieve says: but then most of your cases need a root [5:07:58 PM] Lloyd McKenzie says: I don't think that's complete. [5:08:15 PM] Grahame Grieve says: it's complete for a purpose. [5:08:17 PM] Lloyd McKenzie says: I think that's useful, but wouldn't fit the criteria of mandatory. [5:08:32 PM] Lloyd McKenzie says: It's got potentially useful information, but it's not complete [5:08:54 PM] Grahame Grieve says: well, it doesn't meet the invariants so it's far less than complete at the moment [5:09:05 PM] Grahame Grieve says: I guess you want a default conformance profile for a datatype [5:09:07 PM] Lloyd McKenzie says: II is complete if it has root and extension (and maybe scope-not sure if that's required or not) [5:09:23 PM] Lloyd McKenzie says: Sort of. [5:09:46 PM] Grahame Grieve says: separation see. Datatypes is about what's possible. Conformance is about what's allowed where [5:09:58 PM] Lloyd McKenzie says: Datatypes is about both [5:10:24 PM] Lloyd McKenzie says: Datatypes are about constraints [5:10:33 PM] Lloyd McKenzie says: Conformance is about enforcement of constraints [5:11:03 PM] Lloyd McKenzie says: We have a high level concept at the attribute and association level called mandatory that says "must be present". [5:11:11 PM] Grahame Grieve says: yes and no. Datatypes are about base level - things that are always true [5:11:16 PM] Lloyd McKenzie says: For it to be meaningful, we need to know what "present" means [5:11:50 PM] Grahame Grieve says: yes, I agree. and the more usecases we run into, the more we back off the innate requirements of the datatype - like making ii.root optional [5:12:05 PM] Grahame Grieve says: so we put more into conformance. [5:12:41 PM] Lloyd McKenzie says: But we still need a common definition of "present" [5:13:10 PM] Grahame Grieve says: meets the default conformance requirements unless other requirements are specified [5:14:56 PM] Lloyd McKenzie says: So a datatype flavor might assert additional constraints [5:15:23 PM] Lloyd McKenzie says: And the datatype would need to meet those to be considered "present" [5:15:33 PM] Grahame Grieve says: yes [5:15:43 PM] Lloyd McKenzie says: Or it would be considered "OTH". [5:16:01 PM] Lloyd McKenzie says: And 5 templates in play might each declare a different flavor [5:16:14 PM] Grahame Grieve says: huh> where did OTH get into the picture? [5:18:03 PM] Lloyd McKenzie says: Maybe it doesn't. My brain is too fuzzy to tell right now [5:18:18 PM] Grahame Grieve says: ok. pick it up later. [5:18:34 PM] Grahame Grieve says: I'll knock this into a r2 proposal. That one will be gunf [5:18:37 PM] Grahame Grieve says: I'll knock this into a r2 proposal. That one will be fun [5:18:44 PM] Lloyd McKenzie says: Indeed :)
