This wiki has undergone a migration to Confluence found Here
<meta name="googlebot" content="noindex">

Difference between revisions of "201805 Bulk Data"

From HL7Wiki
Jump to navigation Jump to search
(Created page with "Jan 2018 Proposals __NOTOC__ =Track Name= Bulk Data Access ([http://www.healthintersections.com.au/?p=2689 see blog post...")
 
(→‎Expected participants: added Danielle for Epic)
 
(5 intermediate revisions by 3 users not shown)
Line 1: Line 1:
[[Category:201805_FHIR_Connectathon_Track_Proposals|Jan 2018 Proposals]]
+
[[Category:201805_FHIR_Connectathon_Track_Proposals|May 2018 Proposals]]
 
__NOTOC__
 
__NOTOC__
 
=Track Name=
 
=Track Name=
  
Bulk Data Access ([http://www.healthintersections.com.au/?p=2689 see blog post])
+
Bulk Data Access
  
 
=Track Overview=
 
=Track Overview=
  
Please review: [https://docs.google.com/presentation/d/1QpMUIohFEJJcxxrcWfx80sRtXDWFIQBCIKw1BXsy454/present Slide Presentation from Kick-Off] and [https://youtu.be/y9_VUgv9q84 Orientation Meeting Recording]
+
Please review:  
 +
* [https://docs.google.com/presentation/d/14ZHmam9hwz6-SsCG1YqUIQnJ56bvSqEatebltgEVR6c/edit?usp=sharing Overview Slide Presentation]  
 +
* [https://github.com/smart-on-fhir/fhir-bulk-data-docs Draft Implementation Guide]
  
Please fill out: [https://docs.google.com/spreadsheets/d/11QbX8iB49s_-YMP6GmEej41uT58ir4f6W2EvMEe4cUg/edit#gid=1088922430 Track Registration Spreadsheet]
+
Please fill out:  
 +
* Track Registration Spreadsheet
  
 
Bulk Data API reference implementations:  
 
Bulk Data API reference implementations:  
* [https://github.com/smart-on-fhir/bulk-data-server Bulk Data Server (NodeJs)] ([https://bulk-data.smarthealthit.org online demo])
+
* [https://github.com/smart-on-fhir/bulk-data-server SMART Bulk Data Server (NodeJs)] ([https://bulk-data.smarthealthit.org online demo])
* [https://github.com/smart-on-fhir/sample-apps-stu3/tree/master/fhir-downloader Sample command line client (NodeJs)]   
+
* [https://github.com/smart-on-fhir/sample-apps-stu3/tree/master/fhir-downloader SMART Sample command line client (NodeJs)]   
 +
* [https://github.com/smart-on-fhir/fhir-bulk-data-docs/blob/master/README.md#server-implementations Other implementations]
  
 
==Submitting WG/Project/Implementer Group==
 
==Submitting WG/Project/Implementer Group==
Line 30: Line 34:
 
==Proposed Track Lead==
 
==Proposed Track Lead==
  
Dan Gottlieb and Josh Mandel with support from Grahame Grieve ([[Connectathon_Track_Lead_Responsibilities]])
+
Josh Mandel with support from Grahame Grieve ([[Connectathon_Track_Lead_Responsibilities]])
  
 
==Expected participants==
 
==Expected participants==
  
 
* Grahame Grieve
 
* Grahame Grieve
* Dan Gottlieb
 
 
* Josh Mandel
 
* Josh Mandel
 
* Cerner  
 
* Cerner  
* Epic
+
* Danielle Friend, Epic
 
* CARIN health alliance
 
* CARIN health alliance
 
* Ken Kawamoto (along with CDS Hooks)
 
* Ken Kawamoto (along with CDS Hooks)
Line 55: Line 58:
 
# Secured bulk data export using SMART ([http://docs.smarthealthit.org/authorization/backend-services/ backend services specification])
 
# Secured bulk data export using SMART ([http://docs.smarthealthit.org/authorization/backend-services/ backend services specification])
  
==Scenario 1: Full Bulk Data Export (Open Endpoint)==
+
==Scenario 1: Targeted Bulk Data Export (Open Endpoint)==
  
 
See https://github.com/smart-on-fhir/fhir-bulk-data-docs for a description of the workflow.
 
See https://github.com/smart-on-fhir/fhir-bulk-data-docs for a description of the workflow.
 +
 +
Export targeted by Group, with optional state-date and resource-type filters.
  
 
====Action====
 
====Action====
  
1. Data Consumer requests a bulk data export
+
1. Data Consumer issues one or more of the following requests:
 +
 
 +
<pre>
 +
GET [base]/Group/[id]/$export?_outputFormat=ndjson&$export
 +
Accept: application/fhir+json
 +
Prefer: respond-async
 +
</pre>
  
 
<pre>
 
<pre>
GET [base]/Patient/$export?_outputFormat=ndjson
+
GET [base]/Group/[id]/$export?_outputFormat=ndjson&_since=[date-time]&_type=[FHIR Resource Type],[FHIR Resource Type]
 
Accept: application/fhir+json
 
Accept: application/fhir+json
 
Prefer: respond-async
 
Prefer: respond-async
Line 81: Line 92:
 
7. Optionally, Data Consumer may ETL and process these files.
 
7. Optionally, Data Consumer may ETL and process these files.
  
==Scenario 2: Targeted Bulk Data Export (Open Endpoint)==
+
==Scenario 2: Secured Bulk Data Export (SMART Backend Services Protected Endpoint)==
 +
 
 +
====Action====
 +
 
 +
1. Data Consumer registers itself with Data Provider and obtains an access token as described in the SMART ([http://docs.smarthealthit.org/authorization/backend-services/ backend services specification])
  
https://github.com/smart-on-fhir/fhir-bulk-data-docs for a description of the workflow.
+
2. Data Consumer and Provider follow the workflows described in Scenario 2 with the addition of an authorization header in each request. If the <code>X-FHIR-Links-Require-Authorization</code> header in the final async response is not set to <code>true</code>, the Data Consumer should not include the authorization token in the file download requests.
  
As in Scenario 1, but with a more targeted export request.
+
 
 +
==Scenario 3: Full Bulk Data Export ==
 +
 
 +
This request is for export across the entire Patient population, rather than within a specific group.
  
 
====Action====
 
====Action====
  
1. Data Consumer issues one or more of the following requests:
+
Data Consumer requests a bulk data export
<pre>
 
GET [base]/Patient/$export?_outputFormat=ndjson&start=[date-time]&_type=[FHIR Resource Type],[FHIR Resource Type]
 
Accept: application/fhir+json
 
Prefer: respond-async
 
</pre>
 
 
 
<pre>
 
GET [base]/Group/[id]/$export?outputFormat=ndjson&$everything
 
Accept: application/fhir+json
 
Prefer: respond-async
 
</pre>
 
  
 
<pre>
 
<pre>
GET [base]/Group/[id]/$export?outputFormat=ndjson&start=[date-time]&_type=[FHIR Resource Type],[FHIR Resource Type]
+
GET [base]/Patient/$export?_outputFormat=ndjson&_since=[date-time]&_type=[FHIR Resource Type],[FHIR Resource Type]
 
Accept: application/fhir+json
 
Accept: application/fhir+json
 
Prefer: respond-async
 
Prefer: respond-async
 
</pre>
 
</pre>
  
2. Subsequent workflow proceeds as in Scenario 1
 
 
==Scenario 3:  Secured Bulk Data Export (SMART Backend Services Protected Endpoint)==
 
 
====Action====
 
 
1. Data Consumer registers itself with Data Provider and obtains an access token as described in the SMART ([http://docs.smarthealthit.org/authorization/backend-services/ backend services specification])
 
  
2. Data Consumer and Provider follow the workflows described in Scenario 1 and 2 with the addition of an authorization header in each request. If the <code>X-FHIR-Links-Require-Authorization</code> header in the final async response is not set to <code>true</code>, the Data Consumer should not include the authorization token in the file download requests.
 
  
 
==TestScript(s)==
 
==TestScript(s)==

Latest revision as of 18:20, 20 April 2018


Track Name

Bulk Data Access

Track Overview

Please review:

Please fill out:

  • Track Registration Spreadsheet

Bulk Data API reference implementations:

Submitting WG/Project/Implementer Group

FHIR-I


Justification

Argonaut is taking up Bulk Data and Backend Services Authorization as a core project this year. Goals include:

  • Ecosystem outcome expected to enable many specific use case/business needs: Providers and organizations accountable for managing the health of populations can efficiently access to large volumes of information on a specified group of individuals without having to access one record at a time. This population-level access would enable these stakeholders to: assess the value of the care provided, conduct population analyses, identify at-risk populations, and track progress on quality improvement.
  • Technical Expectations: There would be a standardized method built into the FHIR standard to support access to and transfer of a large amount of data on a specified group of patients and that such method could be reused for any number of specific business purposes.
  • Policy Expectations: All existing legal requirements for accessing identifiable patient information via other bulk methods (e.g., ETL) used today would continue to apply (e.g., through HIPAA BAAs/contracts, Data Use Agreements, etc).

Proposed Track Lead

Josh Mandel with support from Grahame Grieve (Connectathon_Track_Lead_Responsibilities)

Expected participants

  • Grahame Grieve
  • Josh Mandel
  • Cerner
  • Danielle Friend, Epic
  • CARIN health alliance
  • Ken Kawamoto (along with CDS Hooks)

Roles

  • Data Provider: provides data in the manner specified by the bulk data API
  • Data Consumer: consumes data in the manner specified by the bulk data API and displays/processes the data

Scenarios

The bulk data track is divided into the following scenarios:

  1. Full bulk data export, open server without security
  2. Targeted bulk data export, open server without security
  3. Secured bulk data export using SMART (backend services specification)

Scenario 1: Targeted Bulk Data Export (Open Endpoint)

See https://github.com/smart-on-fhir/fhir-bulk-data-docs for a description of the workflow.

Export targeted by Group, with optional state-date and resource-type filters.

Action

1. Data Consumer issues one or more of the following requests:

GET [base]/Group/[id]/$export?_outputFormat=ndjson&$export
Accept: application/fhir+json
Prefer: respond-async
GET [base]/Group/[id]/$export?_outputFormat=ndjson&_since=[date-time]&_type=[FHIR Resource Type],[FHIR Resource Type]
Accept: application/fhir+json
Prefer: respond-async

2. Data Provider responds with a location for progress updates

3. Data Consumer requests a progress update

4. Data Provider responds with the operation's interim status (optional)

5. Data Provider responds with links to the generated data files

6. Data Consumer requests each of the generated files

7. Optionally, Data Consumer may ETL and process these files.

Scenario 2: Secured Bulk Data Export (SMART Backend Services Protected Endpoint)

Action

1. Data Consumer registers itself with Data Provider and obtains an access token as described in the SMART (backend services specification)

2. Data Consumer and Provider follow the workflows described in Scenario 2 with the addition of an authorization header in each request. If the X-FHIR-Links-Require-Authorization header in the final async response is not set to true, the Data Consumer should not include the authorization token in the file download requests.


Scenario 3: Full Bulk Data Export

This request is for export across the entire Patient population, rather than within a specific group.

Action

Data Consumer requests a bulk data export

GET [base]/Patient/$export?_outputFormat=ndjson&_since=[date-time]&_type=[FHIR Resource Type],[FHIR Resource Type]
Accept: application/fhir+json
Prefer: respond-async


TestScript(s)

This is an API extension, and will require extensions to the test script resource in order to be tested

Security and Privacy Considerations

  • Obviously, access to APIs like this in production require both authentication and consent
  • Step 3 tests out application authentication
  • For now, it is assumed that consent is managed elsewhere, though extensions may be added to the stream for this (see [[1]])
  • Audit: For now, it is assumed that applications will audit the initial FHIR retrieval, and a smart on fhir login, but there are no rules about that
  • The X-FHIR-Links-Require-Authorization header is a proposal to support both servers that use SMART authentication to secure the generated files and those that leverage other techniques (eg S3 signed URLs).