Exclusions

By default, Touchstone will treat uploaded files with “.json”, “.xml”, “.groovy”, “.sch”, and “.xslt” extensions as test definitions.

  • JSON and XML test definitions are parsed during upload and are expected to be in proper format or the upload will fail.

  • Files with “.groovy”, “.sch”, and “.xslt” extensions are treated as rule definitions and are expected to contain certain content in its headers or the upload will fail.

  • Uploaded test definitions that contain FHIR resources will be validated periodically using the FHIR Validation Engine and will be flagged for validation errors.

All these restrictions would prevent end users from:

  • Using invalid JSON and XML in their fixtures for negative testing.

  • Storing Schematron and XSLT files alongside their test definitions for schema-validation and transformation purposes.

  • Temporarily excluding test definitions from validation by a validation engine.

  • etc.

Props Location

Touchstone provides the ability to exclude folders and files from upload, parsing, and validations via exclusions in groupProps.json. This file must reside at the root of the main test group. For example, if the main test group that’s being uploaded is FHIR3-0-1-Basic, then groupProps.json file must reside at FHIR3-0-1-Basic:

../_images/exclusion_uploadProps_location_a1.png

If the file is incorrectly located one-level deeper (e.g. at FHIR3-0-1-Basic/D-H), then attempting to upload the test group will produce the following error:

../_images/exclusion_uploadProps_wronglocation2_a1.png

Props format

Regular Expressions are supported in the exclusions. There are four types of exclusions:

  1. excludeFromUpload -> Completely prevents test definitions from ending up in Touchstone.

  2. excludeFromParsing -> Test definitions are uploaded to Touchstone but they’re not parsed or validated.

  3. excludeFromValidation -> Test definitions are uploaded to Touchstone and are parsed but they’re not periodically validated using external Validaiton Engine.

  4. excludeFromConformance -> Test definitions are uploaded to Touchstone and are parsed but they’re excluded from getting included in conformance evaluations.

  • Test definitions that are specified in excludeFromUpload do not need to be specified in excludeFromParsing and excludeFromValidation as they’ll be excluded from upload completely.

  • Test definitions that are specified in excludeFromParsing do not need to be specified in excludeFromValidation as they’ll neither be parsed nor validated.

    Warning

    Minimize the number of entries in excludeFromUpload, excludeFromParsing, and excludeFromValidation as each entry is evaluated separately during upload. Upload could take long if there are too many entries. It’s better to agree upon a naming convention in your organization and use a regular expression that matches the agreed-upon convention.

We will cover various scenarios via examples next.

Upload Exclusions

Upload exclusions are specified using the excludeFromUpload key in groupProps.json. Values can be specified using regular expressions. Matching folders and files will be filtered out from the upload and will not end up in Touchstone.

To exclude the Device test group from getting uploaded to Touchstone:

../_images/exclusion_device_testgroup2_a1.png 
{
   "excludeFromUpload": [
      ".*/FHIR3-0-1-Basic/D-H/Device/.*"
   ]
}

After the FHIR3-0-1-Basic test group is uploaded, the Device sub group will not be there in Touchstone:

../_images/exclusion_device_testgroup3.png

Note

After the groupProps.json file ends up in Touchstone, you can upload sub-groups and files individually as usual i.e. You do not have to upload the main test group (FHIR3-0-1-Basic in this case) every time. Touchstone will enforce the exclusions based on the last version of groupProps.json in the system for the main test group. If you need to update the exclusions, you can re-upload the main test group (FHIR3-0-1-Basic in this case) which includes the updated groupProps.json. You can alternatively upload the updated groupProps.json file alone for the new exclusions to take effect.

To exclude test groups that start with “Device” from getting uploaded to Touchstone:

../_images/exclusion_devicebegins_testgroups_a1.png 
{
   "excludeFromUpload": [
      ".*/FHIR3-0-1-Basic/D-H/Device.*"
   ]
}

After the FHIR3-0-1-Basic test group is uploaded, the “Device”, “DeviceComponent”, and “DeviceMetric” sub groups will not be there in Touchstone:

../_images/exclusion_devicebegins_testgroups2.png

To exclude “D-H” sub group from getting uploaded to Touchstone:

../_images/exclusion_dh_subgroup_a1.png 
{
   "excludeFromUpload": [
      ".*/D-H/.*"
   ]
}

After the FHIR3-0-1-Basic test group is uploaded, the “D-H” sub group will not be there in Touchstone:

../_images/exclusion_dh_subgroup2.png

To exclude “.sch” and “.xslt” files in Devices sub group from getting uploaded to Touchstone:

../_images/exclusion_all_schAndXslt_a1.png 
{
   "excludeFromUpload": [
      ".*/Device/.*\\.sch",
      ".*/Device/.*\\.xslt"
   ]
}

or

{
   "excludeFromUpload": [
      ".*/Device/.*\\.(sch|xslt)"
   ]
}

The second one above is better as it involves fewer evaluations during upload and thereby slightly better performance.

Note

Certain characters must be escaped when included in a json string. The regex backslash must be escaped using another backslash, hence the double backslash in the example.

After the FHIR3-0-1-Basic test group is uploaded, the “.sch” and “.xslt” files in Device sub group will not be there in Touchstone but those in List sub group will be:

../_images/exclusion_device_schAndXslt1.png ../_images/exclusion_device_schAndXslt2.png

To exclude all “.sch” and “.xslt” files from getting uploaded to Touchstone:

../_images/exclusion_all_schAndXslt_a1.png 
{
   "excludeFromUpload": [
      ".*\\.sch",
      ".*\\.xslt"
   ]
}

or

{
   "excludeFromUpload": [
      ".*\\.(sch|xslt)"
   ]
}

The second one above is better as it involves fewer evaluations during upload and thereby slightly better performance.

After the FHIR3-0-1-Basic test group is uploaded, the “.sch” and “.xslt” files will not be anywhere under FHIR3-0-1-Basic in Touchstone:

../_images/exclusion_all_schAndXslt2.png ../_images/exclusion_all_schAndXslt3.png

Parsing Exclusions

Parsing exclusions are specified using the excludeFromParsing key in groupProps.json. Values can be specified using regular expressions. Matching folders and files will still end up in Touchstone but will be filtered out from parsing and periodic validations.

If your folders contain Groovy, Schematron and XSLT files, for example, and you want Touchstone to not treat them as rule definitions, then you can specify a regular expression that matches those files in excludeFromParsing.

You can do the same with JSON and XML fixtures that are not even proper JSON or XML and you want to use them for negative testing.

To exclude “.sch” and “.xslt” files in all sub groups from getting parsed during upload (and treated as rule definitions):

../_images/exclusion_all_schAndXslt_a1.png 
{
   "excludeFromParsing": [
      ".*\\.sch",
      ".*\\.xslt"
   ]
}

After the FHIR3-0-1-Basic test group is uploaded, the “.sch” and “.xslt” files do get included in the upload but they will not be parsed and treated as rule definitions:

../_images/exclusion_parsing_all_schAndXslt2.png ../_images/exclusion_parsing_all_schAndXslt3.png

To exclude certain files from getting parsed and checked for valid JSON or XML:

../_images/exclusion_wrong_xml_format_a1.png 
{
   "excludeFromParsing": [
      ".*/FHIR3-0-1-Basic/D-H/Device/_reference/bad_format.xml"
   ]
}

After the FHIR3-0-1-Basic test group is uploaded, the “bad_format.xml” file does get included in the upload but it will not be parsed and checked for proper XML. It will also avoid periodic validations

../_images/exclusion_wrong_xml_format2.png

Warning

Minimize the number of entries in excludeFromParsing as each entry is evaluated separately during upload. Upload could take long if there are too many entries. It’s better to agree upon a naming convention in your organization and use a regular expression that matches the agreed-upon convention. For example, all such invalid formats could end in “_improper_format.json” or “_improper_format.xml”. Then you could exclude all these files from parsing by specifying a single regular expression that matches all those files:

To exclude all files that end in ‘_improper_format’ from getting parsed and format-checked:

../_images/exclusion_improper_format_a1.png 
{
   "excludeFromParsing": [
      ".*_improper_format\\.(json|xml)"
   ]
}

Validation Exclusions

Validation exclusions are specified using the excludeFromValidation key in groupProps.json. Values can be specified using regular expressions. Matching folders and files will still end up in Touchstone but will be filtered out from periodic validations. JSON and XML files will be parsed though during upload and will be expected to contain proper JSON or XML.

This feature is useful if you want to avoid test definitions from getting marked for validation failures by external Validation Engine.

To exclude all test definitions in the Device test group from getting periodically validated:

../_images/exclusion_device_testgroup2_a1.png 
{
   "excludeFromValidation": [
      ".*/FHIR3-0-1-Basic/D-H/Device/.*"
   ]
}

Conformance Exclusions

Conformance exclusions are specified using the excludeFromConformance key in groupProps.json. Values can be specified using regular expressions. Matching folders and files will still end up in Touchstone but will be filtered out from getting included in Conformance results. JSON and XML files will be parsed though during upload and will be expected to contain proper JSON or XML.

This feature is useful if you’re testing a new operation in your test scripts and Touchstone is unable to map it to its known operation list or if you simply don’t want certain test scripts from getting included in conformance results (but still want it as part of the test groups).

Note that changes to the content of such test scripts will still increment the Conformance Suite version if they’re part of the test group referenced by the suite.

To exclude all test definitions in the Device test group from getting included in Conformance results:

../_images/exclusion_device_testgroup2_a1.png 
{
   "excludeFromConformance": [
      ".*/FHIR3-0-1-Basic/D-H/Device/.*"
   ]
}