Test Case Advisor¶
The TrialGrid Test Case Advisor suggests Test Cases for Edit Checks by analyzing the Check Steps and generating Test Steps which will trigger the Check Actions.
For more information about the structure of Test Cases see Test Cases
Running the Test Case Advisor¶
The Test Case Advisor can be run for a single Edit Check by opening the Edit Check editor, expanding the side bar and clicking the ‘Add’ or ‘Replace’ button. ‘Replace’ will delete any existing Test Cases related to that Edit Check before creating a new one.
The Test Case Advisor can be run for multiple Edit Checks from the Edit Checks list page by clicking the ‘Test Case Advisor’ button. This will create Test Cases for the selected Edit Checks. Existing Test Cases may be deleted by selecting the ‘Delete Existing’ option. If ‘Delete Existing’ is not selected then the Test Case Advisor will create Test Cases only for Edit Checks which do not already have a Test Case. Edit Checks which are inactive or invalid (for example, have no Check Actions defined) will not have Test Cases suggested. A default EDC Role should be selected which will be suitable for running the Test Cases.
Test Case Advisor Results¶
The Test Case Advisor uses the enter and save:when_i_enter_and_save_data data entry step because that step simulates most closely a user entering data into Rave, making sure that custom functions and derivations on the form are triggered.
Test Cases created by the Test Case Advisor will be in one of these states:
Ready to run¶
For most Edit Checks the Test Case Advisor can suggest a valid Test Case containing two Scenarios; one which should cause the Check Actions to run and one which should not trigger the Check Actions. These Test Case should be runnable without requiring any modifications but we do recommend that all suggested Test Cases are reviewed to ensure the Edit Check is being tested optimally.
Informational notes will be added to the Scenario descriptions if the Test Case Advisor has:
detected that a Field has a Default Value and so does not need to have a Data Entry step
inserted data entry steps to make a Field visible. If the Edit Check references a Field which is not visible by default then the Test Case Advisor will look for an Edit Check with a SetDataPointVisible action for that Field and will then add the steps necessary to make the Field visible.
inserted data entry steps to derive a Field value. If the Edit Check references a Field which is the target of a Derivation then the Test Case Advisor will analyze the Derivation and insert data entry steps to derive an appropriate value for the Field. Sometimes the Test Case Advisor will not be able to calculate values for the derived Field which will also trigger the Edit Check in which case an issue will be added and the Test Case will be marked as invalid.
The following example shows how a data entry step has been added for the Date of Birth and Date of Informed Consent fields which are used in Derivation “DER_AGE” to calculate the age. The Age Field is used in Edit Check “AGE_RANGE”.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
@EditCheck:AGE_RANGE @Derivation:DER_AGE @Form:DM Feature: Test Edit Check AGE_RANGE Background: Given I am logged in with role "Investigator" And a subject exists Scenario: Check Actions are run Note: Field "AGE" on Form "DM" is derived by Derivation "DER_AGE" When I enter and save data: | DataPoint | Value | | SCREENING.SUB.ICDAT | 01 JAN 2019 | And I enter and save data: | DataPoint | Value | | SCREENING.DM.BRTHDAT | 01 JAN 1979 | Then I should see the following query on field "SCREENING.DM.AGE" """ Subject age does not meet inclusion criteria """
For a small number of Edit Checks a valid Test Case can be suggested but the Test Case Advisor has made some assumptions which should be reviewed.
The assumptions are added as comments in the suggested Test Case. The assumptions include:
If the Edit Check references a Variable, with no specific Field specified in the Edit Check then the Test Case Advisor must find an appropriate Field on which to do data entry. If there is only one Field associated with that Variable then the selection is obvious but if there are multiple Fields the Test Case Advisor will order them by Form Ordinal and Field Ordinal and select the first one (ie. the Form with the lowest Ordinal number and the Field on that Form which is associated with the Variable).
If the Edit Check has one or more wildcarded Folders then the Test Case Advisor must find a suitable Folder for that Field and Form, using this strategy. Note that the same Folder must be selected for all wildcarded references because the Rave edit check engine will insert one Folder into all Folder wilcards when processing the edit check.
Are all of the Forms for wildcarded Folders present in the same Folder in the default Matrix?
Is it the Primary Form (in which case it will be in the SUBJECT Folder)?
Is there an AddForm Check Action on another Edit Check which will help us find the appropriate Folder?
Are all of the Forms for wildcarded Folders present in the same Folder in any Matrix, except an ‘All Forms’ matrix?
Is a specific Folder referenced elsewhere in the Edit Check which we can use?
If no Folder has been found in the above steps then take the first Folder, ie. the Folder with the lowest Ordinal. In this case a comment will be added to the Test Case.
The following example shows comments added for variable and folder selection:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
@EditCheck:AGE_RANGE @Derivation:DER_AGE @Form:DM # Using Field ICDAT on Form SUB for Variable CDAT # Wildcard Folder replaced with SCREENING Feature: Test Edit Check AGE_RANGE Background: Given I am logged in with role "Investigator" And a subject exists Scenario: Check Actions are run Note: Field "AGE" on Form "DM" is derived by Derivation "DER_AGE" When I enter and save data: | DataPoint | Value | | SCREENING.SUB.ICDAT | 01 JAN 2019 | And I enter and save data: | DataPoint | Value | | SCREENING.DM.BRTHDAT | 01 JAN 1979 | Then I should see the following query on field "SCREENING.DM.AGE" """ Subject age does not meet inclusion criteria """
Some Edit Checks, mainly those using Custom Functions, cannot be created in a complete and valid state because the Test Case Advisor is unable to work out how the Edit Check should be triggered and/or what actions should be taken. For these Edit Checks the Test Case Advisor will create as much of the Test Case as it can and will list the issues which need to be addressed in a table at the start of each scenario. The list of issues should be reviewed and the appropriate steps added to the Test Case, after which the issues should be removed from the Test Case.
The Test Case Advisor is unable to analyse the source code of Custom Functions and Dynamic Search Lists and so test steps must be added to take the appropriate actions for them. An ‘always true’ Custom Function, eg. one that simply does ‘return true’ does not require additional work in the Test Case.
The Test Case Advisor might not be able to find a solution for some Edit Checks, either because there are no possible values which would trigger the Check Actions (for example, a Dictionary Field being compared against a static value which does not exist in the Dictionary) or in rare cases because the Test Case Advisor has not been able to find suitable values.
A Field in a Data Entry step is View or Entry Restricted, at Field or Form level, to the default role. The Test Case Advisor will insert an additional log in step before the restricted Field which will need to be edited to select an appropriate EDC Role. Note that Fields which have a default value will not appear in the data entry step because Rave will automatically enter the default value, and so they will not require this additional log in step. Similarly Fields which are Derivation targets will not appear directly in data entry steps and do not require additional log in steps if they are view or entry restricted.
Edit Checks using Logical Record Position will need additional work to add extra values in the appropriate logical sequence.
The Test Case Advisor will not be able to find suitable values for all Derivations in which case this will be noted as an issue.
If a Field is not visible by default and no Check Action could be found to make it visible then steps must be inserted to make it visible (probably by triggering a Custom Function in another Edit Check).
If a Field and/or Form is inactive then the Edit Check and Test Case must be reviewed.
File Upload and Signature controls are not supported by TrialGrid Test Cases. If an Edit Check references such a Field it will need to be manually tested.
Some Check Actions are not supported by TrialGrid Test Cases and will need to be manually tested.
For more information about the limitations of Test Cases see Automated Testing Limitations
The following example shows a Test Case where the Edit Check references a derived Field and the Test Case Advisor could not find suitable values. The data entry steps are blank and should be manually completed. The Edit Check has a Custom Function action which must also be changed to the appropriate steps:
After review the Test Case might look like this:
When running the Test Case Advisor for a batch of Edit Checks the resulting Test Cases can be labelled based on the Test Case Result (Ready to run, Review recommended, Requires attention) to make searching the results easier.
The Test Case Advisor will insert a cross-reference to the Edit Check, to the Form(s) referenced in Check Steps and Actions, to any Custom Functions used in the Edit Check and to any Derivations needed to populate derived Fields.
How does the Test Case Advisor work?¶
The Test Case Advisor looks for a set of data values for the Fields referenced in the Edit Check steps which will satisfy the logic of the Check Steps and trigger the Check Actions, and a set of data values which will not trigger the Actions and uses these values to construct two Test Case scenarios.
To find data values the Test Case Advisor looks at each Field and generates a list of potential test values based on the data type (integer, string, date), format and other properties such as ranges and dictionary values where applicable. The Test Case Advisor will not choose values which would make the Field non-conformant.
These test values are then processed by a constraint solver <https://en.wikipedia.org/wiki/Constraint_programming> which selects values which trigger (or not) the Check Actions.
In rare cases the constraint solver might not find a solution and an issue will be added to the Test Case (see below).
Why doesn’t the Test Case Advisor generate all possible combinations of test values?¶
The two test scenarios suggested by the Test Case Advisor can be easily copied in the Test Case Editor if more combinations of data are desired for testing. The Test Case Advisor does not attempt to generate all possible combinations because the number of scenarios becomes surprisingly large.
For example, a common type of Edit Check might compare specific Inclusion/Exclusion criteria and the Eligibility Status:
Here there are 5 Fields each of which can have 3 values (Yes, No and empty) leading to 3^5 = 243 different combinations. Already this is more than anyone would probably want to test but this example was taken from a study which actually has 9 Inclusion critera and 36 Exclusion criteria leading to 8,862,938,119,652,501,095,929 possibilities!