Test Case Advisor

The TrialGrid Test Case Advisor suggests Test Cases for Edit Checks by analyzing the Check Steps and generating Test Steps which will trigger the Check Actions.

See also

For more information about the structure of Test Cases see Test Cases

Running the Test Case Advisor

The Test Case Advisor can be run for a single Edit Check by opening the Edit Check editor, expanding the side bar and clicking the ‘Add’ or ‘Replace’ button. ‘Replace’ will delete any existing Test Cases related to that Edit Check before creating a new one.

The Test Case Advisor can be run for multiple Edit Checks from the Edit Checks list page by clicking the ‘Test Case Advisor’ button. This will create Test Cases for the selected Edit Checks. Existing Test Cases may be deleted by selecting the ‘Delete Existing’ option. If ‘Delete Existing’ is not selected then the Test Case Advisor will create Test Cases only for Edit Checks which do not already have a Test Case. Edit Checks which are inactive or invalid (for example, have no Check Actions defined) will not have Test Cases suggested. A default EDC Role should be selected which will be suitable for running the Test Cases.

Test Case Advisor Results

The Test Case Advisor uses the enter and save data entry step because that step simulates most closely a user entering data into Rave, making sure that custom functions and derivations on the form are triggered.

Test Cases created by the Test Case Advisor will be in one of these states:

Ready to run

For most Edit Checks the Test Case Advisor can suggest a valid Test Case containing two Scenarios; one which should cause the Check Actions to run and one which should not trigger the Check Actions. These Test Case should be runnable without requiring any modifications but we do recommend that all suggested Test Cases are reviewed to ensure the Edit Check is being tested optimally.

Informational notes will be added to the Scenario descriptions if the Test Case Advisor has:

  • detected that a Field has a Default Value and so does not need to have a Data Entry step

  • inserted data entry steps to make a Field visible. If the Edit Check references a Field which is not visible by default then the Test Case Advisor will look for an Edit Check with a SetDataPointVisible action for that Field and will then add the steps necessary to make the Field visible.

  • inserted data entry steps to derive a Field value. If the Edit Check references a Field which is the target of a Derivation then the Test Case Advisor will analyze the Derivation and insert data entry steps to derive an appropriate value for the Field. Sometimes the Test Case Advisor will not be able to calculate values for the derived Field which will also trigger the Edit Check in which case an issue will be added and the Test Case will be marked as invalid.

The following example shows how a data entry step has been added for the Date of Birth and Date of Informed Consent fields which are used in Derivation “DER_AGE” to calculate the age. The Age Field is used in Edit Check “AGE_RANGE”.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
 @EditCheck:AGE_RANGE
 @Derivation:DER_AGE
 @Form:DM
 Feature: AGE_RANGE

   Background:
     Given I am logged in with role "Investigator"
     And a subject exists

   Scenario: Check Actions are run
     Note: Field "AGE" on Form "DM" is derived by Derivation "DER_AGE"
     When I enter and save data:
       | DataPoint              | Value       |
       | SCREENING.SUB.ICDAT[0] | 01 JAN 2019 |
     And I enter and save data:
       | DataPoint            | Value       |
       | SCREENING.DM.BRTHDAT | 01 JAN 1979 |
     Then I should see the following query on field "SCREENING.DM.AGE"
     """
     Subject age does not meet inclusion criteria
     """

Requires attention

Some Edit Checks, mainly those using Custom Functions, cannot be created in a complete and valid state because the Test Case Advisor is unable to work out how the Edit Check should be triggered and/or what actions should be taken. For these Edit Checks the Test Case Advisor will create as much of the Test Case as it can and will list the issues which need to be addressed in a table at the start of each scenario. The list of issues should be reviewed and the appropriate steps added to the Test Case, after which the issues should be removed from the Test Case.

  • The Test Case Advisor is unable to analyse the source code of Custom Functions and Dynamic Search Lists and so test steps must be added to take the appropriate actions for them. An ‘always true’ Custom Function, eg. one that simply does ‘return true’ does not require additional work in the Test Case.

  • The Test Case Advisor might not be able to find a solution for some Edit Checks, either because there are no possible values which would trigger the Check Actions (for example, a Dictionary Field being compared against a static value which does not exist in the Dictionary) or in rare cases because the Test Case Advisor has not been able to find suitable values.

  • A Field in a Data Entry step is View or Entry Restricted, at Field or Form level, to the default role. The Test Case Advisor will insert an additional log in step before the restricted Field which will need to be edited to select an appropriate EDC Role. Note that Fields which have a default value will not appear in the data entry step because Rave will automatically enter the default value, and so they will not require this additional log in step. Similarly Fields which are Derivation targets will not appear directly in data entry steps and do not require additional log in steps if they are view or entry restricted.

  • Edit Checks using Logical Record Position will need additional work to add extra values in the appropriate logical sequence.

  • The Test Case Advisor will not be able to find suitable values for all Derivations in which case this will be noted as an issue.

  • If a Field is not visible by default and no Check Action could be found to make it visible then steps must be inserted to make it visible (probably by triggering a Custom Function in another Edit Check).

  • If a Field and/or Form is inactive then the Edit Check and Test Case must be reviewed.

  • File Upload and Signature controls are not supported by TrialGrid Test Cases. If an Edit Check references such a Field it will need to be manually tested.

  • Some Check Actions are not supported by TrialGrid Test Cases and will need to be manually tested.

See also

For more information about the limitations of Test Cases see Automated Testing Limitations

The following example shows a Test Case where the Edit Check references a derived Field and the Test Case Advisor could not find suitable values. The data entry steps are blank and should be manually completed. The Edit Check has a Custom Function action which must also be changed to the appropriate steps:

Test Case requiring attention

After review the Test Case might look like this:

Test Case after review

Labelling results

When running the Test Case Advisor for a batch of Edit Checks the resulting Test Cases can be labelled based on the Test Case Result (Ready to run, Review recommended, Requires attention) to make searching the results easier.

Object references

The Test Case Advisor will insert a cross-reference to the Edit Check, to the Form(s) referenced in Check Steps and Actions, to any Custom Functions used in the Edit Check and to any Derivations needed to populate derived Fields.

How does the Test Case Advisor work?

The Test Case Advisor looks for a set of data values for the Fields referenced in the Edit Check steps which will satisfy the logic of the Check Steps and trigger the Check Actions, and a set of data values which will not trigger the Actions and uses these values to construct two Test Case scenarios.

To find data values the Test Case Advisor looks at each Field and generates a list of potential test values based on the data type (integer, string, date), format and other properties such as ranges and dictionary values where applicable. The Test Case Advisor will not choose values which would make the Field non-conformant.

These test values are then processed by a constraint solver <https://en.wikipedia.org/wiki/Constraint_programming> which selects values which trigger (or not) the Check Actions.

In rare cases the constraint solver might not find a solution and an issue will be added to the Test Case (see below).

Why doesn’t the Test Case Advisor generate all possible combinations of test values?

The two test scenarios suggested by the Test Case Advisor can be easily copied in the Test Case Editor if more combinations of data are desired for testing. The Test Case Advisor does not attempt to generate all possible combinations because the number of scenarios becomes surprisingly large.

For example, a common type of Edit Check might compare specific Inclusion/Exclusion criteria and the Eligibility Status:

Edit Check

Here there are 5 Fields each of which can have 3 values (Yes, No and empty) leading to 3^5 = 243 different combinations. Already this is more than anyone would probably want to test but this example was taken from a study which actually has 9 Inclusion critera and 36 Exclusion criteria leading to 8,862,938,119,652,501,095,929 possibilities!