Creating Test Sets¶
When you are ready to run a Test Cases or a set of Test Cases you create a Test Set. A Test Set is a collection of one or more copies of Test Cases which can be run in a single session.
You can create a Test Set in two ways:
If you wish to run a single Test Case then the Run button in the Test Case Editor will create a Test Set containing only that Test Case.
If you want to run more than one Test Case you can select Test Cases to run by clicking on the checkbox in the first column. Clicking on the checkbox in the header row at the top of the list will select all Test Cases currently in the list. You can filter the Test Case list page to select the Test Cases you want to include in your Test Set using the various search and filter options.
Invalid Test Cases cannot be copied into a Test Set since they cannot be run.
The Test Case is copied into a Test Set is a snapshot of the Test Case at the moment it was added to the Test Set. If you later change the source Test Case, the version in the Test Set does not change.
Selecting the Rave URL and Environment to Run on¶
In order to execute TrialGrid needs some information:
The Rave URL. This is defaulted to the URL provided for the Project but you can enter a different URL.
The Study in Rave. This is defaulted to the Project name but can be changed.
The Environment for the Study (e.g. TEST, UAT or DEV).
The Site Number where test subjects will be created
Medidata Rave Environment names are case-sensitive, e.g. Medidata Rave will allow ‘Dev’ and ‘DEV’ as separate environments. The correct case must be used when running TrialGrid Test Cases.
TrialGrid will allow you to run tests against the Production environment if you enter Prod as the Environment. It is advised not to provide the User Accounts provided to TrialGrid for Automated Testing with access to the Prod environment unless you really know what you are doing.
The User Accounts used for Automated Testing should be given the least permission necessary and should not, in general, have access to production environments.
Test Run settings¶
You can configure the following settings for the Test Set run:
Collect screenshots: when set on screenshots will be captured in the Test Results. Test runs which do not collect screenshots will run a little faster.
Check user roles: when set on the Test Case runner will check that the Rave user accounts have the specified role in Rave. When set to off the Test Case runner will report that the roles are different but will attempt to execute the tests using the given accounts. For example if the Test Case specifies an ‘Investigator’ role and you provide a ‘Data Manager’ user account then TrialGrid will run the test steps but they might fail if the Data Manager role does not have permission, for example, to enter data.
Continue on failure: When set on the Test Case runner will attempt to execute steps after a failed step in a scenario. When off (the default) the runner will stop running steps within a scenario when it encounters a failed step. This option can be useful to report the success or failure of all test steps without having to fix the first issue and then re-run.
Running the Test Set¶
When you are ready to run a Test Set click the Run button. A dialog will appear requesting User Credentials in order to interact with Rave. The credentials the system requests depend on the contents of the Test Cases. For example, if a Test Case has a line:
Given I am logged in with role "Investigator"
This is a signal to the Automated Testing system that it will need credentials of a user which has an Investigator role in the Rave Study / Environment / Site Number provided.
Similarly, a line like:
Given form "PREGNANCY" exists in folder "SCREENING"
is a command to create a Form and Folder in Rave using the Subject Admin permissions. The Automated Testing system will recognize that it needs credentials of a user with Subject Admin permission in Rave.
If the Tests are to be run against a Rave EDC instance rather than a Classic Rave instance then the system will ask for iMedidata credentials for each of the EDC Roles it requires. The system will still need a Classic Rave account for these roles in order to enter data into the system using Rave Web Services (see How does Automated Testing Work?)
Once you have entered the credentials, click Run to start the Automated Testing system executing tests against the Rave instance and Environment you have specified.
TrialGrid does not store the user credentials you enter in the database. Credentials are stored only for the duration of your session and are encrypted in the TrialGrid system.
Monitoring Running Tests¶
Automated Test execution runs in the background. You can navigate away from the Test Set Run page and it will continue to run until it completes. This might take a few minutes or a few hours depending on the number of Test Cases included in the Test Set and the number of Steps (commands) in each Test Case.
As the Test Set is run, a progress meter shows overall progress and individual progress meters show the progress of each Test Case as it is run in turn.
If you want to view the progress of a Test Case you can click the State button for that Test Case and view its progress. Results and screenshots are updated in real-time as the test progresses.
Return to the Test Set run page at any time to see how the Tests are progressing.
Cancelling a Test Set Run¶
Sometimes you may wish to stop a Test Set run from continuing - perhaps because Test Cases have already failed and you want to correct those errors and re-run. The Cancel button requests that the background task stop its execution. It may take a few seconds for the Automated Testing Task to respond to the Cancel request.
Archiving Results of Test Set Runs¶
The progress page for each Test Case shows progress and screenshots (if collected) for the Test Case. When the Test Case has been fully executed (all steps run) then a PDF button becomes available at the top of the page which allows the download of this information in PDF form.
An Archive of all PDFs created for all Test Cases in the Test Set can be downloaded from the Test History page which shows a listing of all Test Sets run and their results.
Re-Running a Test Set¶
A Test Set can be re-run at any time but remember:
The Test Set only maintains results for the last time it was run (including PDFs)
The Test Set only includes Test Case contents at the time that the Test Set was run. If the Test Cases have changed since the Test Set was created you must create a new Test Set to include the changed Test Cases. The system will alert you to changed (or deleted) Test Cases which are included in a Test Set.