Running Test Cases

When you are ready to run a Test Cases or a set of Test Cases you create a Test Set. A Test Set is a collection of one or more copies of Test Cases which can be run in a single session.

You can create a Test Set in two ways:

  1. If you wish to run a single Test Case then the Run button in the Test Case Editor will create a Test Set containing only that Test Case.

  2. If you want to run more than one Test Case you can select Test Cases to run by clicking on the checkbox in the first column. Clicking on the checkbox in the header row at the top of the list will select all Test Cases currently in the list. You can filter the Test Case list page to select the Test Cases you want to include in your Test Set using the various search and filter options.

Note

Invalid Test Cases cannot be copied into a Test Set since they cannot be run.

Important

The Test Case is copied into a Test Set is a snapshot of the Test Case at the moment it was added to the Test Set. If you later change the source Test Case, the version in the Test Set does not change.

Selecting the Rave URL and Environment to Run on

In order to execute TrialGrid needs some information:

  1. The Rave URL. This is defaulted to the URL provided for the Project but you can enter a different URL.

  2. The Study in Rave. This is defaulted to the Project name but can be changed.

  3. The Environment for the Study (e.g. TEST, UAT or DEV).

  4. The Site Number where test subjects will be created

Note

The URL can be entered as a complete URL, like 'https://innovate.mdsol.com', or the sub-domain, e.g. 'innovate' can be entered and TrialGrid will automatically convert it to the full URL.

Note

Medidata Rave Environment names are case-sensitive, e.g. Medidata Rave will allow 'Dev' and 'DEV' as separate environments. The correct case must be used when running TrialGrid Test Cases.

Note

The Medidata Rave Site Number should be entered. In Medidata Cloud Administration this is referred to as 'Client Division Organization ID'. The Site Name, or Study Site Number ('Study Environment Site Number' in Cloud Admin) should NOT be used.

If the Study Site Number is different from the Site Number, and the study is a Rave EDC study, TrialGrid will automatically look up the Study Site Number and use that to navigate in Rave EDC.

Important

TrialGrid will allow you to run tests against the Production environment if you enter Prod as the Environment. It is advised not to provide the User Accounts provided to TrialGrid for Automated Testing with access to the Prod environment unless you really know what you are doing.

The User Accounts used for Automated Testing should be given the least permission necessary and should not, in general, have access to production environments.

Test Run settings

You can configure the following settings for the Test Set run:

Collect screenshots: screenshots will be captured in the Test Results for the selected step type, Given, When or Then. Collecting fewer screenshots will make the test run a little faster. The default setting is to collect screenshots for When and Then steps.

Check user roles: when set on the Test Case runner will check that the Rave user accounts have the specified role in Rave. When set to off the Test Case runner will report that the roles are different but will attempt to execute the tests using the given accounts. For example if the Test Case specifies an 'Investigator' role and you provide a 'Data Manager' user account then TrialGrid will run the test steps but they might fail if the Data Manager role does not have permission, for example, to enter data.

Continue on failure: When set on the Test Case runner will attempt to execute steps after a failed step in a scenario. When off (the default) the runner will stop running steps within a scenario when it encounters a failed step. This option can be useful to report the success or failure of all test steps without having to fix the first issue and then re-run.

Run concurrently: This option is only available if the Test Set run contains more than 1 Test Case. When set on, a number of Test Cases in the Test Set Run will be run concurrently, i.e. instead of running Test Cases one-by-one (when this setting is off) 2 or more Test Cases will run at the same time.

Note

It is good practice to ensure that Rave subjects are not shared between Test Cases, so that the Test Cases can run independently and do not interfere with each other's results. This is particularly important when running Test Cases concurrently.

Note

If you change these settings, they will be remembered and used when you create another Test Set run.

Running the Test Set

When you are ready to run a Test Set click the Run button. A dialog will appear requesting User Credentials in order to interact with Rave. The credentials the system requests depend on the contents of the Test Cases. For example, if a Test Case has a line:

Given I am logged in with role "Investigator"

This is a signal to the Automated Testing system that it will need credentials of a user which has an Investigator role in the Rave Study / Environment / Site Number provided.

Similarly, a line like:

Given form "PREGNANCY" exists in folder "SCREENING"

is a command to create a Form and Folder in Rave using the Subject Admin permissions. The Automated Testing system will recognize that it needs credentials of a user with Subject Admin permission in Rave.

If the Tests are to be run against a Rave EDC instance rather than a Classic Rave instance then the system will ask for iMedidata credentials for each of the EDC Roles it requires. The system will still need a Classic Rave account for these roles in order to enter data into the system using Rave Web Services (see How does Automated Testing Work?)

Once you have entered the credentials, click Run to start the Automated Testing system executing tests against the Rave instance and Environment you have specified.

Important

TrialGrid does not store the user credentials you enter in the database. Credentials are stored only for the duration of your session and are encrypted in the TrialGrid system.

iMedidata Two Factor Authentication

TrialGrid has limited support for running tests against Rave EDC using iMedidata accounts which have two factor authentication.

TrialGrid Ltd will add full support for two-factor enabled iMedidata accounts in a future TrialGrid release.

If your iMedidata account has two-factor enabled, you may see a message 'Two Factor Code required' when you create a Test Set Run, and the Test Cases will not be run.

As a temporary workaround, to run Test Cases using two-factor enabled iMedidata accounts you must first log in via the Draft Import or Export wizard (you do not need to import or export a draft). For example:

  • navigate to the list of Drafts for a Project

  • click the 'Import Draft' button

  • select 'iMedidata' (or 'iMedidata innovate' if you use innovate.imedidata.com)

  • click 'Next'

  • enter your iMedidata username and password

  • click 'Next'

  • enter the two factor code (if you are prompted to enter it)

  • click 'Next'

at this point your iMedidata two factor authentication is available and you can run Test Cases using this account.

You will have to repeat the above process each week because iMedidata two factor authentication has to be repeated every 7 days.

Monitoring Running Tests

Automated Test execution runs in the background. You can navigate away from the Test Set Run page and it will continue to run until it completes. This might take a few minutes or a few hours depending on the number of Test Cases included in the Test Set and the number of Steps (commands) in each Test Case.

As the Test Set is run, a progress meter shows overall progress and individual progress meters show the progress of each Test Case as it is run in turn.

If you want to view the progress of a Test Case you can click the State button for that Test Case and view its progress. Results and screenshots are updated in real-time as the test progresses.

Return to the Test Set run page at any time to see how the Tests are progressing.

Cancelling a Test Set Run

Sometimes you may wish to stop a Test Set run from continuing - perhaps because Test Cases have already failed and you want to correct those errors and re-run. The Cancel button requests that the background task stop its execution. It may take a few seconds for the Automated Testing Task to respond to the Cancel request.

Archiving Results of Test Set Runs

The progress page for each Test Case shows progress and screenshots (if collected) for the Test Case. When the Test Case has been fully executed (all steps run) then a PDF button becomes available at the top of the page which allows the download of this information in PDF form.

An Archive of all PDFs created for all Test Cases in the Test Set can be downloaded from the Test History page which shows a listing of all Test Sets run and their results.

Re-Running a Test Set

A Test Set can be re-run at any time but remember:

  1. The Test Set only maintains results for the last time it was run (including PDFs)

  2. The Test Set only includes Test Case contents at the time that the Test Set was run. If the Test Cases have changed since the Test Set was created you must create a new Test Set to include the changed Test Cases. The system will alert you to changed (or deleted) Test Cases which are included in a Test Set.

Server re-starts

The TrialGrid servers are automatically re-started once per day, and may be re-started more often during maintenance and new version releases. When a server which is running Test Cases is re-started, the Test Case run will start again when the server has started up. TrialGrid will attempt to continue execution of the Test Case from the last step which was previously executed. This is only possible if the step which was being run is an 'immutable' step, i.e. one which either does not change data in Medidata Rave, or which can be re-run without affecting the results. If the step being run is not immutable then the test case will be re-run from the start.

Examples of immutable test steps are:

  • Given I select the subject named "ATN1"

  • Given a subject exists

  • Given folder "AE_FOLDER" exists

  • Given form "AE" exists in folder "AE_FOLDER"

  • Then I should see the following query on "AE_FOLDER.AE.AESTDAT"

All of the 'Then' test steps are immutable.

Examples of mutable test steps are:

  • Given I create a subject named "ATN1"

  • Given I add folder "AE_FOLDER" to the subject

  • Given I add form "AE" to folder "AE_FOLDER"

  • When I enter and save data