Running Test Cases

When you are ready to run a Test Cases or a set of Test Cases you create a Test Set. A Test Set is a collection of one or more copies of Test Cases which can be run in a single session.

You can create a Test Set in two ways:

  1. If you wish to run a single Test Case then the Run button in the Test Case Editor will create a Test Set containing only that Test Case.

  2. If you want to run more than one Test Case you can select Test Cases to run by clicking on the checkbox in the first column. Clicking on the checkbox in the header row at the top of the list will select all Test Cases currently in the list. You can filter the Test Case list page to select the Test Cases you want to include in your Test Set using the various search and filter options.

Note

Invalid Test Cases cannot be copied into a Test Set since they cannot be run.

Important

The Test Case is copied into a Test Set is a snapshot of the Test Case at the moment it was added to the Test Set. If you later change the source Test Case, the version in the Test Set does not change.

Selecting the Rave URL and Environment to Run on

In order to execute TrialGrid needs some information:

  1. The Rave URL. This is defaulted to the URL provided for the Project but you can enter a different URL.

  2. The Study in Rave. This is defaulted to the Project name but can be changed.

  3. For Rave EDC studies, the Protocol ID in Rave.

  4. The Environment for the Study (e.g. DEV or FUNCTEST)

  5. For Rave EDC studies, the Environment Name for the Study (e.g. Development or Functional Testing).

  6. The Site Number where test subjects will be created

Note

The URL can be entered as a complete URL, like 'https://innovate.mdsol.com', or the sub-domain, e.g. 'innovate' can be entered and TrialGrid will automatically convert it to the full URL.

Note

Medidata Rave Environment names are case-sensitive, e.g. Medidata Rave will allow 'Dev' and 'DEV' as separate environments. The correct case must be used when running TrialGrid Test Cases.

Note

The Medidata Rave Site Number should be entered. In Medidata Cloud Administration this is referred to as 'Client Division Organization ID'. The Site Name, or Study Site Number ('Study Environment Site Number' in Cloud Admin) should NOT be used.

If the Study Site Number is different from the Site Number, and the study is a Rave EDC study, TrialGrid will automatically look up the Study Site Number and use that to navigate in Rave EDC.

Important

TrialGrid will allow you to run tests against the Production environment if you enter Prod as the Environment. It is advised not to provide the User Accounts provided to TrialGrid for Automated Testing with access to the Prod environment unless you really know what you are doing.

The User Accounts used for Automated Testing should be given the least permission necessary and should not, in general, have access to production environments.

Test Run settings

You can configure the following settings for the Test Set run:

Collect screenshots: screenshots will be captured in the Test Results for the selected step type, Given, When or Then. Collecting fewer screenshots will make the test run a little faster. The default setting is to collect screenshots for When and Then steps.

Check user roles: when set on the Test Case runner will check that the Rave user accounts have the specified role in Rave. When set to off the Test Case runner will report that the roles are different but will attempt to execute the tests using the given accounts. For example if the Test Case specifies an 'Investigator' role and you provide a 'Data Manager' user account then TrialGrid will run the test steps but they might fail if the Data Manager role does not have permission, for example, to enter data.

Continue on failure: When set on the Test Case runner will attempt to execute steps after a failed step in a scenario. When off (the default) the runner will stop running steps within a scenario when it encounters a failed step. This option can be useful to report the success or failure of all test steps without having to fix the first issue and then re-run.

Run concurrently: This option is only available if the Test Set run contains more than 1 Test Case. When set on, a number of Test Cases in the Test Set Run will be run concurrently, i.e. instead of running Test Cases one-by-one (when this setting is off) 2 or more Test Cases will run at the same time.

Note

It is good practice to ensure that Rave subjects are not shared between Test Cases, so that the Test Cases can run independently and do not interfere with each other's results. This is particularly important when running Test Cases concurrently.

Note

If you change these settings, they will be remembered and used when you create another Test Set run.

User Credentials

The credentials the system requests depend on the contents of the Test Cases. For example, if a Test Case has a line:

Given I am logged in with role "Investigator"

This is a signal to the Automated Testing system that it will need credentials of a user which has an Investigator role in the Rave Study / Environment / Site Number provided.

Test Cases can contain multiple login steps, so that Tests can be run using different roles in Rave:

Given I am logged in with role "Data Manager"

If a Test Case contains more than one role you will be prompted to enter credentials for each role.

Note

TrialGrid supports a maximum of 3 roles in each Test Set, i.e. there can be a maximum of 3 roles in all of the Test Cases being run at the same time in a Test Set.

If the Medidata Rave URL is configured with version 18, or earlier, of the TG_UAT draft, then there are some test steps which will require an additional Rave user account, which has Subject Admin permission:

Given form "PREGNANCY" exists in folder "SCREENING"

TrialGrid TG_UAT draft 19 and higher does not require Subject Admin user accounts.

See Medidata accounts for more information about configuring Medidata Rave accounts for TG_UAT draft 19 and higher.

If the Tests are to be run against a Rave EDC instance rather than a Classic Rave instance then the system will ask for iMedidata credentials for each of the EDC Roles it requires.

if the Medidata Rave URL is configured with version 18, or earlier, of the TG_UAT draft, Classic Rave accounts will be required in addition to Rave EDC account in order to enter data into the system using Rave Web Services (see How does Automated Testing Work?)

TrialGrid TG_UAT draft 19 and higher require one Rave Classic account when running Rave EDC tests, which must have permission to create subjects.

Important

TrialGrid does not store the user credentials you enter in the database. Credentials are stored only for the duration of your session and are encrypted in the TrialGrid system.

iMedidata Two Factor Authentication

TrialGrid TG_UAT draft 19 and higher will prompt for iMedidata Two Factor authentication when running Automated Tests whenever it is required.

TrialGrid TG_UAT draft 18 and lower has limited support for running tests against Rave EDC using iMedidata accounts which have two factor authentication. You may see a message 'Two Factor Code required' when you create a Test Set Run, and the Test Cases will not be run.

As a temporary workaround, you must first log in to iMedidata via the Draft Import or Export wizard (you do not need to import or export a draft). For example:

  • navigate to the list of Drafts for a Project

  • click the 'Import Draft' button

  • select 'iMedidata' (or 'iMedidata innovate' if you use innovate.imedidata.com)

  • click 'Next'

  • enter your iMedidata username and password

  • click 'Next'

  • enter the two factor code (if you are prompted to enter it)

  • click 'Next'

at this point your iMedidata two factor authentication is available and you can run Test Cases using this account.

You will have to repeat the above process each week because iMedidata two factor authentication has to be repeated every 7 days.

Monitoring Running Tests

Automated Test execution runs in the background. You can navigate away from the Test Set Run page and it will continue to run until it completes. This might take a few minutes or a few hours depending on the number of Test Cases included in the Test Set and the number of Steps (commands) in each Test Case.

As the Test Set is run, a progress meter shows overall progress and individual progress meters show the progress of each Test Case as it is run in turn.

If you want to view the progress of a Test Case you can click the State button for that Test Case and view its progress. Results and screenshots are updated in real-time as the test progresses.

Return to the Test Set run page at any time to see how the Tests are progressing.

Cancelling a Test Set Run

Sometimes you may wish to stop a Test Set run from continuing - perhaps because Test Cases have already failed and you want to correct those errors and re-run. The Cancel button requests that the background task stop its execution. It may take a few seconds for the Automated Testing Task to respond to the Cancel request.

Archiving Results of Test Set Runs

The progress page for each Test Case shows progress and screenshots (if collected) for the Test Case. When the Test Case has been fully executed (all steps run) then a PDF button becomes available at the top of the page which allows the download of this information in PDF form.

An Archive of all PDFs created for all Test Cases in the Test Set can be downloaded from the Test History page which shows a listing of all Test Sets run and their results.

Re-Running a Test Set

A Test Set can be re-run at any time but remember:

  1. The Test Set only maintains results for the last time it was run (including PDFs)

  2. The Test Set only includes Test Case contents at the time that the Test Set was run. If the Test Cases have changed since the Test Set was created you must create a new Test Set to include the changed Test Cases. The system will alert you to changed (or deleted) Test Cases which are included in a Test Set.

Server re-starts

The TrialGrid servers are automatically re-started once per day, and may be re-started more often during maintenance and new version releases. When a server which is running Test Cases is re-started, the Test Case run will start again when the server has started up. TrialGrid will attempt to continue execution of the Test Case from the last step which was previously executed. This is only possible if the step which was being run is an 'immutable' step, i.e. one which either does not change data in Medidata Rave, or which can be re-run without affecting the results. If the step being run is not immutable then the test case will be re-run from the start.

Examples of immutable test steps are:

  • Given I select the subject named "ATN1"

  • Given a subject exists

  • Given folder "AE_FOLDER" exists

  • Given form "AE" exists in folder "AE_FOLDER"

  • Then I should see the following query on "AE_FOLDER.AE.AESTDAT"

All of the 'Then' test steps are immutable.

Examples of mutable test steps are:

  • Given I create a subject named "ATN1"

  • Given I add folder "AE_FOLDER" to the subject

  • Given I add form "AE" to folder "AE_FOLDER"

  • When I enter and save data

Test Case Priority

A priority can be set for a Test Case. The priority is a number between 1 and 999999. When multiple Test Case are selected to be run the Test Cases will be run in order of priority, with priority 1 being the first to be run. Test Cases with the same, or no, priority will be run in alphanumeric order.