The testing tool integrations allow you to run a suite of automated tests either on you build workflow or as a secondary process -- in addition to gathering data from tests that are invoked with your build script. Most testing integrations captures the data generated by the tests, and make that information available through the UI. This allows you to perform trending over time, to see which tests are failing, track new tests, etc.
The integrations, implemented as job steps, generally enable AnthillPro to:
Run tests. This step allows AnthillPro to run a test or suite of tests. If your tests are invoked from your build scripts (e.g., using JUnit), the integration will allow AnthillPro to gather the data and store it in the AnthillPro data warehouse.
Capture data and publish a report. This step collects the test results and other information and then stores it in the AnthillPro data warehouse. This allows you to generated reports and make them available through the AnthillPro UI.
Tool-specific commands. For most testing tools, AnthillPro can also perform tasks only supported by that particular tool.
Run Agitate and generate the Dashboard Report with the Agitar integration. AnthillPro users can also add information about JUnit test results to a data warehouse.
In order to use the integration, the Agitar .arx
project-file name and the Eclipse directory path must be available. If
using the JUnit option and running JUnit tests separate from the Agitation
process, the directory with JUnit results (XML format) must also be
available.
The integration is implemented as AnthillPro job steps configured on the Job Configuration page. When using the integration, click the Create Step button (or select the Insert After/Before icon) to add steps to a job. Once the job is configured, it is then added to a workflow (under the Definition tab) as either a standalone workflow or as part of a build job.
Agitar steps:
Agitate. Runs the agitate process against code. See Configure Agitate Step.
Publish Dashboard Report. Publishes the results of the Agitar Dashboard report under the Reports tab. Optionally, this step can be used to automatically generate the report. See Configure Agitar Publish Dashboard Report Step.
This tutorial will follow a simple project configuration that uses the Agitar Agitate and Publish Dashboard Report steps as part of a build workflow. The example in this tutorial uses Subversion, but the basic configuration is similar for any repository type. Your jobs will vary, but the Agitar integration is added as a job step similar to what is described below.
Though the example goes through the manual creation of a build job, it is possible to use the Job Wizard to create a build job and then manually configure a second job to run as part of the same workflow. Optionally, the Agitar integration may also be configured to run as a standalone workflow, typically requiring only a Set Working Directory step; however, depending on the repository used, additional steps may be required.
You must have AnthillPro administrative privileges to configure the integration. See Manage Security.
A project with at least one Build Life must be active in AnthillPro.
A Life-Cycle Model must be configured with the appropriate Status and Artifact Sets. See Using Life-Cycle Models.
The Agitar
.arxproject file name and the Eclipse directory path must be available. If using the JUnit option and running JUnit tests separate from the Agitation process, the directory with JUnit results (XML format) must also be available.
The Agitate step is included after the Populate Workspace, Changelog, Stamp, Dependency, Build, Publish Changelog, and Artifact Delivery steps of the typical Build job. Though the example below runs the Agitate as part of a build workflow, it is possible to use the Agitar integration as part of a standalone workflow. If using this option, typically only a Set Working Directory step is required; however, depending on the repository used and your particular development environment, additional steps may also be required.
Go to Administration page, select the appropriate project, and click the Add Job icon.
On the New Job Configuration page, choose No (do not use the Job Wizard). Click Select.
Follow the steps for creating a build job.
Agitate. Select the Insert After icon of the step prior to the point where the Agitar step is to be included (Artifact Delivery step in this example). Go to Test > Agitar, select the Agitate step, and click Select.
Name the step (required).
Description. Provide a short description.
Project File Name (required). Give the Agitar
.arxproject-file name. Must be the same for both the Agitate and Publish Report steps.Eclipse Home (required). Provide the path where Eclipse is installed. Must be the same for both the Agitate and Publish Report steps.
Command Path. Give the path to the Agitar Command Line Interface. Must be the same for both the Agitate and Publish Report steps.
Base Checkpoint. Provide the checkpoint to base the Agitation off of.
Class Regex. Give one or more regular expressions to defining the classes to be processed. If more than one expression is used, separate them with spaces.
Class List. Provide the path to the file containing a list of classes to process. Each class name must be on a separate line (a regular expression may be used).
Agitate Timeout. Give the Agitate timeout limit in Seconds.
Log Level. Select the Agitate logging level (Fine, Info, Status, Warning, or Severe) from the drop-down menu. If none is selected, "Info" will be used by default.
Solving Level. Select the Agitate solving level (Normal, Extended, or Aggressive) from the drop-down menu.
Has Config. Check the box to only include classes with defined assertions, factory assignments, or object properties.
Merge Coverage Info. Check the box to merge the coverage information for this run with existing coverage values for the project.
Reach Time Limit. Check the box to continue exercising the methods in each class until the time limit for that class has been reached.
Show Additional Options (advanced). Select the Show Additional Options link to configure more options.
Is Active. Select No to temporarily deactivate the step without deleting it; otherwise select Yes.
Pre-Condition Script. From the drop down menu, select the condition which must be met for the step to continue. Before editing an existing script or creating a new one, see Step Pre-Condition Scripts.
Ignore Failures. Select Yes if this step should not effect the determination for step continuation or the status determination of the job.
PostProcessingScript. Select a script for determining when commands should count as fail or succeed. See Post Processing Scripts.
Timeout. Enter the time in minutes after the start of the step when AnthillPro will consider the step as timed out and abort it.
Click Save.
Include the Publish Dashboard Report step after the Agitate step (see Configure Agitate Step). Though the example below runs the Agitar steps as part of a build workflow, it is possible to use the Agitar integration as part of a standalone workflow. If using this option, typically only a Set Working Directory step is required; however, depending on the repository used and your particular development environment, additional steps may also be required.
Checking the Generate Dashboard option allows AnthillPro to invoke the Agitar Dashboard command from the command line. Once selected, additional options appear on the screen. If the Dashboard Report is already generated in an Ant build, not checking the box will result in those results being published.
Select the Insert After icon of the Agitate step. Go to Test > Agitar, select the Publish Dashboard Report step, and click Select.
Name the step.
Description. Provide a short description.
Report Name. Give the name for this report. If no name is given, the Step Name will be used as default.
Reports Output Directory. Enter the location of the Agitar Dashboard management reports.
Test Results Directory. Provide the directory with JUnit results (they must be in XML format). This option is only required if you JUnit tests are run separately from Agitation.
Generate Dashboard. Check the box to invoke the Agitar Dashboard command from the command line. If using this option, proceed to Item Two.
Do not check the box if the Agitar Dashboard report has already been generated during the Ant build and those results are to be published. If not using the Generate Dashboard option, proceed to Item Seven.
Show Additional Options (advanced). Select the Show Additional Options link to configure more options. See Show Additional Options.
If the Generate Dashboard option was not checked, Click Save.
If the Generate Dashboard option is used, provide the following information (all the fields below should be familiar to regular Agitar users):
Project. Give the name of the Agitation
.arxproject file (e.g., MyProject.arx). Must be the same for both the Agitate and Publish Report steps.Eclipse Home (required). Provide the path where Eclipse is installed. Must be the same for both the Agitate and Publish Report steps.
Command Path. Give the path to the Agitar Command Line Interface. Must be the same for both the Agitate and Publish Report steps.
Project Name. Enter the project name to place at the top of generated reports.
Checkpoint. Provide the checkpoint file (with the extension
.ddf) to compare the most recent test results with. If the value does not point to a specific checkpoint, the closest checkpoint to the date specified is used.Log Level. Select the Agitate logging level (Fine, Info, Status, Warning, or Severe) from the drop-down menu. If none is selected, "Info" will be used by default.
Server Root URL. Enter the root URL where generated reports can be accessed.
Test Assignments File. Provide the XML file that maps test classes to project classes.
Trend Window. Determine the period of time the Management Dashboard reports will cover. Specify a checkpoint for the starting point of the window (current time is the ending point).
Show Class Path Options. Select the Show Class Path Options link to configure more options.
Target Classpath. Give the search path for target (project) classes.
Test Classpath. Provide the search path for test classes.
Exclude Classes. Enter a regular expression specifying the classes to exclude from the Dashboard.
Class List. Give the file that contains a list of classes to be processed, with each class name on a separate line. Regular expressions may be used to specify classes.
Class Regex. Provide a list of one or more regular expressions, separated by spaces, defining the classes to process.
Show Directory Options. Select the Show Directory Options link to configure more options.
Agitar Path (for virtual Dashboards only). Give the search path for Agitar directories of projects for a virtual project Dashboard, separated with semicolons.
Config Path. Provide the checkpoint file (with the extension
.ddf) to compare the most recent test results with. If the value does not point to a specific checkpoint, the closest checkpoint to the date specified will be used.Coverage Results Directory (required if running JUnit separate from Agitation). Enter the directory with coverage results in
.acovor.aoutfiles, generated either by AgitarOne or by the instrument Ant task.Data Directory. Give the directory to save checkpoint files (extension
.ddf) in.Lib Path. Provide the search path for required library files of the project.
Source Path (JUnit-only projects). Provide the location of source code for generation of class coverage details. This option is ignored if an agitation project (
.arx) file is specified.
Show Target Options. Select the Show Target Options link to configure more options.
Method Risk Threshold. Determine the acceptable threshold value for method level.
Target High Risk Classes. Set the target percentage classes with risky methods for the project.
Target Classes With Testpoints. Enter the percentage of project classes that must have at least one test point.
Target Coverage. give the target coverage percentage for this project.
Target Methods With Test Points. Set the percentage of methods that must have at least one test point.
Target Test Points. Give the target number of test points for this project.
Show Args Options. Select the Show Args Options link to configure more options.
Has Config. Check the box to only includes classes that have any assertions, factory assignments, or object properties defined.
Override Has Config. Check the box to includes all classes (in an Agitation project) in the generated Management Dashboard reports, whether the classes contain configuration information or not. This option takes precedence over the Has Config option. See above.
Coverage Details. Check the box to includes all classes (in an Agitation project) in the generated Management Dashboard reports, whether the classes contain configuration information or not. This option takes precedence over the Has Config option. See above.
No Coverage Details. Check the box to overrides the inclusion of detailed coverage statistics in generated Management Dashboard reports. This option takes precedence over the Coverage Details option. See above.
Generate XML Dashboard. Check the box to create an XML file in the directory specified for the Reports Output Directory option (containing the Dashboard results). See Reports Output Directory.
Rollup. Check the box to specify the Dashboard is for a virtual project. If Rollup is specified, Agitar Path is required. See Show Directory Options.
Use Dashboard Config. Check the box to use the Dashboard configuration.
Show Additional Options (advanced). Select the Show Additional Options link to configure more options. See Show Additional Options.
Click Save.
The Job created (see Configure Agitate Step) must be executed as part of a workflow. This section will assume that an originating workflow has already been configured, and will cover the process of adding the Agitar Build job to the appropriate workflow. Complete workflow configuration is beyond the scope of this tutorial. The topics covered in detail below are specific to using the Agitar integration.
Go to Administration, select the project, and select the build workflow.
On the workflow Main page, select the Definition tab.
From the drop-down menu, choose Embedded Definition and click Select.
Left-click the Start icon and select Insert Job After.
Select the Build job created in the Configure Agitate Step section, a job pre-condition script, and click Insert Job.
Go to the Dashboard and select the workflow created in the Add Agitar Job to Workflow section.
On the workflow Main page, click the Build button for the workflow.
Once the workflow has completed, select the appropriate Build Life and click the Reports tab.
Select a link to view either the agitateReport or the agitateReport_junit.
Selecting the agitateReport link to open the Agitar Dashboard.
Selecting the agitateReport_junit link to open the Agitar/Junit report.
Click the Tests tab for metrics on all the agitateReport_junit reports. Selecting the Show Test Suites link provides detailed information regarding all the Agitar/Junit tests run on the Build Life.
To drill down on each Agitar step on the Build Life Summary page, see Trace a Build Life to Source.
In order to integrate with CppUnit, your build scripts must run
CppUnit and have it generate the test output XML files in the format
*.xml. AnthillPro takes the information from the output file
and makes it available through the UI and (optionally) via e-mail.
Identify the CppUnit output files by adding a CppUnit Report step to the build job. This allows AnthillPro to store the data and make it available to users.
Go to Administration, select the appropriate project, and click the Add Job icon.
On the New Job Configuration page, choose No (do not use the Job Wizard). Click Select.
Follow the steps for creating a build job.
Before proceeding to Item Four, ensure CppUnit is associated with the build script. See CppUnit documentation.
CppUnit Report. Select the Insert After icon of the step prior to the point where the CppUnit step is to be included. Go to Test > CppUnit, select the CppUnit Report step, and click Select.
Name the step.
Description. Provide a short description.
Report Name. Give the name for this report (default is same as step name) to appear on the Dashboard.
Source Directory. Provide the directory where the CppUnit data files will be retrieved from.
Include Patterns. Give the file name patterns that describe the files to be retrieved. Each include pattern must be entered on a separate line.
You can also use the following wild cards to tell AnthillPro what to include:
**Indicates include every directory within the base directory.*Used to include every file. So, if you use*.zip, the files matching this pattern will be included.**/*Tells AnthillPro to retrieve the entire file tree underneath the base directory.
Exclude Patterns. Provide the file name patterns identifying the files that will NOT be retrieved. This field is set in the same way as the Include Patterns field, only you are telling AnthillPro what NOT to include.
Show Additional Options (advanced). Select the Show Additional Options link to configure more options.
Is Active. Select No to temporarily deactivate the step without deleting it; otherwise select Yes.
Pre-Condition Script. From the drop down menu, select the condition which must be met for the step to continue. Before editing an existing script or creating a new one, see Step Pre-Condition Scripts.
Ignore Failures. Select Yes if this step should not effect the determination for step continuation or the status determination of the job.
PostProcessingScript. Select a script for determining when commands should count as fail or succeed. See Post Processing Scripts.
Timeout. Enter the time in minutes after the start of the step when AnthillPro will consider the step as timed out and abort it.
Click Save.
Once a build has completed, go to the new Build Life and select the Reports and Tests tabs for information on the tests run.
Since AnthillPro is aware of CppUnit, the results can be included in build notifications, just like any other piece of information. For example, you may want to automatically take action depending on CppUnit results. Some common examples are to e-mail the team if a failure occurs; e-mail the QA team leader if there are no errors; or e-mail someone if the tests-per-second metric goes below a target amount.
To edit a notification template, go to the System page and choose the Notification Templates link from the Notification menu. See Managing Notifications.
Lookup the CppUnit report(s) and put them in the Velocity context.
Once the CppUnitReports are available to the Velocity template, you can do whatever you like with them.
Included in an HTML-friendly e-mail template, this will produce a nice little report.
In order to integrate with JUnit (or TestNG), your build scripts must run JUnit and
have it generate the test output XML files in the format of
TEST-*.xml.
In Ant, the example project is configured to generate these files during the Ant JUnit task:
<junit printsummary="on" haltonfailure="false" fork="true">
<classpath refid="tests.classpath"/>
<formatter type="xml"/>
<batchtest todir="">
<fileset dir="" includes="**/*Test*.class"/>
</batchtest>
</junit>The Ant JUnit report task will delete the report files as part of its transformation. If you use that task, make a copy of the
TEST-*.xmlfile for AnthillPro to use.
AnthillPro reads the output XML files to understand what happened during the JUnit tests. To tell AnthillPro about those files, you need to: (a.) identify where the files are generated in the project; and (b.) add a JUnit Report step on the project build-job configuration.
Go to Administration, select the appropriate project, and click the Add Job icon.
On the New Job Configuration page, choose No (do not use the Job Wizard). Click Select.
Follow the steps for creating a build job.
Before proceeding to Item Four, ensure JUnit is associated with the build script. See JUnit documentation.
JUnit Report. Select the Insert After icon of the step prior to the point where the JUnit step is to be included. Go to Test > JUnit, select the JUnit Report step, and click Select.
Name the step.
Description. Provide a short description.
Report Name. Give the name for this report (default is same as step name).
Source Directory. Give the directory where the JUnit data files will be retrieved from. This is relative to the working directory. For example, give
..\..\directoryname\.Include Patterns. Give the file name patterns that describe the files to be retrieved. Each include pattern must be entered on a separate line.
You can also use the following wild cards to tell AnthillPro what to include:
**Indicates include every directory within the base directory.*Used to include every file. So, if you use*.zip, the files matching this pattern will be included.**/*Tells AnthillPro to retrieve the entire file tree underneath the base directory.
Exclude Patterns. Provide the file name patterns identifying the files that will NOT be retrieved. This field is set in the same way as the Include Patterns field, only you are telling AnthillPro what NOT to include.
Show Additional Options (advanced). Select the Show Additional Options link to configure more options.
Is Active. Select No to temporarily deactivate the step without deleting it; otherwise select Yes.
Pre-Condition Script. From the drop down menu, select the condition which must be met for the step to continue. Before editing an existing script or creating a new one, see Step Pre-Condition Scripts.
Ignore Failures. Select Yes if this step should not effect the determination for step continuation or the status determination of the job.
PostProcessingScript. Select a script for determining when commands should count as fail or succeed. See Post Processing Scripts.
Timeout. Enter the time in minutes after the start of the step when AnthillPro will consider the step as timed out and abort it.
Click Save.
Once the new settings are saved, running the build reveals there is a new report stored in the Tests and Reports tabs of the Build Life (Build Life 127 in the example below). This is a basic report checking the JUnit results. More importantly, the JUnit results are now exposed in a meaningful way.
Since AnthillPro is now aware of the JUnit results, they can be included in build notifications, just like any other piece of information. To edit a notification template, go to the System page and choose the Notification Templates link from the Notification menu. See Managing Notifications.
Lookup the JUnit report(s) and put them in the Velocity context by adding the following to the context script:
WorkflowCase workflow = (WorkflowCase)context.get("workflow"); junitReportArray = JUnitReportHelper.getJUnitReportArray(workflow); context.put("junitReportArray", junitReportArray);Once the JUnitReports are available to the Velocity template, you can do whatever you like with them. As an example, enter a template that prints out some summary information, as well as a list of every JUnit test that did not pass.
#foreach($junitReport in $junitReportArray) <h1>JUnit Test Results</h1> <H2>Test Summary:</H2> <div class="data-table-container"> <table class="data-table" cellpadding="4" cellspacing="1" width="100%"> <tr class="data-table-head"> <th>Total:</th> <th>Pass:</th> <th>Fail:</th> <th>Errors:</th> <th>Success Rate:</th> <th>Time:</th> </tr> <tr bgcolor="#ffffff"> <td align="center" style="font-weight: bold;font-size: 14px;"> $junitReport.testSummary.tests </td> <td align="center" style="color: green;font-weight: bold;font-size: 14px;"> $junitReport.testSummary.passingTests </td> <td align="center" style="color: red;font-weight: bold;font-size: 14px;"> $junitReport.testSummary.failures </td> <td align="center" style="color: red;font-weight: bold;font-size: 14px;"> $junitReport.testSummary.errors</B> </td> <td align="center" style="font-weight: bold;font-size: 14px;"> $fn.formatNumber($junitReport.testSummary.successPercentage , '##.#%') </td> <td align="center" style="font-weight: bold;font-size: 14px;"> $fn.formatNumber($fn.duration($junitReport.testSummary.time), '##0.0’) secs </td> </tr> </table> </div> #foreach( $suite in $junitReport.suiteResultArray) #if( $suite.passingTestCount < $suite.tests ) <div class="data-table-container"> <h3> Failed tests in $suite.name </h3> <table class="data-table" cellpadding="4" cellspacing="1" width="100%"> #foreach( $case in $suite.testCaseResults ) #if( $case.errorCount > 0 ) #foreach( $error in $case.errorArray ) <tr bgcolor="#ffffff"> <td valign="top" style="color: red;font-weight: bold;" width="275"> $case.name<br> (error)</td> <td valign="top" style="color: red;"><B>$error.type</b><br> $error.content </td> </tr> #end #end #if( $case.failureCount > 0 ) #foreach( $error in $case.failureArray ) <tr bgcolor="#ffffff"> <td valign="top" style="color: red;font-weight: bold;" width="275"> $case.name<br> (failure)</td> <td valign="top" style="color: red;"><B>$error.type</b><br> <b>$error.message</b><br/> $error.content </td> </tr> #end #end #end </table> </div> #end #end #endIncluded in an HTML-friendly e-mail template, this will produce a nice little report. The example project has one failing test and one passing test. That produces the following in an e-mail.
You may want to automatically take action depending on JUnit results. Some common examples are to e-mail the team if a failure occurs; e-mail the QA team leader if there are no errors; or e-mail someone if the tests-per-second metric goes below a target amount.
Below is a notification event-selector script to notify when JUnit tests fail:
import com.urbancode.anthill3.domain.workflow.*; result = false; if (event instanceof WorkflowEvent && event.getCase().isComplete() && JUnitReportHelper.getJUnitReportArray(event.getCase()) !=null) { workflow = event.getCase(); junitReports = JUnitReportHelper.getJUnitReportArray(workflow); for (int i = 0; i < junitReports.length; i++) { summary = junitReports[i].getTestSummary(); if(summary.getPassingTests() != summary.getTests()) { result = true; } } } return result;
Script that e-mails when some tests take too long:
import com.urbancode.anthill3.domain.workflow.*; double testPerSecondNeeded = 100; result = false; if (event instanceof WorkflowEvent && event.getCase().isComplete() && JUnitReportHelper.getJUnitReportArray(event.getCase()) !=null) { workflow = event.getCase(); junitReports = JUnitReportHelper.getJUnitReportArray(workflow); for (int i = 0; i < junitReports.length; i++) { summary = junitReports[i].getTestSummary(); if(summary.getTestsPerSecond() < testPerSecondNeeded){ result = true; } } } return result;
The JUnit report generated by some build scripts can be valuable as well. Since its creation tends to delete the required XML files, copy those files to another location in the script and generate that report. Then add an AnthillPro publisher to publish the resulting JUnit HTML report for review by the team.
The number of tests that should run per second may differ by project. A different case selector could be made for every project, but it would make more sense to add to each project a project property with the number of tests that should run each second, and reference that property in the case selection script.
Another option would be to create a report tracking the number of passing and failed tests over time.
The MSTest integration, written as a Plugin, is added to your AnthillPro job as a Report Publisher job step. Once added to your build job, the MSTest integration enables AnthillPro to collect the unit-test report and make the findings available on the Dashboard. See View Reports (MSTest).
| The integration is written as an AnthillPro Plugin, included in the normal distribution. For older AnthillPro 3.7 versions, you will need to download the integration from Supportal and then upload it to the server. Once uploaded, ensure the Plugin is active. |
You should already have MSTest running (and producing a report on) your unit tests.
You will need to know the location of the reports.
You will need administrative permissions for the project.
The step should be included in your build job after MSTest has been run and the report generated. Typically, this is will be after the build step.
Go to Administration, select the appropriate project, and click the Add Job icon.
On the New Job Configuration page, choose No (do not use the Job Wizard). Click Select.
Follow the steps for creating a build job.
Select the Insert After icon of the step that will run before the MSTest Report Publisher step.
Go to Test > MSTest, select the MSTest Report Publisher step, and click Select.
Name the step.
Description. Provide a short description.
Working Directory Offset (optional). Give the working directory to use when executing this command. This is relative to current working directory. To use the current directory, leave this field blank.
Report Name. Give the name for this report to appear on the AnthillPro Dashboard.
Base Directory. Provide the directory for resolving MSTest XML files. Unless absolute, this is relative to the job working directory.
Include Patterns. Give the file name patterns that describe the files to be retrieved. Each include pattern must be entered on a separate line.
You can also use the following wild cards to tell AnthillPro what to include:
**Indicates include every directory within the base directory.*Used to include every file. So, if you use*.zip, the files matching this pattern will be included.**/*Tells AnthillPro to retrieve the entire file tree underneath the base directory.
Exclude Patterns. Provide the file name patterns identifying the files that will NOT be retrieved. This field is set in the same way as the Include Patterns field, only you are telling AnthillPro what NOT to include.
Show Additional Options (advanced). Select the Show Additional Options link to configure more options.
Is Active. Select No to temporarily deactivate the step without deleting it; otherwise select Yes.
Pre-Condition Script. From the drop down menu, select the condition which must be met for the step to continue. Before editing an existing script or creating a new one, see Step Pre-Condition Scripts.
Ignore Failures. Select Yes if this step should not effect the determination for step continuation or the status determination of the job.
PostProcessingScript. Select a script for determining when commands should count as fail or succeed. See Post Processing Scripts.
Timeout. Enter the time in minutes after the start of the step when AnthillPro will consider the step as timed out and abort it.
Show Environment Variables (optional; advanced). Give any optional environment variables in "name=value" format.
Environment variables values may contain references to existing values in the following format:
name=${env/<NAME>};value. If the value of the <NAME> variable is "value2" in the current environment, then the above example will be expanded to:name=value2;value.Using this technique, it is possible add an entry to PATH in the following manner:
PATH=my/path/entry;0. Case is significant even on Windows systems.
Click Save.
Once a build has completed, go to the new Build Life and select the Tests tab for information on the unit tests run. There, you can view the results and perform trending to see how your unit tests are doing over time.
Since AnthillPro is aware of MSTest, the results can be included in build notifications, just like any other piece of information. For example, you may want to automatically take action depending on results. Some common examples are to e-mail the team if a failure occurs; e-mail the QA team leader if there are no errors; or e-mail someone if the tests-per-second metric goes below a target amount.
To edit a notification template, go to the System page and choose the Notification Templates link from the Notification menu. See Managing Notifications.
Lookup the MSTest report(s) and put them in the Velocity context.
Once the MSTest Reports are available to the Velocity template, you can do whatever you like with them.
Included in an HTML-friendly e-mail template, this will produce a nice little report.
In order to integrate with NUnit, your build scripts must run NUnit
and locate the test output XML files (typically named
TestResults-*.xml). AnthillPro takes the information from the
output file and makes it available through the UI and (optionally) via
e-mail.
Identify the NUnit output files by adding a NUnit Report step to the build job. This allows AnthillPro to store the data and make it available to users.
Go to Administration, select the appropriate project, and click the Add Job icon.
On the New Job Configuration page, choose No (do not use the Job Wizard). Click Select.
Follow the steps for creating a build job.
Before proceeding to Item Four, ensure NUnit is associated with the build script. See NUnit documentation.
NUnit Report. Select the Insert After icon of the step prior to the point where the NUnit step is to be included. Go to Test > NUnit, select the NUnit Report step, and click Select.
Name the step.
Description. Provide a short description.
Report Name. Give the name for this report (default is same as step name) to appear on the Dashboard.
Source Directory. Provide the directory where the NUnit data files will be retrieved from.
Include Patterns. Give the file name patterns that describe the files to be retrieved. Each include pattern must be entered on a separate line.
You can also use the following wild cards to tell AnthillPro what to include:
**Indicates include every directory within the base directory.*Used to include every file. So, if you use*.zip, the files matching this pattern will be included.**/*Tells AnthillPro to retrieve the entire file tree underneath the base directory.
Exclude Patterns. Provide the file name patterns identifying the files that will NOT be retrieved. This field is set in the same way as the Include Patterns field, only you are telling AnthillPro what NOT to include.
Show Additional Options (advanced). Select the Show Additional Options link to configure more options.
Is Active. Select No to temporarily deactivate the step without deleting it; otherwise select Yes.
Pre-Condition Script. From the drop down menu, select the condition which must be met for the step to continue. Before editing an existing script or creating a new one, see Step Pre-Condition Scripts.
Ignore Failures. Select Yes if this step should not effect the determination for step continuation or the status determination of the job.
PostProcessingScript. Select a script for determining when commands should count as fail or succeed. See Post Processing Scripts.
Timeout. Enter the time in minutes after the start of the step when AnthillPro will consider the step as timed out and abort it.
Click Save.
Once a build has completed, go to the new Build Life and select the Reports and Tests tabs for information on the tests run.
Since AnthillPro is aware of NUnit, the results can be included in build notifications, just like any other piece of information. For example, you may want to automatically take action depending on NUnit results. Some common examples are to e-mail the team if a failure occurs; e-mail the QA team leader if there are no errors; or e-mail someone if the tests-per-second metric goes below a target amount.
To edit a notification template, go to the System page and choose the Notification Templates link from the Notification menu. See Managing Notifications.
Lookup the NUnit report(s) and put them in the Velocity context.
Once the NUnit Reports are available to the Velocity template, you can do whatever you like with them.
Included in an HTML-friendly e-mail template, this will produce a nice little report.
The Quality Center integration allows Windows users (through the COM interface) of AnthillPro to run test sets and then publish a report to the AnthillPro UI. The report is published to the Reports tab of the Build Life, and provides metrics, such as performance trends, on the tests that have been run.
| The integration will only work if the agent(s) running the Quality Center steps uses the 32-bit JVM. If the agent(s) use a 64-bit JVM, the steps will fail. However, the AnthillPro server, which does not run the steps, can run on the 64-bit JVM. |
The Quality Center integration is implemented as AnthillPro job steps configured on the Job Configuration page. When using the integration, click the Create Step button (or select the Insert After/Before icon) to add steps to a job. Once the job is configured, it is then added to the workflow under the Definition tab.
Quality Center Testing Steps:
Run Test Set. Runs a Test Set (QTP, LoadRunner, WinRunner) using Mercury Quality Center either directly on the server or on the Quality Center remote agent(s). Creating a separate testing workflow with the Run Test Sets and Test Set Report is advisable.
Publish Test Set Report. Publishes a Quality Center TestSet Report for (QTP, LoadRunner, WinRunner). Creating a separate testing workflow with the run test sets and test set report is advisable.
| The Quality Center integration also allows you to track issues with Quality Center and run QTP tests. If you want to use the QTP integration, you must configure Quality Center integration on the System page. See Quality Center Issue Tracking and QuickTest Pro. |
You must have AnthillPro administrative privileges to configure the integration. See Manage Security.
The Mercury Quality Center URL must be available.
AnthillPro must be able to access Quality Center as a user with permissions to run tests.
An AnthillPro agent must be installed on a Windows machine (XP, 2003, Vista, Windows 7) with the Quality Center API installed. For example, the Quality Center server. This may require configuring a Fixed Agent Filter to ensure the job runs on the appropriate machine. Alternately, the agent with the Quality Center API can be added to the Build Farm. This requires the use of a scripted filter to look for the presence of a variable/property (e.g., qc.agent=true) which must be added to the agent. If this option is chosen, it is advisable to include a filter on every build job that excludes the Quality Center agent.
The integration will only work if the agent(s) running the Quality Center steps uses the 32-bit JVM. If the agent(s) use a 64-bit JVM, the steps will fail. However, the AnthillPro server, which does not run the steps, can run on the 64-bit JVM. See Configure and Edit Agent Filters.
The AnthillPro agent that will perform the QualityCenter steps must be installed on the same machine as the QualityCenter client. You can download the client from the QualityCenter server, typically located here:
http://<qc_server>:8080/qcbin/ClientSide_index.html. Follow the instructions on that page to install.
Go to System > Quality Center under the Integration menu.
On the Mercury Quality Center Integration page, click Edit.
Configure the integration:
Enter the Quality Center server URL.
Issue URL. You can have AnthillPro automatically generate a link to all of the issues it associates with a Build Life if you give the Issue URL here. If you are also using the Quality Center Issue Tracking integration, you will need to complete this field.
Once you give the URL pattern, the issues that appear on the Issues Tab of a Build Life will be linked to the issue in your issue tracker tool for reviewing the issue, adding additional comments, making edits, etc. For example, provide a URL template such as
http://bugs.company.com/browse/${issueId}. The value${issueId}will be replaced in the template with the issue id of the associated issue. This field provides a template which is used throughout AnthillPro to generate links from issues directly to an issue description page within your issue tracker.User Name. Enter the user name to be used to connect to the Quality Center server. Make sure that Quality Center user AnthillPro will use to connect to the server has the appropriate permissions.
Password. Enter the password to be used to connect to the Quality Center server.
Password Script (optional). To use a script or property lookups for the password, leave the Password field blank and enter it here. See Scripting.
Click Set and click Done.
Configure the job to automatically run a Quality Center test set and publish the report by setting up the job to run on the agent with the QC COM API installed. While each job is different, every job should run a get changelog step; run steps to interact with the changelog and Quality Center; and generate reports and make comments.
The Run Test Set and Publish Report steps are included after the Populate Workspace, Changelog, Stamp, Dependency, Build, Publish Changelog, and Artifact Delivery steps of the typical job.
Go to Administration, select the appropriate project, and follow the steps for creating a build job.
Quality Center - Run Test Set. Select the Insert After icon of the step prior to where the step is to be included. Go to Tests > Mercury, select Quality Center - Run Test Set, and click Select.
If you are publishing the Test Set Report, make sure Test Set, Folder, Domain, and Project fields correspond to the Publish Test Set Report step configured below. See Quality Center - Publish Test Set Report. Name. Provide a name that will be used by AnthillPro if the default name is not used.
Description (optional). Give a short description.
Test Set. Give the name of the Quality Center Test Set AnthillPro will execute.
Folder. Give the full folder path to the test set above.
Remote Agent. If you want this step to run on a remote agent, give the location here. Otherwise, AnthillPro will use the local agent.
Domain Name. Give the name of the Quality Center Domain where your projects are located.
Project Name. Give the name of the Quality Center Project where your issues are located.
Show Additional Options (advanced). Select the Show Additional Options link to configure more options.
Is Active. Select No to temporarily deactivate the step without deleting it; otherwise select Yes.
Pre-Condition Script. From the drop down menu, select the condition which must be met for the step to continue. Before editing an existing script or creating a new one, see Step Pre-Condition Scripts.
Ignore Failures. Select Yes if this step should not effect the determination for step continuation or the status determination of the job.
PostProcessingScript. Select a script for determining when commands should count as fail or succeed. See Post Processing Scripts.
Timeout Enter the time in minutes after the start of the step when AnthillPro will consider the step as timed out and abort it.
Click Save.
Quality Center - Publish Test Set Report. Select the Insert After icon of the Quality Center - Run Test Set step. Go to Test > Mercury, select the Quality Center - Publish Test Set Report step, and click Select. This step retrieves the Report.xml files generated by the tests. Once they are retrieved from the build, AnthillPro will be able to make them available to the Build Life Tests tab.
When configuring this step, make sure Test Set, Folder, Domain, and Project corresponds to the Run Test Set step configured above. See Quality Center - Run Test Set. Name the step.
Description (optional). Provide a brief description.
Report Name. Provide a name for the report (if left blank, it will be the step name).
Test Set. Give the name of the Quality Center Test Set AnthillPro executed in the Run Test Set step configured above.
Folder. Give the full folder path to the test set configured in the Run Test Set step.
Domain Name. Give the name of the Quality Center Domain identified in the Run Test Set step above.
Project Name. Give the name of the Quality Center Project you configured in the Run Test Set step above.
Show Additional Options (advanced). See Show Additional Options.
Click Save.
Complete workflow configuration is beyond the scope of this entry. The topics covered in detail below are specific to using the Quality Center integration.
Go Administration and select the appropriate workflow. You can add the job to either your build workflow or as part of a secondary (i.e., testing) workflow.
Go to workflow Main > Definition tab.
From the drop-down menu, choose Embedded Definition and click Select.
Left-click the Start icon and select Insert Job After.
Select the job you just created, a job pre-condition script, and click Insert Job.
Select the Properties tab and add a workflow property. This will ensure that the job runs on the appropriate agent when using a scripted agent filter. Make sure that the property you set here is also configured on the agent.
Name. Give the name of the Property (the property <name> will be accessed as ${property:<name>}).
Description. Provide a description for this Property shown when prompting users for value.
Default Value. Input the value for this property.
User May Override. Check if users are able to specify a new value when running this workflow.
Label. Provide a label for this Property shown when prompting users for value (leave blank to use the Name as the Label).
Is Required. Check if a non-empty value for this property is required to run workflow.
Allowed Values. Give the values users are allowed to select for this property (blank for no restriction of value). Separate each value by entering it on its own line.
Click Save.
Click Save.
Go to Dashboard and select the appropriate workflow.
On the workflow Main page, click the Build button.
Once the workflow is complete, select the Tests tab to view the report.
Following is a list of the function calls used in the Quality Center integration:
| TDApiOle80.TDConnection | TestSet.StartExecution | TestSet.TSTestFactory | Step.Field |
| TDConnection.BugFactory | TSScheduler.RunAllLocally | TSTestFactory.NewList | BugFactory.Filter |
| TDConnection.InitConnectionEx | TSScheduler.TdHostName | TSTest.Name | BugFactory.Item |
| TDConnection.Login | TSScheduler.Run | TSTest.Type | BugFactory.AddItem |
| TDConnection.Connect | TSScheduler.ExecutionStatus | TSTest.Status | Filter.Filter |
| TDConnection.Disconnect | ExecutionStatus.RefreshExecStatusInfo | TSTest.HostName | Filter.NewList |
| TDConnection.Logout | ExecutionStatus.Finished | TSTest.TestId | Bug.ID |
| TDConnection.ReleaseConnection | ExecutionStatus.EventsList | TSTest.TestName | Bug.Summary |
| TDConnection.TestSetTreeManager | ExecEventInfo.EventType | TSTest.LastRun | Bug.Field |
| TestSetTreeManager.NodeByPath | ExecEventInfo.EventDate | Run.Name | Bug.AssignedTo |
| TestSetFolder.FindTestSets | ExecEventInfo.EventTime | Run.Status | Bug.DetectedBy |
| TestSet.Id | TestExecStatus.TestId | Run.Field | Bug.Priority |
| TestSet.Item | TestExecStatus.TestInstance | Run.StepFactory | Bug.Project |
| TestSet.Name | TestExecStatus.TsTestId | StepFactory.NewList | Bug.Status |
| TestSet.TestSetFolder | TestExecStatus.Message | Step.Name | Bug.Post |
| TestSet.Status | TestExecStatus.Status | Step.Status |
The Quality Center integration allows Windows users (through the COM interface) of AnthillPro to run test sets and then publish a report to the AnthillPro UI. The report is published to the Reports tab of the Build Life, and provides metrics, such as performance trends, on the tests that have been run.
| The integration is written as an AnthillPro Plugin, and expands upon the existing Quality Center (Testing) integration. The integration is included in the normal distribution. For older AnthillPro 3.7 versions, you will need to download the integration from Supportal and then upload it to the server. Once uploaded, ensure the Plugin is active. |
The Quality Center integration is implemented as AnthillPro job steps configured on the Job Configuration page. When using the integration, click the Create Step button (or select the Insert After/Before icon) to add steps to a job.
| The integration will only work if the agent(s) running the Quality Center steps uses the 32-bit JVM. If the agent(s) use a 64-bit JVM, the steps will fail. However, the AnthillPro server, which does not run the steps, can run on the 64-bit JVM. |
Once the job is configured, it is then added to the workflow under the Definition tab.
Quality Center Testing Steps:
Run Quality Center Test Set. Runs a Test Set (QTP, LoadRunner, WinRunner) using Mercury Quality Center either directly on the server or on the Quality Center remote agent(s). Creating a separate testing workflow with the Run Test Sets and Test Set Report is advisable.
Publish Quality Center Test Report. Publishes a Quality Center TestSet Report for (QTP, LoadRunner, WinRunner). Creating a separate testing workflow with the run test sets and test set report is advisable.
The AnthillPro agent that will perform the QualityCenter steps must be installed on the same machine as the QualityCenter client. You can download the client from the QualityCenter server, typically located here:
http://<qc_server>:8080/qcbin/ClientSide_index.html. Follow the instructions on that page to install.
| The Quality Center integration also allows you to track issues with Quality Center and run QTP tests. If you want to use the QTP integration, you must configure Quality Center integration on the System page. See Quality Center Plugin (Issue Tracking) and QuickTest Pro. |
You must have AnthillPro administrative privileges to configure the integration. See Manage Security.
The Mercury Quality Center URL must be available.
AnthillPro must be able to access Quality Center as a user with permissions to run tests.
An AnthillPro agent must be installed on a Windows machine (XP, 2003, Vista, Windows 7) with the Quality Center API installed. For example, the Quality Center server. This may require configuring a Fixed Agent Filter to ensure the job runs on the appropriate machine. Alternately, the agent with the Quality Center API can be added to the Build Farm. This requires the use of a scripted filter to look for the presence of a variable/property (e.g., qc.agent=true) which must be added to the agent. If this option is chosen, it is advisable to include a filter on every build job that excludes the Quality Center agent. See Configure and Edit Agent Filters.
The information given here will be used by your AnthillPro projects. If you are using both the Quality Center (Testing) and Quality Center (Issue Tracking) Plugins, you need only configure the integration once (assuming you have only one Quality Center server) -- both integrations use the same System configuration.
| If you are configuring integrations with multiple Quality Center servers, create a new integration for each one. |
Go to System > Quality Center Plugin under the Integration menu.
On the Quality Center Plugin page, click Create New.
Configure the integration:
Name. Give a unique name for this integration. The name given here will be used throughout the AnthillPro system -- specifically during job creation. If you are configuring integrations with multiple Quality Center servers, ensure that each name is unique.
Server URL. Enter the base URL to the TeamForge installation base URL:
http://qualitycenter.company.com.User Name. Enter the user name to be used to connect to the Quality Center server. Make sure that Quality Center user AnthillPro will use to connect to the server has the appropriate permissions.
Password. Enter the password to be used to connect to the Quality Center server. If using a password script below, leave this filed blank.
Confirm password.
Password Script. To use a script or property lookups for the password, leave the Password field blank and enter it here. See Scripting.
Click Set and click Done.
Configure the job to automatically run a Quality Center test set and publish the report by setting up the job to run on the agent with the QC COM API installed. While each job is different, every job should run a get changelog step; run steps to interact with the changelog and Quality Center; and generate reports and make comments.
The Run Test Set and Publish Report steps are included after the Populate Workspace, Changelog, Stamp, Dependency, Build, Publish Changelog, and Artifact Delivery steps of the typical job.
Go to Administration, select the appropriate project, and follow the steps for creating a build job.
Run Quality Center Test Set. Select the Insert After icon of the step prior to where the step is to be included. Go to Test > Quality Center Plugin, select Run Quality Center Test Set, and click Select.
If you are publishing the Test Set Report, make sure Test Set, Folder, Domain, and Project fields correspond to the Publish Test Set Report step configured below. Name. Provide a name that will be used by AnthillPro if the default name is not used.
Description (optional). Give a short description.
Working Directory Offset (optional). Give the working directory to use when executing this command. This is relative to current working directory. To use the current directory, leave this field blank.
QC Server. Select the correct integration from the drop-down menu (this is the integration set up in the Configure Quality Center Plugin (Testing) section). If you configured multiple integrations on the AnthillPro System page, make sure you select the correct one. Note that it is possible for a single job -- but not a step -- to use different AnthillPro/Quality-Center-server configurations.
Domain Name. Give the name of the Quality Center Domain where your projects are located.
Project Name. Give the name of the Quality Center Project where your issues are located.
Folder. Give the full folder path to the test set above.
Test Set. Give the name of the Quality Center Test Set AnthillPro will execute.
Remote Host. If you want this step to run on a remote host (i.e., a remote agent), give the location here. Otherwise, AnthillPro will use the agent on the local host.
Show Environment Variables (optional; advanced). Give any optional environment variables in "name=value" format.
Environment variables values may contain references to existing values in the following format:
name=${env/<NAME>};value. If the value of the <NAME> variable is "value2" in the current environment, then the above example will be expanded to:name=value2;value.Using this technique, it is possible add an entry to PATH in the following manner:
PATH=my/path/entry;0. Case is significant even on Windows systems.Show Additional Options (advanced). Select the Show Additional Options link to configure more options.
Is Active. Select No to temporarily deactivate the step without deleting it; otherwise select Yes.
Pre-Condition Script. From the drop down menu, select the condition which must be met for the step to continue. Before editing an existing script or creating a new one, see Step Pre-Condition Scripts.
Ignore Failures. Select Yes if this step should not effect the determination for step continuation or the status determination of the job.
PostProcessingScript. Select a script for determining when commands should count as fail or succeed. See Post Processing Scripts.
Timeout Enter the time in minutes after the start of the step when AnthillPro will consider the step as timed out and abort it.
Click Save.
Publish Quality Center Test Report. Select the Insert After icon of the previous step. Go to Test > Quality Center Plugin, select the Publish Quality Center Test Report step, and click Select. This step retrieves the Report.xml files generated by the tests. Once they are retrieved from the build, AnthillPro will be able to make them available to the Build Life Tests tab.
When configuring this step, make sure Test Set, Folder, Domain, and Project corresponds to the Run Test Set step configured above. Name the step.
Description (optional). Provide a brief description.
Working Directory Offset (optional). Give the working directory to use when executing this command. This is relative to current working directory. To use the current directory, leave this field blank.
QC Server. Select the correct integration from the drop-down menu (this is the integration set up in the Configure Quality Center Plugin (Testing) section). If you configured multiple integrations on the AnthillPro System page, make sure you select the correct one. Note that it is possible for a single job -- but not a step -- to use different AnthillPro/Quality-Center-server configurations.
Domain Name. Give the name of the Quality Center Domain identified in the Run Test Set step above.
Project Name. Give the name of the Quality Center Project you configured in the Run Test Set step above.
Folder. Give the full folder path to the test set configured in the Run Test Set step.
Test Set. Give the name of the Quality Center Test Set AnthillPro executed in the Run Test Set step configured above.
Show Environment Variables (optional; advanced). See Show Environment Variables above.
Show Additional Options (advanced). See Show Additional Options.
Click Save.
Complete workflow configuration is beyond the scope of this entry. The topics covered in detail below are specific to using the Quality Center integration.
Go Administration and select the appropriate workflow. You can add the job to either your build workflow or as part of a secondary (i.e., testing) workflow.
Go to workflow Main > Definition tab.
From the drop-down menu, choose Embedded Definition and click Select.
Left-click the Start icon and select Insert Job After.
Select the job you just created, a job pre-condition script, and click Insert Job.
Select the Properties tab and add a workflow property. This will ensure that the job runs on the appropriate agent when using a scripted agent filter. Make sure that the property you set here is also configured on the agent.
Name. Give the name of the Property (the property <name> will be accessed as ${property:<name>}).
Description. Provide a description for this Property shown when prompting users for value.
Default Value. Input the value for this property.
User May Override. Check if users are able to specify a new value when running this workflow.
Label. Provide a label for this Property shown when prompting users for value (leave blank to use the Name as the Label).
Is Required. Check if a non-empty value for this property is required to run workflow.
Allowed Values. Give the values users are allowed to select for this property (blank for no restriction of value). Separate each value by entering it on its own line.
Click Save.
Click Save.
Go to Dashboard and select the appropriate workflow.
On the workflow Main page, click the Build button.
Once the workflow is complete, select the Tests tab to view the report.
Following is a list of the function calls used in the Quality Center integration:
| TDApiOle80.TDConnection | TestSet.StartExecution | TestSet.TSTestFactory | Step.Field |
| TDConnection.BugFactory | TSScheduler.RunAllLocally | TSTestFactory.NewList | BugFactory.Filter |
| TDConnection.InitConnectionEx | TSScheduler.TdHostName | TSTest.Name | BugFactory.Item |
| TDConnection.Login | TSScheduler.Run | TSTest.Type | BugFactory.AddItem |
| TDConnection.Connect | TSScheduler.ExecutionStatus | TSTest.Status | Filter.Filter |
| TDConnection.Disconnect | ExecutionStatus.RefreshExecStatusInfo | TSTest.HostName | Filter.NewList |
| TDConnection.Logout | ExecutionStatus.Finished | TSTest.TestId | Bug.ID |
| TDConnection.ReleaseConnection | ExecutionStatus.EventsList | TSTest.TestName | Bug.Summary |
| TDConnection.TestSetTreeManager | ExecEventInfo.EventType | TSTest.LastRun | Bug.Field |
| TestSetTreeManager.NodeByPath | ExecEventInfo.EventDate | Run.Name | Bug.AssignedTo |
| TestSetFolder.FindTestSets | ExecEventInfo.EventTime | Run.Status | Bug.DetectedBy |
| TestSet.Id | TestExecStatus.TestId | Run.Field | Bug.Priority |
| TestSet.Item | TestExecStatus.TestInstance | Run.StepFactory | Bug.Project |
| TestSet.Name | TestExecStatus.TsTestId | StepFactory.NewList | Bug.Status |
| TestSet.TestSetFolder | TestExecStatus.Message | Step.Name | Bug.Post |
| TestSet.Status | TestExecStatus.Status | Step.Status |
Running tests with the QTP integration, it is possible to publish the Test Directory to store the tests in the SCM; publish the tests to AnthillPro during the build; then resolve the tests during the deployment to be run (on a machine with QTP) after an instance of the application has been deployed/setup/installed.
| The integration will only work if the agent(s) running the QuickTest Pro steps uses the 32-bit JVM. If the agent(s) use a 64-bit JVM, the steps will fail. However, the AnthillPro server, which does not run the steps, can run on the 64-bit JVM. |
The QuickTest Pro integration is implemented as AnthillPro job steps configured on the Job Configuration page. When using the integration, click the Create Step button (or select the Insert After/Before icon) to add steps to a job. Once the job is configured, it is then added to the workflow under the Definition tab.
QTP Steps:
QuickTest Pro - Run Tests. Runs a set of tests in the Mercury QuickTest Pro tool.
QuickTest Pro - Publish Report. Publishes a QuickTest Pro Report.
| The QuickTest Pro integration can be used in conjunction with Quality Center. You must configure the Quality Center integration on the System page. See Quality Center Testing and Quality Center Issue Tracking. |
The Quality Center integration must be configured on the system page.
You must have administrative privileges to create workflows. See Manage Security.
A Fixed Agent Filter that runs on the agent with the QC COM API installed.
Alternately, the agent with the Quality Center API can be added to the Build Farm. This requires the use of a scripted filter to look for the presence of a variable/property (e.g., qc.agent=true) which must be added to the agent. If this option is chosen, it is advisable to include a filter on every build job that excludes the Quality Center agent. See Configure and Edit Agent Filters.
The integration will only work if the agent(s) running the QuickTest Pro steps uses the 32-bit JVM. If the agent(s) use a 64-bit JVM, the steps will fail. However, the AnthillPro server, which does not run the steps, can run on the 64-bit JVM.
Go to Administration, select the project that is to be tested, and click the Add Job icon.
On the New Job Configuration page, choose No (do not use the Job Wizard). Click Select.
On the next page, name the job, give a description (optional) and click Set.
On the job configuration page, click Create Step.
QuickTest Pro - Run Tests. Select the Insert After icon of the step prior to where the QuickTest Pro - Run Tests step is to be included (typically after the Set Working Directory step). Go to Tests > Mercury, select QuickTest Pro - Run Tests step, and click Select.
Name. Provide a name.
Description (optional). Give a short description.
Base Test Path. Provide the directory containing all QuickTest Pro tests to be run.
Web Application URL. If testing a web application, this field will override the application URL.
Result Base Path. To override the default result output location for the tests, give the new path. A new directory will be created for each test under the specified path.
Browser. Give the browser being used during testing. In some cases, the browser must be specified in order for the test to run successfully.
Fail On Error. Check the box to fail the step if any errors occur during the test.
Fail On Warning. Check the box to fail the step if any warnings occur during the test.
Show Additional Options (advanced). See Show Additional Options.
Click Save.
QuickTest Pro - Publish Report. Select the Insert After icon of the QuickTest Pro - Run Tests step. Go to Test > Mercury, select the QuickTest Pro - Publish Report step, and click Select. This step retrieves the Report.xml files generated by QuickTest Pro. Once they are retrieved from the build, AnthillPro will be able to access the results in a meaningful way.
Name the step.
Description (optional). Provide a brief description.
Report Name. Provide a name for the report (if left blank, it will be the step name).
Source Directory. The directory where the QuickTest Pro data files will be retrieved from.
Include Patterns. File name patterns that describe the files that will be retrieved.
Excluded Patterns. File name patterns identifying the files that will NOT be retrieved.
Show Additional Options (advanced). See Show Additional Options.
Click Save.
Complete workflow configuration is beyond the scope of this tutorial. The topics covered in detail below are specific to using the QuickTest Pro integration.
Go Administration and select the appropriate workflow. You can add the job to either your build workflow or as part of a secondary (i.e., testing) workflow.
Go to workflow Main > Definition tab.
From the drop-down menu, choose Embedded Definition and click Select.
Left-click the Start icon and select Insert Job After.
Select the job created in Configure Run QTP Test Job of this tutorial, a job pre-condition script, and click Insert Job.
The Job will appear on the workflow definition page.
Go to Dashboard > Build Life to be tested.
On the Build Life page, click the Run Secondary Process button.
From the drop-down box, choose the workflow to be run, and click Next.
Select the environment and click the Run button.
Once the workflow is complete, go to the Build Life page and select the Reports tab.
In addition to publishing the test results on the Build Life page, AnthillPro also stores the results in its data warehouse, thus enabling you to compare your test runs over time (trending).
Select the Publish_QTP_Results link to view the QuickTest Pro report.
To drill down on each Quality Center step on the Build Life Summary page, see Trace a Build Life to Source.
Use the Selenium integration to run any Selenium test suite. Selenium may be integrated as part of the build job or used as part of a non-originating workflow (secondary process) on any Build Life. Once the tests have been run, metrics are available on the Build Life Reports and Tests tabs.
How the integration is used will typically be determined by (a.) the duration of the test suite to be run; and (b.) when the test suite is to be run. For most users, running Selenium as part of a non-originating workflow (secondary process) is advisable. For example, to automatically run the test suite every time a build is performed, add a 'Run Another Workflow' job to the build (make sure the second job does not start until the build is complete). This will tell AnthillPro to run the secondary workflow that kicks off the test suite. Or, you can set up a manual task that runs the Selenium workflow (see Execute Workflow via Task). It is also possible to create a schedule that runs the test suite by creating a scheduled trigger on the workflow (see Use Triggers and Scheduled Builds).
The integration is implemented as an AnthillPro job step configured on the Job Configuration page. When using the integration, click the Create Step button (or select the Insert After/Before icon) to add the step to a job. Once the job is configured, it is then added to the workflow under the Definition tab.
Selenium Step:
Run Tests. Runs a Selenium test suite. May be used as part of an originating and/or non-originating workflow. See Using the Selenium Integration.
Selenium must be active, and access available to AnthillPro. See Selenium documentation.
You must have access to the Administration page. See Manage Security.
At least one Build Life must be active.
The Selenium integration allows AnthillPro users to run a Selenium test suite by configuring a non-originating workflow (secondary process) to be run on any successful Build Life. This section only covers the steps necessary for creating a job that uses Selenium and a secondary workflow that kicks off the test suite. While your job configuration will vary, the Selenium integration will most likely use a Set Working Directory and Run Selenium Test step in the secondary workflow (for other options, see Selenium).
Create a secondary workflow that will run the test suite.
Create a job to be added to the newly created workflow.
On the job page, click the Create Step button to add a Set Working Directory step to the job.
Select the Insert After icon of the Set Working Directory step. Go to Test > Selenium, select Run Tests, and click Select.
Configure Step:
Name the step.
Description. Provide a description.
Test Suite Path. Give the path to the Selenium test suite to be run. This is relative to the working directory.
Web Application URL. If you need to override the application's base URL, give it here.
Browser. Give the Selenium browser designation. Tests can only be run in one browser at a time. To run tests in multiple browsers, create a new step for each browser type.
Results file. Give the name of the file that contains the Selenium test results.
Port. If Selenium is using a port other than the default (4444), give it here. Otherwise, leave this filed blank.
Show Additional Options (advanced). Select the Show Additional Options link to configure more options.
Is Active. Select No to temporarily deactivate the step without deleting it; otherwise select Yes.
Pre-Condition Script. From the drop down menu, select the condition which must be met for the step to continue. Before editing an existing script or creating a new one, see Step Pre-Condition Scripts.
Ignore Failures. Select Yes if this step should not effect the determination for step continuation or the status determination of the job.
PostProcessingScript. Select a script for determining when commands should count as fail or succeed. See Post Processing Scripts.
Timeout. Enter the time in minutes after the start of the step when AnthillPro will consider the step as timed out and abort it.
Click Save.
See Get Selenium Reports.
Go to the Dashboard and select the originating workflow to be tested.
On the Main page, select the appropriate Build Life.
On the Build Life Main page, click the Run Secondary Process button.
From the drop-down box, choose the workflow created in the Using the Selenium Integration section, and click Next.
Select the environment and click the Run button.
Select the test from the Publish Reports or Tests menu.
Use the SilkCentral integration to run any test suite managed by Borland's SilkCentral Test Manager (see Using the SilkCentral Integration). SilkCentral may be integrated as part of the build job or used as part of a non-originating workflow (secondary process) on any Build Life. Once the tests have been run, metrics are available on the Build Life Reports and Tests tabs (see also Run SilkCentral Build Workflow and Run SilkCentral Secondary Workflow).
How the integration is used will typically be determined by (a.) the type and duration of the test suite to be run; and (b.) when the test suite is to be run. For example, to run Unit (or other short duration) tests as part of a Continuous Integration build, the Run Tests step must be included as part of the build process. See Use SilkCentral as Part of a Build.
The integration may also be used to run long tests, such as functional tests, as part of a secondary workflow that does not require any other job steps. See Use SilkCentral as Part of a Secondary Workflow.
In order to use the integration, AnthillPro must first be configured with SilkCentral (see Set Up SilkCentral). The integration is implemented as an AnthillPro job step configured on the Job Configuration page. When using the integration, click the Create Step button (or select the Insert After/Before icon) to add the step to a job. Once the job is configured, it is then added to the workflow under the Definition tab.
SilkCentral Step:
Run Tests. Runs a test suite using Borland SilkCentral Test Manager. May be used as part of an originating and/or non-originating workflow.
This tutorial will follow a simple project configuration that first uses the SilkCentral Run Tests step as part of an originating workflow (i.e., a Continuous Integration build job) to run Unit tests, and as a secondary workflow to run functional tests, etc. The example in this tutorial uses Subversion, but the basic configuration is similar for any repository type. Your jobs will vary, but the SilkCentral integration is added as a job step similar to what is described below.
Once set up as a SilkCentral user, AnthillPro must be able to access the server in order for the integration to work. Any steps relying on the SilkCentral integration will not work until the configuration is complete.
All fields may contain scripts and/or property lookups. See Scripting.
You must have AnthillPro administrative privileges. See Manage Security.
A project must be active in AnthillPro.
The SilkCentral URL must be available.
AnthillPro must be set up as a SilkCentral user. See SilkCentral documentation.
Go to System > SilkCentral Test Manager from the Integrations menu.
On the Borland SilkCentral Test Manager Integration page, click Edit.
Configure the integration:
Borland SilkCentral Server URL. Provide the SilkCentral server URL.
User Name. Give the SilkCentral user name assigned to AnthillPro.
Password. Provide the password AnthillPro will to connect to the SilkCentral server.
Password Script. To use a script or property lookups for the password, leave the Password field blank and enter the script here. See Scripting.
Click Set then Done.
The SilkCentral integration allows AnthillPro users to automate the test orchestration process by configuring a build (originating) workflow to run SilkCentral-managed tests during the build, or as a non-originating workflow (secondary process) to be run on any successful Build Life. This section only covers the steps necessary for creating a build job that uses SilkCentral and a secondary workflow that kicks off the test manager. While your job configuration will vary, the SilkCentral integration is used similar to what is described below.
The Set Up SilkCentral section of this tutorial must be complete.
You must have administrative privileges. See Manage Security.
A project with at least one Build Life must be active in AnthillPro (if using the integration as part of a secondary workflow).
To run Unit (or other short duration) tests as part of a Continuous Integration build, the Run Tests step must be included as part of the build process that typically includes a set working directory, get changelog, and publish changelog step. Once the SilkCentral Run Tests step is added to the build job, AnthillPro will kick off the tests associated with the step as part of the build, and provide feedback on the Dashboard Build Life page.
The items below only cover the steps necessary to using the SilkCentral integration as part of a build workflow.
Each build job is different; however the SilkCentral step is configured similar to what is below. It is also possible to configure AnthillPro to make decisions to pass or fail the build based on the results of the tests run during the build.
Go to Administration, select the project that is to be tested, and click the Add Job icon.
On the New Job Configuration page, choose No (do not use the Job Wizard). Click Select.
On the next page, name the job, give a description (optional) and click Set.
Follow the steps for creating a job.
Run Tests. Click the Insert After button of the step prior to the point where the Run Tests step is to be included (e.g., typically after the set working directory, build, get changelog steps). Go to Test > Silk Central Test Manager, select Run Tests step, and click Select.
Name the step.
Description. Provide an optional description.
Project Name. Give the name of the SilkCentral project.
Tests. Provide the path, relative to the SilkCentral project, of all the tests to be run by this job. If the tests are to be identified by scripts, separate them by a comma; if the path is hard coded, each test can be input on a separate line.
Ignore Missing Tests. Check the box to have AnthillPro ignore any missing tests. If a test is defined in the Tests field above and is missing, AnthillPro will fail the job unless the box is checked.
Execution Server Host. To restrict the host on which the tests can run, give the location of the SilkCentral Execution Host. If left blank, AnthillPro will run the tests on any of the available hosts.
Execution Server Port. If the execution server is restricted to a particular port, give it here. Inputting a value of -1 (negative one) will allow the tests to run on any available port.
Show Additional Options (advanced). Select the Show Additional Options link to configure more options.
Is Active. Select No to temporarily deactivate the step without deleting it; otherwise select Yes.
Pre-Condition Script. From the drop down menu, select the condition which must be met for the step to continue. Before editing an existing script or creating a new one, see Step Pre-Condition Scripts.
Ignore Failures. Select Yes if this step should not effect the determination for step continuation or the status determination of the job.
PostProcessingScript. Select a script for determining when commands should count as fail or succeed. See Post Processing Scripts.
Timeout Enter the time in minutes after the start of the step when AnthillPro will consider the step as timed out and abort it.
Click Save.
The SilkCentral job must be executed as part of a build workflow in order to run Unit tests as part of the Continuous Integration build. This section assumes familiarity with the originating workflow creation process, and only covers the topics necessary to adding the job (see Configure SilkCentral Build Job) to the workflow.
Go to Administration and select the Add Workflow icon of the appropriate project.
Check Originating Workflow and click Select.
Configure the workflow.
On the workflow Main page, select the Definition tab.
From the drop-down menu, choose Embedded Definition and click Select.
Left-click the Start icon and select Insert Job After.
Select the Build job created in the Configure SilkCentral Build Job section, a job pre-condition script, and click Insert Job.
Go to Dashboard and select the workflow created in the Configure SilkCentral Build Workflow section.
On the workflow Main page, click the Build button.
Once the workflow is complete, go to the Build Life page and select the Reports tab.
Select the test from the Publish Reports menu. A detail of each test run (not shown) is also available.
Once the workflow is complete, go to the Build Life page and select the Tests tab.
To drill down on each step on the Build Life Summary page, see Trace a Build Life to Source.
To automatically kick off long tests (e.g., functional tests), the SilkCentral Run Tests step is typically configured as part of a secondary workflow which is run on an existing Build Life. The integration does not require any other job steps to be run as part of the workflow, and, for example, can be scheduled to kick off the test suite using a schedule or trigger (see Triggers and Scheduled Builds).
The items below only cover the steps necessary to using the SilkCentral integration as part of a secondary (non-originating) workflow.
The Run Test step may be used on its own; however, depending on the needs of each project, the job may require other steps to start a virtual machine, pass links, evaluate scripts, etc.
Go to Administration, select the project that is to be tested, and click the Add Job icon.
On the New Job Configuration page, choose No (do not use the Job Wizard). Click Select.
On the next page, name the job, give a description (optional) and click Set.
Follow the steps for creating a job.
Run Tests. Click the Create Step button. Go to Test > Silk Central Test Manager, select Run Tests step, and click Select.
Name the step.
Description. Provide an optional description.
Project Name. Give the name of the SilkCentral project.
Tests. Provide the path, relative to the SilkCentral project, of all the tests to be run by this job. If the tests are to be identified by scripts, separate them by a comma; if the path is hard coded, each test can be input on a separate line.
Ignore Missing Tests. Check the box to have AnthillPro ignore any missing tests. If a test is defined in the Tests field above and is missing, AnthillPro will fail the job unless the box is checked.
Execution Server Host. To restrict the host on which the tests can run, give the location of the SilkCentral Execution Host. If left blank, AnthillPro will run the tests on any of the available hosts.
Execution Server Port. If the execution server is restricted to a particular port, give it here. Inputting a value of -1 (negative one) will allow the tests to run on any available port.
Show Additional Options (advanced). Select the Show Additional Options link to configure more options.
Is Active. Select No to temporarily deactivate the step without deleting it; otherwise select Yes.
Pre-Condition Script. From the drop down menu, select the condition which must be met for the step to continue. Before editing an existing script or creating a new one, see Step Pre-Condition Scripts.
Ignore Failures. Select Yes if this step should not effect the determination for step continuation or the status determination of the job.
PostProcessingScript. Select a script for determining when commands should count as fail or succeed. See Post Processing Scripts.
Timeout Enter the time in minutes after the start of the step when AnthillPro will consider the step as timed out and abort it.
Click Save.
The SilkCentral job can be executed as part of a secondary (non-originating) workflow in order to run functional (or other) tests on an existing Build Life. While there are numerous ways to use the SilkCentral integration to automatically run testing suites, your particular workflow will be similar to what is presented here. For example, you can set up the workflow to run on a scheduled trigger (see Triggers and Scheduled Builds) that kicks off the testing suite, or can even include functional and other long running testing as part of the build process (see also Use SilkCentral as Part of a Build).
Go to Administration and select the Add Workflow icon of the appropriate project.
Check Non-originating Workflow and click Select.
Configure workflow.
On the workflow Main page, select the Definition tab.
From the drop-down menu, choose Embedded Definition and click Select.
Left-click the Start icon and select Insert Job After.
Select the Build job created in the Configure SilkCentral Secondary Job section, a job pre-condition script, and click Insert Job.
If setting a trigger, see Triggers and Scheduled Builds.
Go to Dashboard and select the originating workflow to be tested.
On the Main page, select the appropriate Build Life.
On the Build Life Main page, click the Run Secondary Process button.
From the drop-down box, choose the workflow created in the Configure SilkCentral Secondary Workflow section, and click Next.
Select the environment and click the Run button.
Once the workflow is complete, go to the Build Life page and select the Reports tab.
Select the test from the Publish Reports menu. A detail of each test run (not shown) is also available.
Once the workflow is complete, go to the Build Life page and select the Tests tab.
To drill down on each step on the Build Life Summary page, see Trace a Build Life to Source.
To use TestNG with AnthillPro, publish the results in the JUnit format (See TestNG documentation) and configure the JUnit Integration. Once TestNG conforms to the JUnit output AnthillPro expects, the reports can then be made available in AnthillPro or sent in a notification -- exactly like any other JUnit report (see the JUnit integration for more information).
AnthillPro has an integration with TestNG that publishes the test data to the Build Life Test tab. The integration consists of a single job step that is added near the end of your job.
| The integration is written as an AnthillPro Plugin, included in the normal distribution. For older AnthillPro 3.7 versions, you will need to download the integration from Supportal and then upload it to the server. Once uploaded, ensure the Plugin is active. |
Once you have AnthillPro running your TestNG tests, you can add the TestNG Publish Test Report step near the end of your job:
Go to Administration, select the appropriate project, and click the Add Job icon.
On the New Job Configuration page, choose No (do not use the Job Wizard). Click Select.
Follow the steps for creating a job that runs your tests.
Select the Insert After icon of the step that will run before the TestNG Report Publisher step.
Go to Test > TestNG, select the TestNG Report Publisher step, and click Select.
Name the step.
Description. Provide a short description.
Working Directory Offset (optional). Give the working directory to use when executing this command. This is relative to current working directory. To use the current directory, leave this field blank.
Report Name. Give the name for this report to appear on the AnthillPro Dashboard. If none is given, TestNG will be used.
Base Directory. Provide the directory for resolving TestNG XML files. Unless absolute, this is relative to the job working directory.
Include Patterns. Give the file name patterns that describe the files to be retrieved. Each include pattern must be entered on a separate line.
You can also use the following wild cards to tell AnthillPro what to include:
**Indicates include every directory within the base directory.*Used to include every file. So, if you use*.zip, the files matching this pattern will be included.**/*Tells AnthillPro to retrieve the entire file tree underneath the base directory.
Exclude Patterns. Provide the file name patterns identifying the files that will NOT be retrieved. This field is set in the same way as the Include Patterns field, only you are telling AnthillPro what NOT to include.
Show Additional Options (advanced). Select the Show Additional Options link to configure more options.
Is Active. Select No to temporarily deactivate the step without deleting it; otherwise select Yes.
Pre-Condition Script. From the drop down menu, select the condition which must be met for the step to continue. Before editing an existing script or creating a new one, see Step Pre-Condition Scripts.
Ignore Failures. Select Yes if this step should not effect the determination for step continuation or the status determination of the job.
PostProcessingScript. Select a script for determining when commands should count as fail or succeed. See Post Processing Scripts.
Timeout. Enter the time in minutes after the start of the step when AnthillPro will consider the step as timed out and abort it.
Show Environment Variables (optional; advanced). Give any optional environment variables in "name=value" format.
Environment variables values may contain references to existing values in the following format:
name=${env/<NAME>};value. If the value of the <NAME> variable is "value2" in the current environment, then the above example will be expanded to:name=value2;value.Using this technique, it is possible add an entry to PATH in the following manner:
PATH=my/path/entry;0. Case is significant even on Windows systems.
Click Save.
Once the build is run, go to the new Build Life and select the Test tab. There, you can investigate the findings.