Differential / Regression Test Generation

Using EvoSuite’s Regression test suite generation (EvoSuiteR), you can generate a test suite revealing differences between two versions of a Java class. This tutorial guides you through generating a differential test using EvoSuiteR, on the command-line.

(Improve this document by helping on our wiki)


Basic Structure

Given that EvoSuiteR requires a pair of versions, the general structure of the commands are as follows:

java -jar evosuite.jar -regressionSuite -projectCP "$originalCP" -Dregressioncp="$regressionCP" <target> [options]

In the command above, $originalCP refers to the Java class path to the original version of the program, and $regressionCP refers to the class path to the modified version of the program.

The term original in our context refers to the version which EvoSuiteR considers as correct, and the term regression refers to the version which EvoSuiteR considers to have potential regressions. Thus, the generated tests are intended to pass on the original version, and fail on the regression version.

Generating a Regression Test Suite for a Single Class

Consider you have a Java class my.pkg.HelloWorld, that is compiled and available under original/build/classes/my/pkg/HelloWorld.class. After you make a change to this class, you compile a new version under new/build/classes/my/pkg/HelloWorld.class.

To generate a regression test suite revealing the change between the two classes, you can run the following command:

java -jar evosuite.jar -regressionSuite -projectCP "original/build/classes" -Dregressioncp="new/build/classes" -class my.pkg.HelloWorld

Class dependencies can also be provided to the projectCP/regressioncp, using the Java class-path structure. Additional options such as the search budget can be provided using the same properties as EvoSuite options.

Generating a Regression Test Suite for Whole Projects

EvoSuiteR can also be used to generate tests for all class pairs between two .jar packages.

Assuming we have a program a/project.jar and a new version of this program b/project.jar, and the two programs have different set of dependencies – which for the sake of brevity, we only include one dependency, but as many dependencies can be put on the classpath – the following command can be used to generate a test suite for all pairs of classes across the two versions:

java -jar evosuite.jar -regressionSuite -projectCP "a/project.jar:a/dependency.jar" -Dregressioncp="b/project-new.jar:b/dependency.jar" -target project.jar

When generating differential tests across two versions of a program, it is likely that not all classes have been modified across the two versions. EvoSuiteR has a functionality to only generate tests for pairs of classes where the underlying code has been changed. To enable this, you can add the option -Dregression_skip_similar=true.