Java unit tests order

Saved searches

Use saved searches to filter your results more quickly

You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session.

Test execution order

Clone this wiki locally

By design, JUnit does not specify the execution order of test method invocations. Originally the methods were simply invoked in the order returned by the reflection API. However, using the JVM order is unwise since the Java platform does not specify any particular order, and in fact JDK 7 returns a more or less random order. Of course, well-written test code would not assume any order, but some do, and a predictable failure is better than a random failure on certain platforms.

From version 4.11, JUnit will by default use a deterministic, but not predictable, order. The following sections describe how to change the method execution order

From version 4.13, to can specify a different method execution order via the @OrderWith annotation. See the example below.

The parameter to @OrderWith is an instance of Ordering.Factory . JUnit provides implementations of Ordering.Factory in org.junit.tests.manipulation . Users can create their own instances of Ordering.Factory to provide Ordering implementations that reorder tests. Implementations of Ordering.Factory should have a public constructor that takes in a Ordering.Context (see the Alphanumeric source code for an example).

The Ordering class provides some static methods that simplify creation of Ordering instances (for example, Ordering.shuffledBy(Random random) will reorder the tests using the passed-in Random instance). The Request class has been updated to have a orderWith(Ordering) method that mirrors the older sortWith(Comparator comparator) method,.

The @OrderWith annotation works with any runner that implements Orderable . The ParentRunner abstract class was been retrofitted to implement Orderable so the runners provided by JUnit support @OrderWith , and many third-party runners will support it.

import org.junit.Test; import org.junit.runner.OrderWith; import org.junit.runner.manipulation.Alphanumeric; @OrderWith(Alphanumeric.class) public class TestMethodOrder < @Test public void testA() < System.out.println("first"); > @Test public void testB() < System.out.println("second"); > @Test public void testC() < System.out.println("third"); > >

Above code will execute the test methods in the order of their names, sorted in ascending order.

From version 4.11, you can change the test execution order simply annotate your test class using @FixMethodOrder and specify one of the available MethodSorters:

@FixMethodOrder(MethodSorters.JVM) : Leaves the test methods in the order returned by the JVM. This order may vary from run to run.

@FixMethodOrder(MethodSorters.NAME_ASCENDING) : Sorts the test methods by method name, in lexicographic order.

If you do no specify either @FixMethodOrder or @OrderWith , the default ordering is equivalent to @FixMethodOrder(MethodSorters.DEFAULT)

import org.junit.FixMethodOrder; import org.junit.Test; import org.junit.runners.MethodSorters; @FixMethodOrder(MethodSorters.NAME_ASCENDING) public class TestMethodOrder < @Test public void testA() < System.out.println("first"); > @Test public void testB() < System.out.println("second"); > @Test public void testC() < System.out.println("third"); > >

Above code will execute the test methods in the order of their names, sorted in ascending order.

Источник

JUnit 5 Test Execution Order

In JUnit 5, writing ordered tests is highly discouraged, there may be situations where test execution ordering becomes an important factor for testing the application functionality such as integration testing.

For example, we may want the integration testing for a user management module where the tests should execute in the following manner: create a new user, update the user and then delete the user.

JUnit 5 (checked with 5.8.1) supports the following ways to order tests in the interface MethodOrderer :

• MethodName – sorts tests alphanumerically based on their names.
• DisplayName – sorts tests alphanumerically based on their display names.
• OrderAnnotation – sorts tests based on the @Order annotation. Tests with same order will be sorted arbitrarily adjacent to each other. Not ordered tests will appear at the end.
• Random – orders tests pseudo-randomly. Randomness of tests in each execution can be controlled by property junit.jupiter.execution.order.random.seed .
• Custom – orders the test with provided ordering logic.

Another ordering class Alphanumeric has been marked deprecated and will be removed in JUnit 6. So you can avoid using it.

An example of sorting the tests based on their method names.

@TestMethodOrder(MethodOrderer.MethodName.class) public class MethodNameOrderedTests < @Test void testE() < assertTrue(true); >@Test void testA() < assertTrue(true); >@Test void testD() < assertTrue(true); >@Test void testC() < assertTrue(true); >@Test void testB() < assertTrue(true); >>

testA() testB() testC() testD() testE()

An example of sorting the tests based on their display names.

import org.junit.jupiter.api.DisplayName; import org.junit.jupiter.api.MethodOrderer; import org.junit.jupiter.api.Test; import org.junit.jupiter.api.TestMethodOrder; @TestMethodOrder(MethodOrderer.DisplayName.class) public class DisplayNameOrderedTests < @DisplayName("5") @Test void testE() < assertTrue(true); >@DisplayName("3") @Test void testA() < assertTrue(true); >@DisplayName("1") @Test void testD() < assertTrue(true); >@DisplayName("2") @Test void testC() < assertTrue(true); >@DisplayName("4") @Test void testB() < assertTrue(true); >>

An example of sorting the tests based on their ordering as mentioned in @Order annotation.

import org.junit.jupiter.api.MethodOrderer; import org.junit.jupiter.api.Order; import org.junit.jupiter.api.Test; import org.junit.jupiter.api.TestMethodOrder; @TestMethodOrder(MethodOrderer.OrderAnnotation.class) public class OrderAnnotationTests < @Order(5) @Test void testE() < assertTrue(true); >@Order(4) @Test void testA() < assertTrue(true); >@Order(3) @Test void testD() < assertTrue(true); >@Order(2) @Test void testC() < assertTrue(true); >@Order(1) @Test void testB() < assertTrue(true); >>

testB() testC() testD() testA() testE()

An example of sorting the tests randomly. By default, the random seed used for ordering methods is the System.nanoTime() which is generated during the static initialization of the test class.

We can change the randomness by providing a custom seed through property junit.jupiter.execution.order.random.seed.

junit.jupiter.execution.order.random.seed=9999

@TestMethodOrder(MethodOrderer.Random.class) public class RandomOrderedTests < @Test void testE() < assertTrue(true); >@Test void testA() < assertTrue(true); >@Test void testD() < assertTrue(true); >@Test void testC() < assertTrue(true); >@Test void testB() < assertTrue(true); >>

By default, the output will be different for each execution of the test class. But if we use a fixed seed using the property files, the order of the tests will be fixed and will not change between different executions.

The given output is after we have specified the custom seed 9999.

//Execution 1 testB() testC() testE() testA() testD() //Execution 2 testB() testC() testE() testA() testD()

It is possible to define your own custom ordering by implementing the interface MethodOrderer .

In the given example, we are executing the deprecated tests in the end.

import java.util.Comparator; import org.junit.jupiter.api.MethodDescriptor; import org.junit.jupiter.api.MethodOrderer; import org.junit.jupiter.api.MethodOrdererContext; public class DeprecatedInEndTestOrder implements MethodOrderer < private Comparatorcomparator = Comparator .comparing(md -> md.getMethod().isAnnotationPresent(Deprecated.class)); @Override public void orderMethods(MethodOrdererContext context) < context.getMethodDescriptors().sort(comparator); >>

import static org.junit.Assert.assertTrue; import org.junit.jupiter.api.Test; import org.junit.jupiter.api.TestMethodOrder; @TestMethodOrder(DeprecatedInEndTestOrder.class) public class CustomOrderTests < @Test @Deprecated void testC() < assertTrue(true); >@Test void testA() < assertTrue(true); >@Test void testD() < assertTrue(true); >@Deprecated @Test void testE() < assertTrue(true); >@Test void testB() < assertTrue(true); >>

Notice the test output. The deprecated tests have been executed in the last.

testA() testB() testD() testC() testE()

Reiterating the fact that test classes typically should not rely on the execution order, there are times when it may be needed.

2.1. Supported Ordering Types

JUnit 5 (checked with 5.8.1) supports the following ways to order tests in the interface ClassOrderer. We can use the class order in the same way as method ordering seen in the above sections.

• ClassName – sorts test classes alphanumerically based on their fully qualified class names.
• DisplayName – sorts test classes alphanumerically based on their display names.
• OrderAnnotation – sorts test classes numerically based on values specified via the @Order annotation.
• Random – orders test classes pseudo-randomly. Supports custom seed using the property junit.jupiter.execution.order.random.seed.

To set the order globally for all test classes, use the configuration property junit.jupiter.testclass.order.default .

junit.jupiter.testclass.order.default = org.junit.jupiter.api.ClassOrderer$OrderAnnotation

2.2. Test Class Ordering Example

Given is an example for test class ordering in nested test classes.

@TestClassOrder(ClassOrderer.OrderAnnotation.class) class OrderedTestClassesExample < @Nested @Order(1) class SetupTests < @Test void test1() < >> @Nested @Order(2) class AppFlowTests < @Test void test2() < >> >

SetupTests - test1() AppFlowTests - test2()

Let me iterate again that forcing an ordering on the test execution order is not recommended and should be avoided. Still, if you come across such a requirement then use above explained techniques to order the test classes and test methods.

I will recommend using @Order annotation with ClassOrderer.OrderAnnotation and MethodOrderer.OrderAnnotation to have full control over the ordering of tests if the test ordering is such important in your case.

Источник