Learn About Sanity Testing With TestNG

Optimizing execution time for test builds is an all-time hot topic for test automation engineers, and organizing tests and their dependencies plays a crucial part. So, this article showcases the use cases of Sanity tests with some practical highlights and usage.

Quite recently, I have faced a requirement to minimize the overall test execution time for the unit and integration testing jobs that were running overnight. Generally, there are two ways in which we can figure out how to minimize the overall test execution time.

Write your test cases in a way that they execute independently and fast. This becomes possible by using less redundant validations, annotations such as skip exceptions, and soft or hard assertions. However, going deeper into the explanation of this topic is not going to be covered in this article.
Designing overall project structure in a way that it gives a look and feel of optimized and organized independent test suites (groups or categories – based on the individual product feature). This becomes possible when we include concepts such as priority test execution, groups, dependencies, etc. in our project structure while designing the testing framework. And this is what we are going to talk about in this short article – sanity testing using the concept of grouping and depending on the feature of TestNG.

That said, before proceeding, note that TestNG is a testing framework to perform end-to-end testing, including Unit, Component, Integration, Regression, UI, etc. That means it offers a wide range of features to cater to the requirements of all these test types. However, we will only be talking about Sanity testing.

Sanity Testing

Sanity represents the rationality measure and mental health of a person (object). This drives the concept of sanity testing. Sanity testing refers to some tests that should run ahead of all other test cases in the suite, and its outcome (pass/fail) will decide whether the rest of the test cases should execute or not. Clear enough, no? Let me give you some real-project examples to understand the applications of sanity testing, and how it helps us to optimize the execution time which is the real meat of this article.

Demo Application

MorningSmoothie is a smoothie business application. For the sake of a demo and a handy understanding of the concept of Sanity Testing using TestNG, I have covered unit testing for only one feature of this application, which is product management. Here goes the project structure,

Demo Application

The code can be found on my git. This is obviously a simple and standard structure for any Java testing framework. Two major folders, main and test extend from the src folder and contain further sub-packages. main contains codebase (application-related classes, constants, business logic layer, utilities, etc.). Whereas, folder test is for the feature-specific test cases. For this sample, the resources folder in both, the main and test are empty. We store configuration files here.

Look closer and you can see that I kept my POJO separated from my repository (ProductManager) class in main to augment its interactivity in an independent way. Similarly, in the test folder, there are dedicated packages for logically different classes.

However, this is my approach to logically designing a small testing application. You can design your classes and their derivation in any way you want as long as they remain reusable and make the scalability of your framework easy.

Code Explanation

// src/main/java/com/morningSmoothies/repositories/ProductManager.java
public class ProductManager {
    private List<FreshSmoothie> productStorage;
    
    public ProductManager() {
        productStorage = new ArrayList<FreshSmoothie>();
    }
    
    // TODO: you can add any fancy business logic before performing any of the CRUD operations
    public boolean addProduct(FreshSmoothie smoothie) {
        return productStorage.add(smoothie);
    }
    
    public FreshSmoothie getProduct(int id) {
        for (FreshSmoothie s : productStorage) {
            if (s.equals(id)) {
                return s;
            }
        }
        return null;
    }
    
    public boolean deleteProduct(final int id) {
        return productStorage.removeIf(e -> e.equals(id));
    }
    
    public List<FreshSmoothie> getAllProducts() {
        return productStorage;
    }
}

Starting with the ProductManager class [CodeSample1]. We have some methods to perform different day-to-day business operations related to our product. These are simple, CRUD operations (Create, read, update, and delete) that consume our private product Storage. Now, we will write some unit tests to check if these functions are working as expected [CodeSample2].

// src/test/java/unit/FreshSmoothie/CRUDTests.java
public class CRUDTests extends BaseClassUnitTesting {
    ProductManager pm;
    FreshSmoothie fs1, fs2, fs3;

    @BeforeMethod
    public void localSetup() {
        // Arrange setup
        fs1 = new FreshSmoothie(1, "MalonSmoothie", 25);
        fs2 = new FreshSmoothie(2, "PeachSmoothie", 30);
        fs3 = new FreshSmoothie(3, "MangoSmoothie", 35);
        pm = new ProductManager();
    }

    @Test(description = "Verify that addProduct method returns true when adds product successfully")
    public void successfulProductAdditionReturnsTrue() {
        // Act
        boolean result = pm.addProduct(fs1);
        // Assert
        Assert.assertTrue(result);
    }

    @Test(description = "Verify that getProduct method returns null if product does not exist")
    public void nonExistingProductReturnsNull() {
        // Arrange
        boolean result = pm.addProduct(fs1);
        // Act
        FreshSmoothie fsReturned = pm.getProduct(fs1.getID());
        // Assert
        Assert.assertNull(fsReturned, "The method should return null if it doesn't find an added product for the given ID");
    }

    @Test(description = "Verify that getAllProducts method returns valid product collection")
    public void productStorageReturnValidCount() {
        // Arrange
        pm.addProduct(fs1);
        pm.addProduct(fs2);
        pm.addProduct(fs3);
        // Act
        List<FreshSmoothie> smoothies = pm.getAllProducts();
        // Assert
        Assert.assertEquals(3, smoothies.size());
    }
}

I cannot bring myself to explain the code line by line, as I think this is too explanatory. Therefore, I have arranged my tests in AAA notation (Arrange – Act – Assert). These tests are really simple ones, but do the job for the sake of the demo. One thing to note here is, we have a local setup which is to arrange mutual holders for all the tests in the class, take this as a local suite setup which has to run before every test in this class. Moreover, I have added GlobalSetup to mimic the behavior of the global suite setup which will run once before every execution of the suite. This global suite setup is added in the base class – BaseClassUnitTesting.

Obviously, you will not get the importance of these local and global suite and method level calls at this point, however, these are crucial in a real application to clear the mutual resources, creating metadata and initial sessions, or for the reporting logs to its least parts.

Noticeably, there are five tests in this test/unit folder with the @Test notation. I have written Sanity test cases in the test/unit/sanity tests folder; however, I have not added any group and DependsOn fields yet. So, we expect our test to run as normal in alphabetical order.

As a TestNG IntelliJ user, you must know multiple ways to run your tests in a class or package. So, I’m running them manually by selecting the following option on my test/unit folder.

Unit folder

Once your tests get executed, you will see the below output window. By scanning the output window, you can see the sanity tests have been executed but in normal order. Nevertheless, the object is to run our sanity test before other tests and ensure the execution of those other tests based on the successful outcome of our sanity tests.

Outcome

So, I will just add fields groups and dependsOnGroups with the sanity and other CRUDTests class tests respectively.

You must be intrigued about the group with the name sanity. Refer to the screenshot below for this. These sanity tests were already part of our previous execution but without the field @Test(groups = "sanity").

Test

Default suite

Now we will run our tests again as previously and note if there is any difference in the execution sequence.

Fair enough, right? As this time the test execution sequence is not alphabetical but logically dependent. Sanity tests ran first before the other tests. Even now, I am sure, most of you have not gotten the idea and benefit we can get just by making our test execution sequence dependent on some logic.

Therefore, for the demo, I will make my sanity test fail intentionally and then let you observe the difference.

Difference

Please note the highlighted parts of the image. I have generated a ‘pided by Zero Exception’ to make my sanity test fail, and the execution of the dependent tests is ignored. In contrast, if there is no such group and dependsOnGroups fields used in our tests, our execution time will be wasted if some very basic check that has to execute successfully fails and makes the execution of all other tests eventually fail too.

Conclusion

The article was meant to be a short guide to using TestNG features groups and dependsOnGroups to promote Sanity Testing. I walked you through the concept of how we should organize our tests in a logically dependent order so that they can optimize our execution time at times when very fundamental checks are failing.

The sanity test written for the demo application covered the basic functionality that needed to perform well in order to pass all the other tests, such as the creation of product storage and a successful add operation for our product. And if any of this fails, execution of the rest of the tests is meaningless, as they will have to fail anyway. Lastly, the code sample is uploaded on the github which you can clone and start off right away.