Mutation testing

Mutation testing is a way to evaluate the quality of our unit tests. For mutation testing, the Stryker.NET tool automatically performs mutations in your code, runs tests, and generates a detailed report with the results.

Example test scenario

Consider a sample PriceCalculator.cs class with a Calculate method that calculates the price, taking into account the discount.

public class PriceCalculator
{
    public decimal CalculatePrice(decimal price, decimal discountPercent)
    {
        if (price <= 0)
        {
            throw new ArgumentException("Price must be greater than zero.");
        }

        if (discountPercent < 0 || discountPercent > 100)
        {
            throw new ArgumentException("Discount percent must be between 0 and 100.");
        }

        var discount = price * (discountPercent / 100);
        var discountedPrice = price - discount;

        return Math.Round(discountedPrice, 2);
    }
}

The preceding method is covered by the following unit tests:

[Fact]
public void ApplyDiscountCorrectly()
{
    decimal price = 100;
    decimal discountPercent = 10;

    var calculator = new PriceCalculator();

    var result = calculator.CalculatePrice(price, discountPercent);

    Assert.Equal(90.00m, result);
}

[Fact]
public void InvalidDiscountPercent_ShouldThrowException()
{
    var calculator = new PriceCalculator();

    Assert.Throws<ArgumentException>(() => calculator.CalculatePrice(100, -1));
    Assert.Throws<ArgumentException>(() => calculator.CalculatePrice(100, 101));
}

[Fact]
public void InvalidPrice_ShouldThrowException()
{
    var calculator = new PriceCalculator();

    Assert.Throws<ArgumentException>(() => calculator.CalculatePrice(-10, 10));
}

The preceding code highlights two projects, one for the service that acts as a PriceCalculator and the other is the test project.

Install the global tool

First, install Stryker.NET. To do this, you need to execute the command:

dotnet tool install -g dotnet-stryker

To run stryker, invoke it from the command line in the directory where the unit test project is located:

dotnet stryker

After the tests have run, a report is displayed in the console.

Stryker.NET saves a detailed HTML report in the StrykerOutput directory.

Now, consider what mutants are and what 'survived' and 'killed' mean. A mutant is a small change in your code that Stryker makes on purpose. The idea is simple: if your tests are good, they should catch the change and fail. If they still pass, your tests might not be strong enough.

In our example, a mutant will be the replacement of the expression price <= 0, for example, with price < 0, after which unit tests are run.

Stryker supports several types of mutations:

For additional mutation types, see the Stryker.NET: Mutations documentation.

Incremental improvement

If, after changing your code, the unit tests pass successfully, then they aren't sufficiently robust, and the mutant survived. After mutation testing, five mutants survive.

Let's add test data for boundary values and run mutation testing again.

[Fact]
public void InvalidPrice_ShouldThrowException()
{
    var calculator = new PriceCalculator();

    // changed price from -10 to 0
    Assert.Throws<ArgumentException>(() => calculator.CalculatePrice(0, 10));
}

[Fact] // Added test for 0 and 100 discount
public void NoExceptionForZeroAnd100Discount()
{
    var calculator = new PriceCalculator();

    var exceptionWhen0 = Record.Exception(() => calculator.CalculatePrice(100, 0));
    var exceptionWhen100 = Record.Exception(() => calculator.CalculatePrice(100, 100));

    Assert.Null(exceptionWhen0);
    Assert.Null(exceptionWhen100);
}

As you can see, after correcting the equivalent mutants, we only have string mutations left, which we can easily 'kill' by checking the text of the exception message.

[Fact]
public void InvalidDiscountPercent_ShouldThrowExceptionWithCorrectMessage()
{
    var calculator = new PriceCalculator();

    var ex1 = Assert.Throws<ArgumentException>(() => calculator.CalculatePrice(100, -1));
    Assert.Equal("Discount percent must be between 0 and 100.", ex1.Message);

    var ex2 = Assert.Throws<ArgumentException>(() => calculator.CalculatePrice(100, 101));
    Assert.Equal("Discount percent must be between 0 and 100.", ex2.Message);
}

[Fact]
public void InvalidPrice_ShouldThrowExceptionWithCorrectMessage()
{
    var calculator = new PriceCalculator();

    var ex = Assert.Throws<ArgumentException>(() => calculator.CalculatePrice(0, 10));
    Assert.Equal("Price must be greater than zero.", ex.Message);
}

Mutation testing helps to find opportunities to improve tests that make them more reliable. It forces you to check not only the 'happy path', but also complex boundary cases, reducing the likelihood of bugs in production.