How I simplified testing in .NET via Storm Petrel

1. Introduction 1.1 The Challenge of Managing Expected Baselines in .NET Testing In .NET testing, expected baselines are a critical component of test cases. These baselines can take various forms, including: Hardcoded values in C#: Constants, initialized instances, or inline expressions within test methods, attributes, or data source methods called by test frameworks. This is often referred to as the "traditional testing approach." External files: JSON, XML, CSV, or other text-based formats, as well as binary data like images, PDFs, or DOCX files. This is commonly associated with the "snapshot testing approach." The lifecycle of baselines typically involves: Initial creation: Baselines are created during test development, often alongside the code being tested. 1.2 Why Traditional Approaches Can Be Inefficient Manually managing baselines can significantly slow down development, especially when: There are a large number of use cases or assertions. The expected baselines have complex structures or many properties. 1.3 Introducing Storm Petrel .NET To address these challenges, tools like Scand Storm Petrel Baselines Rewriter can automate the process of updating baselines. This tool provides a "Rewrite Baseline" feature through auto-generated tests, which we will explore in this article. 2. The Problem with Baseline Management in .NET Testing 2.1 Overview of Expected Baselines in .NET Testing .NET unit and integration tests support a wide variety of expected baselines and assertion methods. Developers can write test code and assert baselines using different approaches, such as: - xUnit with inline baselines: Assert.Equal(expected: 100, actual: actualObject.IntProperty); Assert.Equal("Expected string property", actualObject.StringProperty); - FluentAssertions with object equivalency: BaselineClass expected = new() { IntProperty = 100, StringProperty = "Expected string property", ` }; actual.Should().BeEquivalentTo(expected); - Snapshot testing: var expectedSnapshotBytes = SnapshotProvider.ReadAllBytes(); var actualSnapshotBytes = actualForm.AsPngSnapshotStream().ToArray(); actualSnapshotBytes.Should().Equal(expectedSnapshotBytes); - NUnit with JSON assertions: var actual = await response.Content.ReadAsStringAsync(); Assert.That(actual, Is.EqualTo((@"{""value"":100,""text"":""Some expected text""}")); These examples can vary based on: Test frameworks: xUnit, NUnit, MSTest, etc. Assertion libraries: FluentAssertions, Shouldly, or built-in assertions. Baseline locations: Local variables, inline expressions, test attributes, data source methods, or snapshot files. C# syntax or serialization characteristics: Inclusion of default values, verbatim strings, number prefixes/suffixes, etc. 2.2 Common Pain Points Each approach has its own advantages and disadvantages in terms of readability, maintainability, and extensibility. There is no one-size-fits-all solution, and developers often need to work with multiple approaches due to: Variations in product layers or components. Differences in product domains. Historical reasons for test code variations. When fixing bugs or adding new features, developers often need to update or create new baselines. This can be time-consuming, especially when dealing with: Complex object structures. A large number of test cases or assertions. For example: Adding a new Web API endpoint that returns a complex object with nested properties. Fixing a bug that requires updating multiple baselines. Adding a new property to a class, which requires updating all related tests. 2.3 Impact on Maintainability and Developer Experience To address these challenges, developers can: Manually update baselines. Convert tests to file snapshot testing and use tools to automate baseline updates. Use Scand Storm Petrel Baselines Rewriter to auto-generate and update baselines. Each approach has its pros and cons, which we will discuss below. 3. Snapshot Testing in .NET: Benefits and Challenges 3.1 What Is Snapshot Testing? Snapshot testing involves capturing the output of code execution (e.g., JSON, XML, images) and comparing it to an expected baseline. Baselines can be stored as: C# code: Primitive elements like strings or byte arrays. External files: JSON, XML, or other formats. However, the term "snapshot testing" is not always clearly defined. For example: If a Web API returns JSON, it is clearly a snapshot. If the same data is asserted as a C# object, it is considered "traditional testing". 3.2 Advantages of Snapshot Testing Specialized tools: Snapshot files can be reviewed and compared using specialized software. Easier updates: Developers can inject simple code like "File.WriteAllText(expectedValueJsonFilePath, actualValueJson); " to automatically create or update snapshots. Flexibility: Any test can be converted to a snapshot test using serialization. 3.3 Challenge

Mar 31, 2025 - 11:36

How I simplified testing in .NET via Storm Petrel

1. Introduction

1.1 The Challenge of Managing Expected Baselines in .NET Testing
In .NET testing, expected baselines are a critical component of test cases. These baselines can take various forms, including:

Hardcoded values in C#: Constants, initialized instances, or inline expressions within test methods, attributes, or data source methods called by test frameworks. This is often referred to as the "traditional testing approach."
External files: JSON, XML, CSV, or other text-based formats, as well as binary data like images, PDFs, or DOCX files. This is commonly associated with the "snapshot testing approach."
The lifecycle of baselines typically involves:
Initial creation: Baselines are created during test development, often alongside the code being tested.

1.2 Why Traditional Approaches Can Be Inefficient
Manually managing baselines can significantly slow down development, especially when:

There are a large number of use cases or assertions.
The expected baselines have complex structures or many properties.

1.3 Introducing Storm Petrel .NET

To address these challenges, tools like Scand Storm Petrel Baselines Rewriter can automate the process of updating baselines. This tool provides a "Rewrite Baseline" feature through auto-generated tests, which we will explore in this article.

2. The Problem with Baseline Management in .NET Testing

2.1 Overview of Expected Baselines in .NET Testing

.NET unit and integration tests support a wide variety of expected baselines and assertion methods. Developers can write test code and assert baselines using different approaches, such as:

- xUnit with inline baselines:
    Assert.Equal(expected: 100, actual: actualObject.IntProperty);
    Assert.Equal("Expected string property", actualObject.StringProperty);
- FluentAssertions with object equivalency:
    BaselineClass expected = new()
    {
        IntProperty = 100,
        StringProperty = "Expected string property",
`    };
    actual.Should().BeEquivalentTo(expected);
- Snapshot testing:
    var expectedSnapshotBytes = SnapshotProvider.ReadAllBytes();
    var actualSnapshotBytes = actualForm.AsPngSnapshotStream().ToArray();
    actualSnapshotBytes.Should().Equal(expectedSnapshotBytes);
- NUnit with JSON assertions:
    var actual = await response.Content.ReadAsStringAsync();
    Assert.That(actual, Is.EqualTo((@"{""value"":100,""text"":""Some expected text""}"));

These examples can vary based on:

Test frameworks: xUnit, NUnit, MSTest, etc.
Assertion libraries: FluentAssertions, Shouldly, or built-in assertions.
Baseline locations: Local variables, inline expressions, test attributes, data source methods, or snapshot files.
C# syntax or serialization characteristics: Inclusion of default values, verbatim strings, number prefixes/suffixes, etc.

2.2 Common Pain Points

Each approach has its own advantages and disadvantages in terms of readability, maintainability, and extensibility. There is no one-size-fits-all solution, and developers often need to work with multiple approaches due to:

Variations in product layers or components.
Differences in product domains.
Historical reasons for test code variations.

When fixing bugs or adding new features, developers often need to update or create new baselines. This can be time-consuming, especially when dealing with:

Complex object structures.
A large number of test cases or assertions.

For example:

Adding a new Web API endpoint that returns a complex object with nested properties.
Fixing a bug that requires updating multiple baselines.
Adding a new property to a class, which requires updating all related tests.

2.3 Impact on Maintainability and Developer Experience

To address these challenges, developers can:

Manually update baselines.
Convert tests to file snapshot testing and use tools to automate baseline updates.
Use Scand Storm Petrel Baselines Rewriter to auto-generate and update baselines.

Each approach has its pros and cons, which we will discuss below.

3. Snapshot Testing in .NET: Benefits and Challenges

3.1 What Is Snapshot Testing?

Snapshot testing involves capturing the output of code execution (e.g., JSON, XML, images) and comparing it to an expected baseline. Baselines can be stored as:

C# code: Primitive elements like strings or byte arrays.
External files: JSON, XML, or other formats.

However, the term "snapshot testing" is not always clearly defined. For example:

If a Web API returns JSON, it is clearly a snapshot.
If the same data is asserted as a C# object, it is considered "traditional testing".

3.2 Advantages of Snapshot Testing

Specialized tools: Snapshot files can be reviewed and compared using specialized software.
Easier updates: Developers can inject simple code like "File.WriteAllText(expectedValueJsonFilePath, actualValueJson); " to automatically create or update snapshots.
Flexibility: Any test can be converted to a snapshot test using serialization.

3.3 Challenges of Snapshot Testing

Loose coupling drawbacks: Serialized snapshots (e.g., JSON) make it harder to refactor code (e.g., renaming properties) or discover where a property is referenced.
Complex assertions: Comparing serialized data (e.g., JSON) can result in false positives due to formatting inconsistencies. Alternatively, deserializing the data back for equivalency checks introduces performance overhead.
Manual updates: Updating C# code snapshots or reviewing individual file snapshots can be time-consuming.

4. Storm Petrel .NET: A Better Approach to Managing Expected Baselines

4.1 Eliminating the Need for Separate Snapshot Files

Storm Petrel is a .NET tool that automates the rewriting of expected baselines, whether they are stored in C# code or external files. It uses .NET Incremental Generators to auto-generate StormPetrel tests that update baselines.

4.2 Key features

Storm Petrel can rewrite baselines located in:

Local variables, static properties, or inline expressions.
Test case attribute arguments or data source method return values.
Snapshot files.

4.3 How Storm Petrel Works in Practice

After configuring Storm Petrel for your test project, the workflow is simple:

Detect failed tests or new baselines.
Run auto-generated StormPetrel tests to update baselines. -Traditional Tests: GIF Image -File Snapshot Tests: GIF Image
Review the updated baselines to ensure correctness.

5. Comparing Storm Petrel to Other .NET Testing Tools

5.1 Verify vs. Storm Petrel

Verify: Requires converting tests to snapshot-style and uses specialized Verify plugins for comparison.
Storm Petrel: Works with existing tests and updates baselines directly in C# code or snapshot files.

5.2 Other Tools

Tools like Snapshooter, Meziantou.Framework.InlineSnapshotTesting, and ApprovalTests.Net also support snapshot testing but require significant changes to test code similar to Verify.

5.3 When to Choose Storm Petrel

Use Storm Petrel if you need to work with baselines stored in C# code.
Use Verify or Storm Petrel if you need to store baselines in external files.

6. How to Get Started with Storm Petrel

6.1 Step-by-Step Integration

According to Storm Petrel Getting Started:

Add the following NuGet packages to your test project:
Scand.StormPetrel.Generator: Auto-generates tests to update baselines.
An object-to-C# dumper (e.g., VarDump, ObjectDumper.NET, or a custom implementation).
Optionally, Scand.StormPetrel.FileSnapshotInfrastructure for file snapshot testing.
Configure Storm Petrel via appsettings.StormPetrel.json.
Run the auto-generated StormPetrel tests to update baselines.

6.2 Tips for effective snapshot verification in unit tests

In addition to the primary examples, Storm Petrel supports a variety of use cases, including:

Supported Use Case Variations

Baselines in C# Code:
- Local variables, static properties, or inline expressions used in assertions.
- Test case attribute arguments or data source method return values mapped to the "expected" parameters of test methods.
Snapshot Files:
- Configuration: Default settings, default configuration with custom options, or fully custom configurations.
- Snapshot Read Kind: Read file text, binary data, or open files as streams.

Additional Features and Tips

According to the FAQs, developers can:

Create or Update Baselines:

Generate expected baselines for new tests.
Rewrite expected baselines for existing tests.

Optimize for CI/CD:

Keep Storm Petrel disabled in CI/CD pipelines to minimize side effects. Enable it only in development environments.

Manage Backups:

Disable test code file backups, as developers typically track changes using Git.

Logging:

Disable logging by default. Enable it only when debugging issues with Storm Petrel.

Customize Assertions:

Configure ignored properties to exclude unnecessary assertions from being updated.

Selective Test Generation:

Ignore specific tests to prevent Storm Petrel from generating baseline updates for them.

Run Auto-Generated Tests:

Execute auto-generated Storm Petrel tests from command-line tools or IDE test runners on both Windows and Unix-like platforms. Execute Storm Petrel test individually or as a batch.

7. Final Thoughts & Call to Action

7.1 Recap of Storm Petrel’s Benefits

Scand Storm Petrel utilizes NuGet and .NET Incremental Generators infrastructure and can be actively used in Traditional or Snapshot Testing. With minimal or no side effects on unit/integration test structure and behavior, it efficiently allows the creation/rewriting of expected baselines or snapshots, speeding up and simplifying software development.

7.2 Who should consider using it and when
Storm Petrel should be considered by .NET developers and architects who:

Value efficient unit/integration testing.
Actively develop and maintain tests with complex baselines.
Seeks a tool that adapts to their tests without disrupting CI/CD processes.

7.3 Learn More
For more details, visit the Storm Petrel product page.

7.4 Share Your Experience
Feel free to share your experiences or ask questions in the comments below!