Building a Comprehensive Observability Stack for a .NET API with Core Banking Integration

Introduction In this article, I'll walk you through how I implemented a complete observability solution for a .NET 8 minimal API that integrates with a core banking system. I'll cover the architecture decisions, implementation process, challenges faced, and the final result that provides comprehensive monitoring capabilities. The Challenge My .NET API needed to reliably interface with a core banking system through SOAP services while providing comprehensive monitoring capabilities. The initial codebase had limited observability features, making it difficult to: Track performance issues Monitor service health Trace requests across system boundaries Identify and diagnose errors quickly The Solution Architecture I implemented a complete observability stack consisting of: Structured Logging with Serilog Distributed Tracing with OpenTelemetry and Jaeger Metrics Collection with Prometheus Visualization with Grafana Health Checks for the API and its dependencies The architecture looks like this: ┌───────────────────┐ ┌───────────────┐ ┌─────────────┐ │ .NET API │────►│ Prometheus │────►│ Grafana │ │ - Metrics endpoint│ │ (Metrics DB) │ │ (Dashboards)│ │ - Health checks │ └───────────────┘ └─────────────┘ │ - OpenTelemetry │ │ - Serilog │ ┌───────────────┐ └───────┬───────────┘────►│ Jaeger │ │ │ (Tracing) │ │ └───────────────┘ │ │ ┌───────────────┐ └────────────────►│ Seq │ │ (Logs) │ └───────────────┘ Implementation Steps 1. Setting Up Structured Logging I implemented Serilog for structured logging with multiple sinks: builder.Host.UseSerilog((context, configuration) => configuration .ReadFrom.Configuration(context.Configuration) .Enrich.FromLogContext() .Enrich.WithMachineName() .Enrich.WithCorrelationId() .WriteTo.Console() .WriteTo.Seq("http://localhost:5342") .WriteTo.File("logs/log-.txt", rollingInterval: RollingInterval.Day)); 2. Adding OpenTelemetry for Distributed Tracing I integrated OpenTelemetry with Jaeger for distributed tracing: builder.Services.AddOpenTelemetry() .WithTracing(tracing => tracing .AddAspNetCoreInstrumentation() .AddHttpClientInstrumentation() .AddSource("BankingService") .AddOtlpExporter(options => options.Endpoint = new Uri("http://localhost:4317"))); This configuration sends trace data to Jaeger, where I can visualize complete request flows and identify performance bottlenecks across service boundaries. 3. Configuring Metrics Collection For metrics, I initially tried OpenTelemetry.Exporter.Prometheus.AspNetCore but switched to prometheus-net due to stability issues: builder.Services.AddSingleton(); // Prometheus metrics endpoint app.UseMetricServer(); 4. Implementing Health Checks I added comprehensive health checks: builder.Services.AddHealthChecks() .AddCheck("API", () => HealthCheckResult.Healthy()) .AddCheck("BankingService", tags: new[] { "services" }); 5. Docker Compose for Observability Infrastructure I created a dedicated docker-compose file for the observability stack: version: '3.8' services: prometheus: image: prom/prometheus volumes: - ./prometheus:/etc/prometheus ports: - "9090:9090" grafana: image: grafana/grafana volumes: - ./grafana/provisioning:/etc/grafana/provisioning - ./grafana/dashboards:/var/lib/grafana/dashboards ports: - "3001:3000" jaeger: image: jaegertracing/all-in-one ports: - "16686:16686" - "4317:4317" seq: image: datalust/seq ports: - "5341:80" - "5342:5341" Challenges Encountered 1. Prometheus Integration Issues The OpenTelemetry.Exporter.Prometheus.AspNetCore package lacked a stable release. I switched to prometheus-net, which required refactoring my metrics collection approach but provided better stability. 2. Docker Networking Problems Initially, Prometheus couldn't scrape my API metrics. The issue was in the Prometheus configuration, which needed to use host.docker.internal:5071 to access the API running on the host: scrape_configs: - job_name: 'banking-integration' scrape_interval: 5s static_configs: - targets: ['host.docker.internal:5071'] 3. Port Conflicts I encountered port conflicts with Grafana's default port (3000). The solution was simple - changing it to 3001 in my docker-compose configuration. The Final Result My observability solution provides comprehensive monitoring capabilities: Structured logs view

May 8, 2025 - 21:09

Building a Comprehensive Observability Stack for a .NET API with Core Banking Integration

Introduction

In this article, I'll walk you through how I implemented a complete observability solution for a .NET 8 minimal API that integrates with a core banking system. I'll cover the architecture decisions, implementation process, challenges faced, and the final result that provides comprehensive monitoring capabilities.

The Challenge

My .NET API needed to reliably interface with a core banking system through SOAP services while providing comprehensive monitoring capabilities. The initial codebase had limited observability features, making it difficult to:

Track performance issues
Monitor service health
Trace requests across system boundaries
Identify and diagnose errors quickly

The Solution Architecture

I implemented a complete observability stack consisting of:

Structured Logging with Serilog
Distributed Tracing with OpenTelemetry and Jaeger
Metrics Collection with Prometheus
Visualization with Grafana
Health Checks for the API and its dependencies

The architecture looks like this:

┌───────────────────┐     ┌───────────────┐     ┌─────────────┐
│ .NET API          │────►│ Prometheus    │────►│ Grafana     │
│ - Metrics endpoint│     │ (Metrics DB)  │     │ (Dashboards)│
│ - Health checks   │     └───────────────┘     └─────────────┘
│ - OpenTelemetry   │     
│ - Serilog         │     ┌───────────────┐     
└───────┬───────────┘────►│ Jaeger       │     
        │                 │ (Tracing)    │     
        │                 └───────────────┘     
        │                 
        │                 ┌───────────────┐     
        └────────────────►│ Seq          │     
                          │ (Logs)       │     
                          └───────────────┘

Implementation Steps

1. Setting Up Structured Logging

I implemented Serilog for structured logging with multiple sinks:

builder.Host.UseSerilog((context, configuration) => 
    configuration
        .ReadFrom.Configuration(context.Configuration)
        .Enrich.FromLogContext()
        .Enrich.WithMachineName()
        .Enrich.WithCorrelationId()
        .WriteTo.Console()
        .WriteTo.Seq("http://localhost:5342")
        .WriteTo.File("logs/log-.txt", rollingInterval: RollingInterval.Day));

2. Adding OpenTelemetry for Distributed Tracing

I integrated OpenTelemetry with Jaeger for distributed tracing:

builder.Services.AddOpenTelemetry()
    .WithTracing(tracing => tracing
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        .AddSource("BankingService")
        .AddOtlpExporter(options => options.Endpoint = new Uri("http://localhost:4317")));

This configuration sends trace data to Jaeger, where I can visualize complete request flows and identify performance bottlenecks across service boundaries.

3. Configuring Metrics Collection

For metrics, I initially tried OpenTelemetry.Exporter.Prometheus.AspNetCore but switched to prometheus-net due to stability issues:

builder.Services.AddSingleton<IMetricsRegistry, MetricsRegistry>();
// Prometheus metrics endpoint
app.UseMetricServer();

4. Implementing Health Checks

I added comprehensive health checks:

builder.Services.AddHealthChecks()
    .AddCheck("API", () => HealthCheckResult.Healthy())
    .AddCheck<BankingServiceHealthCheck>("BankingService", tags: new[] { "services" });

5. Docker Compose for Observability Infrastructure

I created a dedicated docker-compose file for the observability stack:

version: '3.8'

services:
  prometheus:
    image: prom/prometheus
    volumes:
      - ./prometheus:/etc/prometheus
    ports:
      - "9090:9090"

  grafana:
    image: grafana/grafana
    volumes:
      - ./grafana/provisioning:/etc/grafana/provisioning
      - ./grafana/dashboards:/var/lib/grafana/dashboards
    ports:
      - "3001:3000"

  jaeger:
    image: jaegertracing/all-in-one
    ports:
      - "16686:16686"
      - "4317:4317"

  seq:
    image: datalust/seq
    ports:
      - "5341:80"
      - "5342:5341"

Challenges Encountered

1. Prometheus Integration Issues

The OpenTelemetry.Exporter.Prometheus.AspNetCore package lacked a stable release. I switched to prometheus-net, which required refactoring my metrics collection approach but provided better stability.

2. Docker Networking Problems

Initially, Prometheus couldn't scrape my API metrics. The issue was in the Prometheus configuration, which needed to use host.docker.internal:5071 to access the API running on the host:

scrape_configs:
  - job_name: 'banking-integration'
    scrape_interval: 5s
    static_configs:
      - targets: ['host.docker.internal:5071']

3. Port Conflicts

I encountered port conflicts with Grafana's default port (3000). The solution was simple - changing it to 3001 in my docker-compose configuration.

The Final Result

My observability solution provides comprehensive monitoring capabilities:

Structured logs viewable in Seq with full context and correlation IDs
Distributed traces visualized in Jaeger showing complete request flows
Custom metrics dashboards in Grafana displaying:
- API health status
- HTTP request rates and status codes
- Request durations
- Banking service performance metrics
- Error rates
- Resource usage

Key Learnings

Start with health checks - They're the foundation of observability and help define what "healthy" means for your service.
Plan for correlations - Ensure correlation IDs flow through logs, traces, and metrics for effective troubleshooting.
Jaeger is invaluable for complex systems - Distributed tracing with Jaeger provides insights that logs and metrics alone cannot offer, especially for understanding request flows across service boundaries.
Use Docker Compose for local development - Makes it easy to spin up complex observability infrastructure locally.
Custom metrics matter - Generic metrics are helpful, but domain-specific metrics provide the most valuable insights.

Conclusion

Implementing a comprehensive observability solution for my .NET API has significantly improved my ability to monitor, diagnose, and optimize the core banking integration service. The combination of structured logging, distributed tracing with Jaeger, and metrics collection provides a complete picture of system behavior.

The most significant benefit is the reduced time to identify and resolve issues - what previously might have taken hours of debugging can now be spotted in minutes through my Grafana dashboards or trace analysis in Jaeger.

What observability tools are you using in your .NET projects? I'd love to hear about your experiences in the comments!