Observability and Monitoring Production Systems: 2025 Guide

It's 3 AM. Alerts wake you up: "System is down".

You check the status page - everything green. CPU and memory normal. Logs show only standard entries. But users are screaming they can't place orders.

Where is the problem? Without proper observability you waste precious minutes jumping between dashboards, searching for a needle in a haystack.

This is the difference between monitoring and observability. Monitoring tells you that the system has a problem. Observability tells you why - and leads you directly to the cause. In the world of microservices, distributed systems and cloud deployments, observability isn't a luxury. It's a survival necessity.

01Observability vs Monitoring - The Difference That Matters

Most companies think monitoring is observability. That's like thinking owning a thermometer means understanding medicine.

The difference is fundamental - and can save you hours of late-night debugging.

02Three Pillars of Observability

Observability is based on three data pillars: metrics, logs and distributed tracing.

Each pillar provides a different view of the system. Individually? Limited value. Together? Complete picture of what's happening in production.

{
  "timestamp": "2025-11-22T14:23:45.123Z",
  "level": "ERROR",
  "service": "order-service",
  "trace_id": "abc123xyz",
  "user_id": "user_456",
  "message": "Payment gateway timeout",
  "details": {
    "payment_method": "credit_card",
    "amount": 129.99,
    "gateway": "stripe",
    "timeout_ms": 5000
  }
}

03OpenTelemetry - The Observability Standard

For years each vendor had their own format. Prometheus - its own. Jaeger - different. DataDog - yet another.

Integration? Nightmare. Changing vendors? Rewriting all instrumentation. Then came OpenTelemetry - vendor-neutral standard supported by Microsoft, Google, AWS and the entire CNCF. It's the de facto industry standard for observability in 2025.

dotnet add package OpenTelemetry.Extensions.Hosting
dotnet add package OpenTelemetry.Instrumentation.AspNetCore
dotnet add package OpenTelemetry.Instrumentation.Http
dotnet add package OpenTelemetry.Instrumentation.SqlClient
dotnet add package OpenTelemetry.Exporter.Prometheus
dotnet add package OpenTelemetry.Exporter.Jaeger

using OpenTelemetry.Resources;
using OpenTelemetry.Trace;
using OpenTelemetry.Metrics;

var builder = WebApplication.CreateBuilder(args);

// Define service name and version
var serviceName = "order-api";
var serviceVersion = "1.0.0";

builder.Services.AddOpenTelemetry()
    .ConfigureResource(resource => resource
        .AddService(serviceName: serviceName, serviceVersion: serviceVersion))

    // Traces configuration
    .WithTracing(tracing => tracing
        .AddAspNetCoreInstrumentation(options =>
        {
            options.RecordException = true;
            options.Filter = (httpContext) =>
            {
                // Don't trace health checks
                return !httpContext.Request.Path.StartsWithSegments("/health");
            };
        })
        .AddHttpClientInstrumentation()
        .AddSqlClientInstrumentation(options =>
        {
            options.RecordException = true;
            options.SetDbStatementForText = true;
        })
        .AddSource("OrderService.*") // Custom traces
        .AddJaegerExporter(options =>
        {
            options.AgentHost = "jaeger";
            options.AgentPort = 6831;
        }))

    // Metrics configuration
    .WithMetrics(metrics => metrics
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        .AddRuntimeInstrumentation()
        .AddMeter("OrderService.*") // Custom metrics
        .AddPrometheusExporter());

var app = builder.Build();

// Expose Prometheus metrics endpoint
app.MapPrometheusScrapingEndpoint("/metrics");

app.Run();

using System.Diagnostics;

public class OrderService
{
    private static readonly ActivitySource ActivitySource =
        new("OrderService.Processing");

    public async Task<Order> ProcessOrderAsync(CreateOrderRequest request)
    {
        using var activity = ActivitySource.StartActivity("ProcessOrder");
        activity?.SetTag("order.amount", request.TotalAmount);
        activity?.SetTag("order.items", request.Items.Count);
        activity?.SetTag("user.id", request.UserId);

        try
        {
            // Validate inventory
            using (var inventoryActivity = ActivitySource.StartActivity("CheckInventory"))
            {
                await _inventoryService.ValidateStockAsync(request.Items);
                inventoryActivity?.SetTag("inventory.valid", true);
            }

            // Process payment
            using (var paymentActivity = ActivitySource.StartActivity("ProcessPayment"))
            {
                var payment = await _paymentService.ChargeAsync(
                    request.UserId,
                    request.TotalAmount
                );
                paymentActivity?.SetTag("payment.id", payment.Id);
                paymentActivity?.SetTag("payment.method", payment.Method);
            }

            // Create order
            var order = await _repository.CreateOrderAsync(request);
            activity?.SetTag("order.id", order.Id);

            return order;
        }
        catch (Exception ex)
        {
            activity?.SetStatus(ActivityStatusCode.Error, ex.Message);
            activity?.RecordException(ex);
            throw;
        }
    }
}

04Prometheus and Grafana - Metrics in Practice

Prometheus is the de facto standard for metrics in the Kubernetes and microservices ecosystem. Used by Google, Uber, SoundCloud, DigitalOcean and thousands of other companies.

Grafana is the visualization platform that turns raw metrics into readable dashboards. Together they create a powerful, open-source monitoring stack used worldwide.

# Add Prometheus Community Helm repo
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

# Install complete stack (Prometheus + Grafana + AlertManager + Node Exporter)
helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --create-namespace \
  --set prometheus.prometheusSpec.retention=30d \
  --set prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.resources.requests.storage=50Gi \
  --set grafana.adminPassword=admin123 \
  --set grafana.persistence.enabled=true \
  --set grafana.persistence.size=10Gi

# Port-forward Grafana (dev)
kubectl port-forward -n monitoring svc/kube-prometheus-stack-grafana 3000:80

# P95 latency last 5 minutes
histogram_quantile(0.95,
  rate(http_request_duration_seconds_bucket[5m])
)

# Requests per second
rate(http_requests_total[5m])

# Error rate (5xx responses)
rate(http_requests_total{status=~"5.."}[5m])
/
rate(http_requests_total[5m]) * 100

# CPU usage
100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# Memory usage
(1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100

05SLI, SLO, SLA - Measuring Reliability

You have metrics. You have dashboards. But how do you determine if your system is "good enough"?

This is where the SLI/SLO/SLA framework comes in - Google Site Reliability Engineering's methodology for objectively measuring and ensuring system reliability. It's the difference between "I feel the system is running OK" and "I know exactly what level of service I guarantee".

06Production Best Practices

ERROR: Payment failed for user 123

{
  "level": "ERROR",
  "timestamp": "2025-11-22T14:30:00Z",
  "message": "Payment failed",
  "user_id": "123",
  "payment_method": "credit_card",
  "amount": 99.99,
  "error_code": "insufficient_funds",
  "trace_id": "abc123xyz"
}

// Generate correlation ID at entry point (API Gateway)
app.Use(async (context, next) =>
{
    var correlationId = context.Request.Headers["X-Correlation-ID"].FirstOrDefault()
        ?? Guid.NewGuid().ToString();

    context.Response.Headers.Add("X-Correlation-ID", correlationId);

    // Add to all logs in this request
    using (LogContext.PushProperty("CorrelationId", correlationId))
    {
        await next();
    }
});

// Propagate to downstream services
httpClient.DefaultRequestHeaders.Add("X-Correlation-ID", correlationId);

07Frequently Asked Questions