Azure Kubernetes Service (AKS) in Production: Lessons from Real Deployments

1. Resource Limits and Requests Misconfiguration

Problem: Pods get OOMKilled (Out of Memory Killed) or nodes run out of resources because you didn't set proper limits/requests.

I've seen production outages where a single pod consumed all node memory, causing the entire node to become unresponsive. Kubernetes couldn't evict the pod because there were no resource limits.

apiVersion: v1
kind: Pod
metadata:
  name: my-app
spec:
  containers:
  - name: app
    image: myapp:latest
    # No resources defined - dangerous in production!

apiVersion: v1
kind: Pod
metadata:
  name: my-app
spec:
  containers:
  - name: app
    image: myapp:latest
    resources:
      requests:
        memory: "256Mi"
        cpu: "250m"
      limits:
        memory: "512Mi"
        cpu: "500m"

Rule of thumb: Set requests to typical usage, limits to maximum expected usage. Monitor actual usage for 1-2 weeks, then adjust.

2. Cost Overruns from Over-Provisioning

Problem: Your AKS bill is 3x higher than expected because you're running nodes that are 80% idle.

Common scenario: You provision 5x Standard_D8s_v3 nodes (8 cores, 32GB RAM each) "to be safe", but your actual workload uses 30% of resources. That's $1,400/month wasted. This is a classic example of how technical debt accumulates when infrastructure decisions are made without proper monitoring and analysis.

3. Security Gaps: No Network Policies

Problem: By default, any pod can talk to any other pod in your cluster. If one pod gets compromised, attackers can access your database.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: backend-policy
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: backend
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: frontend
    ports:
    - protocol: TCP
      port: 8080

This policy ensures only frontend pods can reach backend pods on port 8080. Everything else is denied by default.

4. Networking Complexity: Address Space Planning

Problem: You run out of IP addresses because you chose a /24 subnet for a cluster that needs to scale to 100+ pods.

With Azure CNI, every pod gets an IP from your virtual network. If you have 10 nodes with max 30 pods each, that's 300 IPs needed. A /24 subnet only has 256 IPs.

5. Monitoring Blind Spots

Problem: Your app is slow, but you don't know why. No metrics, no logs, no visibility into what's happening inside the cluster.

In production, you need metrics for: pod CPU/memory usage, request latency, error rates, node health, persistent volume usage, network traffic. Without monitoring, you're flying blind.