Performance Testing and Optimization - Comprehensive Guide 2025

k6 - Modern Developer-Friendly Tool

k6 is an open-source load testing tool from Grafana Labs. JavaScript/TypeScript scripting, minimal footprint, great CI/CD integration. Best choice for modern DevOps teams.

// k6 load test example
import http from 'k6/http';
import { check, sleep } from 'k6';

export const options = {
  stages: [
    { duration: '2m', target: 100 },  // ramp up to 100 users
    { duration: '5m', target: 100 },  // stay at 100 users
    { duration: '2m', target: 200 },  // spike to 200 users
    { duration: '5m', target: 200 },  // stay at 200 users
    { duration: '2m', target: 0 },    // ramp down to 0
  ],
  thresholds: {
    http_req_duration: ['p(95)<500', 'p(99)<1000'], // 95% < 500ms, 99% < 1s
    http_req_failed: ['rate<0.01'],  // error rate < 1%
  },
};

export default function () {
  const res = http.get('https://api.example.com/products');

  check(res, {
    'status is 200': (r) => r.status === 200,
    'response time < 500ms': (r) => r.timings.duration < 500,
  });

  sleep(1);
}

Advantages: JavaScript scripting (easy for web devs), minimal resource usage, built-in metrics, Grafana integration. Disadvantages: No GUI (code only), younger community vs JMeter.

Apache JMeter - Industry Standard

JMeter is a mature open-source tool from Apache Foundation. GUI-based test creation, huge community, support for legacy protocols (SOAP, FTP, LDAP). Ideal for enterprise with complex requirements.

Configuration: Thread Groups (users), Samplers (HTTP requests), Listeners (results visualization), Assertions (validation), Timers (think time). XML-based test plans versioned in Git.

Advantages: GUI for non-coders, plugins ecosystem, distributed testing, wide protocol support. Disadvantages: Heavy resource consumption, XML complexity, slow UI.

Gatling - High-Performance Scala Tool

Gatling is an open-source tool written in Scala. Asynchronous non-blocking architecture, DSL for test scenarios, excellent reporting. Best for high-load testing (10k+ concurrent users).

// Gatling Scala DSL example
import io.gatling.core.Predef._
import io.gatling.http.Predef._
import scala.concurrent.duration._

class ApiLoadTest extends Simulation {

  val httpProtocol = http
    .baseUrl("https://api.example.com")
    .acceptHeader("application/json")

  val scn = scenario("API Load Test")
    .exec(
      http("Get Products")
        .get("/products")
        .check(status.is(200))
        .check(responseTimeInMillis.lte(500))
    )
    .pause(1)

  setUp(
    scn.inject(
      rampUsers(100) during (2.minutes),
      constantUsersPerSec(50) during (5.minutes)
    )
  ).protocols(httpProtocol)
   .assertions(
     global.responseTime.percentile3.lt(1000),
     global.failedRequests.percent.lt(1)
   )
}

Advantages: Highest throughput, reactive architecture, beautiful HTML reports, efficient resource usage. Disadvantages: Scala learning curve, smaller community vs JMeter.

Database indexing strategy

Indexes are the most important database optimization. Missing index on WHERE/JOIN columns = full table scan instead of fast lookup. Difference? 5ms vs 5000ms for a million-row table. That's a thousand-fold difference in application performance.

-- PostgreSQL: Find missing indexes
EXPLAIN ANALYZE
SELECT * FROM orders
WHERE customer_id = 123 AND status = 'active';
-- Output: Seq Scan on orders (cost=0.00..25000.00)  <- BAD!

-- Add composite index
CREATE INDEX idx_orders_customer_status
ON orders (customer_id, status);

-- After adding index
EXPLAIN ANALYZE
SELECT * FROM orders
WHERE customer_id = 123 AND status = 'active';
-- Output: Index Scan using idx_orders_customer_status (cost=0.42..8.44)  <- GOOD!

-- Index column order matters!
-- Most selective column first (customer_id before status)
-- Index on (A, B) supports queries on A or (A, B), but not B alone

Eliminating N+1 Queries

N+1 problem is a classic ORM trap. Instead of 1 query you execute 101 - one for the list, then one for each item. Entity Framework, Hibernate, TypeORM without eager loading do this automatically. Result? Your application makes 100 times more queries than it should.

// ❌ BAD: N+1 queries (1 + 100 = 101 queries)
const orders = await db.orders.findMany(); // 1 query
for (const order of orders) {
  const customer = await db.customers.findUnique({
    where: { id: order.customerId }
  }); // N queries (100x!)
  console.log(customer.name);
}

// ✅ GOOD: Eager loading (2 queries or 1 JOIN query)
const orders = await db.orders.findMany({
  include: { customer: true } // Eager load relationship
});
orders.forEach(order => {
  console.log(order.customer.name);
});

// ✅ ALTERNATIVE: DataLoader pattern (batching + caching)
const customerLoader = new DataLoader(async (ids) => {
  return await db.customers.findMany({ where: { id: { in: ids } } });
});
const orders = await db.orders.findMany();
const customers = await Promise.all(
  orders.map(o => customerLoader.load(o.customerId))
);

Connection Pooling

Creating a new database connection is expensive. TCP handshake, authentication, memory allocation - all this takes time. Connection pool reuses connections instead of creating and destroying them with each query. PgBouncer for PostgreSQL, HikariCP for Java - the standard in production environments.

// Node.js pg connection pool configuration
const { Pool } = require('pg');

const pool = new Pool({
  host: 'postgres.example.com',
  database: 'myapp',
  user: 'apiuser',
  password: process.env.DB_PASSWORD,
  max: 20,              // Max connections in pool
  min: 5,               // Min idle connections
  idleTimeoutMillis: 30000,    // Close idle connections after 30s
  connectionTimeoutMillis: 2000, // Timeout getting connection
  maxUses: 7500,        // Recycle connection after 7500 uses
});

// Use pool.query() instead of client.connect()
app.get('/api/orders', async (req, res) => {
  const result = await pool.query('SELECT * FROM orders LIMIT 100');
  res.json(result.rows);
});

// PgBouncer config (external pooler before PostgreSQL)
// pool_mode = transaction  (best for microservices)
// max_client_conn = 1000   (frontend connections)
// default_pool_size = 25   (backend connections per database)

Query Optimization & Denormalization

Sometimes a normalized database schema is not optimal for read-heavy applications. Denormalization, materialized views, read-only replicas - you trade consistency for application performance. This is a conscious trade-off that can speed up your application 10-100 times.

• • Materialized Views - precomputed query results refreshed periodically
• • Denormalization - duplicate data to avoid JOINs (product.category_name instead of JOIN categories)
• • Read Replicas - read queries on replicas, writes on primary (PostgreSQL streaming replication)
• • Partitioning - horizontal partition large tables by date/range (orders partitioned by month)
• • Computed Columns - store calculated values (total_price = quantity * unit_price)

Cache-Aside Pattern

Most popular caching pattern: application checks cache before database query. Cache miss → query DB → save to cache with TTL.

// Cache-aside implementation with Redis
async function getProduct(productId) {
  // 1. Try cache first
  const cacheKey = `product:${productId}`;
  const cached = await redis.get(cacheKey);

  if (cached) {
    console.log('Cache HIT');
    return JSON.parse(cached);
  }

  // 2. Cache miss - query database
  console.log('Cache MISS - querying DB');
  const product = await db.products.findUnique({
    where: { id: productId }
  });

  // 3. Set cache with TTL (5 minutes)
  await redis.setex(cacheKey, 300, JSON.stringify(product));

  return product;
}

// Cache invalidation on update
async function updateProduct(productId, data) {
  const product = await db.products.update({
    where: { id: productId },
    data: data
  });

  // Invalidate cache
  await redis.del(`product:${productId}`);

  return product;
}

CDN Caching - performance at the network edge

CDN (Content Delivery Network) caches static assets (images, CSS, JS, fonts) in data centers close to users. Cloudflare, Azure CDN, AWS CloudFront reduce latency from 200ms to 20ms for users worldwide. It's like having a copy of your site in every city.

• • Static assets - long TTL (1 year), immutable files with hash in filename
• • API responses - short TTL (1-5 min), Cache-Control headers, purge on update
• • Image optimization - CDN automatic format conversion (WebP, AVIF), resizing, compression
• • Edge computing - Cloudflare Workers, AWS Lambda@Edge - compute at edge

Cache Invalidation Strategies

"There are only two hard things in Computer Science: cache invalidation and naming things" - Phil Karlton. Stale cache = wrong data shown to users.

• • TTL-based - cache expires after fixed time (300s). Simple, eventual consistency
• • Event-based - invalidate on data change (update/delete triggers cache delete)
• • Tag-based - associate cache entries with tags, invalidate all by tag (product:123 tagged with category:electronics)
• • Write-through - update cache synchronously during write (consistency guarantee, slower writes)

LCP - Largest Contentful Paint (target: <2.5s)

LCP measures loading performance - time until largest visible element renders. Usually hero image or main content block. Slow LCP = users bounce.

• • Image optimization - WebP/AVIF format, srcset responsive images, lazy loading (loading="lazy")
• • Preload critical resources - <link rel="preload" as="image" href="hero.webp">
• • Server-side rendering - Next.js SSR delivers HTML fast, no client-side fetch wait
• • CDN delivery - serve images/fonts from edge locations near users
• • Remove render-blocking resources - async/defer scripts, inline critical CSS

INP - Interaction to Next Paint (target: <200ms)

INP (replaced FID in 2024) measures responsiveness - time from user interaction to visual response. Long tasks (>50ms) blocking main thread = poor INP.

• • Code splitting - dynamic imports, load code on-demand (React.lazy, Next.js dynamic)
• • Minimize JavaScript - tree shaking, remove unused code, bundle size budget
• • Web Workers - offload heavy computation from main thread (data processing, image manipulation)
• • Debounce/throttle - limit high-frequency events (scroll, resize, input)
• • React optimization - useMemo, useCallback, React.memo, virtualization (react-window)

CLS - Cumulative Layout Shift (target: <0.1)

CLS measures visual stability - how much content jumps during loading. Layout shifts frustrate users - clicking button → ad loads → button moves → wrong click.

• • Image dimensions - always width/height attributes (reserve space before load)
• • Font loading - font-display: swap, preload fonts, system fonts fallback
• • Ad/embed containers - fixed height for dynamic content (ads, embeds, iframes)
• • Avoid inserting content - no content injected above existing (banners, notifications)
• • CSS contain - contain: layout for isolated components (prevents reflow propagation)