Guide

Reducing p95 latency on Kubernetes: read gateway logs, find indexes, set resources and HPA right

When a SaaS platform grows and 95th-percentile response times rise, the cause is almost always in one of four places: missing database indexes, wrongly set requests and limits, an autoscaler that scales too late or on the wrong metric, or a gateway that hides timeouts. This article shows in which order to look and how to measure every change.

It is written for teams that run Kubernetes on AWS or GCP and are about to take on a large customer.

1. Why p95 and not the average

The average hides the problem. If 94 of 100 requests answer in 80 ms and six in 4 seconds, the mean is about 315 ms and looks harmless. But the six users who wait four seconds or give up are exactly the ones who complain. The 95th percentile (p95) is the value below which 95 per cent of requests fall; in the example that is 4 seconds. p99 likewise, for the slowest one per cent. Both show what the slowest users experience.

When we took over platform SRE responsibility for a logistics SaaS, a customer with around 50,000 users was about to be onboarded, and the timeouts started during the preparation. The order in which we looked back then is the order of this article: first see what actually hits the users, then the database, then the resources, and the autoscaler last.

2. Measure first: an SLO and an error budget

Before you change anything, define what “good” means. A service level objective (SLO) is a commitment in numbers, for example: “95 per cent of API requests answer in under 500 ms, measured at the gateway, over 30 days.” The error budget follows from the SLO: the share of requests that may be slower or fail before the commitment is broken. If there is a service level agreement (SLA) with the customer, the SLO is derived from it and is stricter than the SLA.

The error budget is the steering instrument. If it is used up, the team stops feature releases and works on reliability; if it is well filled, riskier roll-outs are allowed. Without this number, every latency discussion remains a matter of opinion.

3. Gateway logs: what users actually experience

Application metrics show what the service measured. The gateway at the edge shows what the user got, including the requests that never reached the service because an upstream timeout struck first.

Have the gateway logs analysed automatically, not by hand. A small job that runs every hour is enough:

  • Group by route (without IDs in the path), status code and upstream.
  • Per group: count, p50, p95, p99 of the response time, share of 5xx and 504.
  • Flag deviations against the previous week.

The result is a list of the ten routes that burn the most error budget. In practice, rarely more than three account for most of it. Those three are the work list; everything else waits.

A typical find at this point: 504 responses from the gateway that do not appear in the application metrics at all, because the application is still processing the request while the gateway has already given up on it. The gateway timeout and the application timeout have to be aligned: the gateway waits longer than the application, so that the user sees an error message from the application and not an empty 504.

4. Database: missing indexes are the most common cause

A route that was fast at 1,000 users and gets slow at 50,000 very often has a query that reads a whole table. The table used to be small. Two tools find this in minutes:

pg_stat_statements (PostgreSQL) collects the call count and total time per query pattern. Sorted by total time, the most expensive query is at the top.

SELECT calls, round(total_exec_time) AS total_ms,
       round(mean_exec_time) AS mean_ms, left(query, 80) AS q
FROM pg_stat_statements
ORDER BY total_exec_time DESC
LIMIT 10;

EXPLAIN (ANALYZE, BUFFERS) shows for that query whether a sequential scan runs over a large table. If it says Seq Scan on a table with millions of rows and a filter on a column without an index, the case is clear.

Two rules when creating them:

  1. CREATE INDEX CONCURRENTLY, so that the table is not locked while the index is built. On a production database this is not an option but a must.
  2. Measure before and after, with the same query and EXPLAIN ANALYZE. An index that is not used only costs write time.

What an index does not solve: N+1 queries (one query per list item) and missing connection pools. Both also show up in pg_stat_statements, as very many very short calls. The fix then lies in the application or in a pooler such as PgBouncer.

5. Requests and limits: resource allocation on Kubernetes

Kubernetes schedules pods by their requests and caps them with limits. Wrong values create latency in three ways:

  • CPU limit too low: the container is throttled (CPU throttling) even though the node has capacity. The metric container_cpu_cfs_throttled_periods_total shows it. For latency-critical services, CPU limits are often counterproductive; set requests correctly and leave the limit out or choose it generously.
  • Memory limit too low: the container is killed with OOM, requests break off, and the restart costs seconds. Visible as OOMKilled in the pod events.
  • Requests too high: the scheduler finds no room, new pods stay Pending, and the autoscaler scales into nothing. Node pools and requests have to fit together.

The right values come from measurement: kubectl top or the cluster’s metrics over a week, with the p95 of actual consumption as the request. Starting values from a template are to be replaced after the first load test.

6. HPA: when and on what to scale

The Horizontal Pod Autoscaler (HPA) increases the number of pods when a metric exceeds a target value. The default, CPU utilisation, is the wrong metric for many services: a service that waits for the database has low CPU and long response times all the same. If the HPA does not scale because CPU sits at 30 per cent, latency stays high.

Better: a metric that reflects the bottleneck. Requests per second per pod, queue length, or the p95 response time directly via a metrics adapter (Prometheus Adapter or KEDA). Plus three settings that are often missing:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  minReplicas: 3
  maxReplicas: 30
  metrics:
    - type: Pods
      pods:
        metric:
          name: http_requests_per_second
        target:
          type: AverageValue
          averageValue: '50'
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
        - type: Percent
          value: 100
          periodSeconds: 30
    scaleDown:
      stabilizationWindowSeconds: 300
  • minReplicas high enough that a node failure does not create a bottleneck.
  • behavior.scaleUp fast, scaleDown slow. An autoscaler that scales down immediately after a load peak creates the next peak itself.
  • Readiness probes that only turn green once the pod can really serve requests; otherwise a cold pod gets load straight away and answers slowly.

The HPA scales pods, not nodes. Without a cluster autoscaler or Karpenter, new pods stay Pending when the node pool is full. Test both together before the big customer arrives.

7. The onboarding as a load test

An onboarding with tens of thousands of users can be planned, and that is exactly why it is the best load test you will get. How to go about it:

  1. Derive the expected load per route from the customer’s figures (users, peak hours, typical actions).
  2. Run this load against an environment with production-sized data, with a load tool that reports p95 and error rate per route.
  3. Work on the three slowest routes following sections 4 to 6, and run it again.
  4. Accompany the onboarding day with a dashboard that shows error budget consumption in real time, and a person who can throttle at the gateway if needed.

Rate limiting at the gateway is not a sign of weakness here but the protection of existing customers: a new tenant that generates ten thousand requests per second during its import must not slow down everyone else.

8. Dashboards the team uses

One dashboard per service with exactly four rows: requests per second, p50/p95/p99, error rate, saturation (CPU throttling, memory, connection pool). Above them, one line with the month’s error budget. Everything else belongs in detail views.

In the project mentioned, we built the dashboards with ELK for log analysis and Grafana for the metrics, ran the platform dailies and set up a dedicated platform SRE team; in the end, p95 response times were about 80 % lower, and the error rate fitted within the error budget from the SLA. The number matters less than the order: first the three most expensive routes, first the database, then the resources, then the autoscaler.

9. Checklist

Measure:

  • SLO per service in writing; error budget calculated; dashboard shows consumption.
  • Gateway logs analysed automatically: top 10 routes by p95 and 5xx.
  • Gateway timeout longer than the application timeout.

Database:

  • pg_stat_statements active; top 10 by total time checked.
  • EXPLAIN ANALYZE for each of these queries; seq scans on large tables removed.
  • Indexes created with CONCURRENTLY, effect measured.
  • N+1 patterns and connection pool checked.

Resources:

  • Requests from measured p95 consumption; CPU throttling checked; no OOMKilled.
  • Node pool fits the requests; no Pending pods under load.

Autoscaling:

  • HPA metric reflects the bottleneck, not just CPU.
  • minReplicas, fast scaleUp, slow scaleDown, readiness probes.
  • Cluster autoscaler or Karpenter tested.

Onboarding:

  • Load test run with the customer’s figures; three slowest routes worked on.
  • Rate limiting configured at the gateway per tenant.
  • Onboarding day accompanied with a dashboard and a way to intervene.

10. Frequently asked questions

Should I add more pods or more nodes first? Neither, before you know what is slow. A missing index does not get faster with more pods; it only creates more parallel scans on the database.

How do I find out whether the database or the application is the bottleneck? Compare the time the application spends in the database (from tracing or the driver metrics) with the total response time. If the database share is over half, start there.

Do I need tracing? For the first three routes, logs and pg_stat_statements are enough. Distributed tracing pays off once a request passes through more than three services and nobody can say any more where the time goes.

How often should the error budget be reviewed? Daily in the platform daily, as one number. Weekly in the retrospective, with the routes that used it up.

Where this leads

We run our own hardware in a data centre in Munich, and our clients’ platforms on AWS and GCP. How we offer SRE work like the one described here, from analysing the gateway to building a platform SRE team, is described under DevOps & SRE for AWS and GCP.

Example from a project: Logistics SaaS: onboarding a customer with 50,000 users.

How to test changes like these in an environment of their own before the release: Release environments on Kubernetes with Crossplane and Argo Workflows.

German version: p95-Latenz auf Kubernetes senken: Gateway-Logs lesen, Indizes finden, Ressourcen und HPA richtig setzen.

Sources

All read on 20 Oct 2026.

A big onboarding coming up?

Tell us what load you expect and where things are already tight. In the intro call we go through your gateway figures and your three most expensive routes.