Logistics SaaS for plant and transport logistics

Logistics SaaS: 50,000 new users, p95 response time cut by about 80 %

Industry
Logistics software (SaaS for plant and transport logistics)
Frybad's role
Site reliability engineering; platform ownership of the edge gateway, database, eventing and queueing; coaching the SRE teams
Stack
Kubernetes on AWS EKS, Horizontal Pod Autoscaler, edge gateway, relational database, eventing and queueing, ELK, Grafana

Starting point

The platform runs plant and transport logistics as a SaaS: time slots, shipment tracking and yard processes for shippers, retailers and logistics service providers. Then a major customer arrived, a drugstore retail chain with about 50,000 users, and with it further smaller customers and the logistics service providers that work with them on the platform. The user base grew from about 10,000 to about 60,000.

Under the new load, requests timed out. The causes sat in several places at once: resource allocation on Kubernetes (EKS) and horizontal autoscaling were tuned to the old traffic profile, the database lacked indexes, and monitoring showed system metrics but not the errors users actually saw at the edge.

What we did

  • Make the errors customers really hit visible. We analysed the edge gateway’s logs automatically: which requests timed out or failed, on which routes and for which customers. That became the ELK and Grafana dashboards the team uses day to day.
  • Re-size resources and autoscaling. Requests, limits and the Horizontal Pod Autoscaler thresholds were set from the measured load profile instead of assumptions from before the onboarding.
  • Database. Missing indexes were identified and added; database I/O was optimised, and rate limiting was introduced at the gateway so that a single tenant cannot slow the platform down for everyone.
  • Operations as a team. We moderated the platform dailies, built the release orchestration and coached the SRE teams. The SLA was turned into an error budget that releases and priorities are measured against.

What rate limiting at the gateway does: it caps requests per tenant or key before they reach the application. A tenant sending an unusual number of requests gets a targeted 429 instead of slowing every other tenant down.

What this means for your project

If your platform is about to onboard a large customer or already shows timeouts under load, that is the shape of Cloud-native DevOps & SRE. If the platform runs on your own hardware or should move there, start with Kubernetes platforms, built and run.

Result

Example from a client project.

The p95 response time fell by about 80 %, compared over one month before and one month after the changes.

The error rate was brought within the error budget defined by the SLA and kept there.

The onboarding ran with zero downtime; the platform carries five times the earlier traffic.

A dedicated platform SRE team now owns the edge gateway, database, eventing and queueing.

What we would do differently today

Load tests before the onboarding, not after. A major customer's traffic profile can be simulated in advance. Timeouts in production are the most expensive way to find resource limits.

Define the error budget before the first major customer. The SLA existed; the budget derived from it as a steering tool for releases only came with us.

Index review as a fixed step in the release process. Missing indexes show up under load, not in code review. A look at the slowest queries before each release is cheaper.

Similar situation?

Tell us what load is coming and what already creaks today. We will say where we would measure first.