Who We Worked With
Our client was a Goa-based EdTech company — an online training platform that certifies travel agents, tour operators, and hospitality staff across India, with course content ranging from destination knowledge to booking-system workflows. A growing part of their product used machine learning models hosted on Amazon SageMaker AI to power adaptive assessments and personalized course recommendations for learners. We're keeping their identity anonymous at their request; we'll just call them "the client" here.
The Problem: Guessing Instead of Knowing
The client's enrollment numbers spiked hard every year in the run-up to peak travel-hiring season, when agencies and hotel groups pushed staff through certification in batches before the tourist season opened. Their ML-backed assessment engine had never been tested under anything close to that kind of concurrent load.
Their engineering team could tell us how the model behaved with a handful of requests in staging. What they couldn't tell us was what would happen when a few thousand learners hit the same endpoint within the same hour, which instance type would actually hold up, or where the first crack would appear. Every attempt to find out meant standing up test scripts by hand, coordinating a load run, and picking through logs afterward — a process that ate the better part of a week each time, and that nobody had the appetite to repeat often enough to actually trust.
They came to us wanting a way to answer that question on demand, not once a year under pressure.
What We Did: Automating the Load Test Instead of Running It by Hand
Rather than building a bespoke load-testing harness from scratch, we deployed OLAF (One Load Audit Framework) — Observe.ai's open-source load-testing tool for SageMaker endpoints — against the client's own inference setup.
The approach, in practice:
- Containerized the test harness. We ran OLAF as a Docker application, giving the client a repeatable, disposable test environment instead of a script someone had to babysit.
- Wired it into their SageMaker endpoint. Using the Boto3 SDK under the hood, OLAF invoked the client's actual model endpoint the same way their production application would — not a synthetic stand-in.
- Provisioned supporting infrastructure with CloudFormation and short-lived AWS STS credentials, so nothing about the test setup lingered as a standing security liability once a run was done.
- Drove concurrency with Locust through OLAF's UI, letting us dial up simulated learners in a controlled ramp instead of guessing at a single worst-case number.
- Watched latency, CPU, and memory in real time, rather than waiting for a post-mortem after the run finished.
We ran the client's assessment-serving model on a right-sized GPU instance and pushed concurrency up in stages, watching for the point where P95 latency started climbing away from P50 — the earliest honest signal that an instance type was running out of headroom, well before outright errors would show up.
What We Found
The test runs gave the client three things they'd never had in one place before: a P50/P95 latency curve against concurrent load, a throughput ceiling in requests per second for their current instance type, and a clear CPU/memory profile showing where the bottleneck actually sat. None of that needed a single line of custom test code — OLAF's dashboard and downloadable CSV reports covered it.
That let the client make an instance-sizing decision backed by numbers instead of a guess carried over from last year, ahead of the season where it would have mattered most.
Where We Left Things
We handed the client a repeatable process, not just a one-time report:
- A containerized OLAF setup they could re-run themselves before every future peak season, without looping us back in for each test.
- A documented baseline for their current model and instance type, so any future change — a model update, a new course feature calling the endpoint more often — has something concrete to be tested against.
- A short list of instance-sizing options, with the latency and throughput trade-offs of each laid out from the actual test data, not vendor spec sheets.
Why This Approach Worked
The client didn't need a bigger infrastructure team or a custom-built testing platform — they needed an honest answer to "will this hold up," delivered fast enough to act on before the season that mattered. Reaching for an existing, purpose-built open-source tool instead of building one from scratch meant the answer came back in hours instead of the better part of a week, and it's a process they now own themselves.
That's the same standard we hold every engagement to: solve the problem the client actually has, with the least new machinery required to do it properly.
Wondering how your ML infrastructure holds up under real load — before your customers find out for you? Metasips helps engineering teams pressure-test what they've already built, using proven tooling instead of reinventing it. Get in touch and let's find your breaking point on your terms.
This case study is based on the approach described in AWS's "Speed meets scale: Load testing SageMaker AI endpoints with Observe.ai's testing tool" — referenced here for readers who want the full technical walkthrough.