Session 2 of 3: turning an architecture into a delivery plan
Session 1 was about closing the architecture. That gave us a direction. It did not give us a working system.
The next challenge was building enough capacity for 15 Tbps while coordinating more than five service teams. Bandwidth was the obvious constraint, but it was not the only one. The network needed a direct path so the customer GPU fleet could reach Object Storage faster. Service Gateway capacity, web servers, storage servers, and the Oracle Database metadata layer all had to be ready together.
A weak plan would have treated those as separate workstreams and hoped they met in the middle.
We did the opposite.
Fixing a known storage problem first
There was one product gap we already understood. Listing objects in OCI Object Storage was slow for this use case.
The answer was to cache object names through the Inventory API. That gave the customer a faster way to inventory objects without repeatedly depending on slow object-list operations.
It was a specific service improvement for a specific customer need. But it also mattered operationally. A high-throughput storage instance is not useful if the surrounding workflows still make customers wait.
A detailed plan, not a status meeting
We created a detailed Confluence page with the work broken into clear tasks, milestones, owners, and dependencies.
Then we ran two cadences:
- A daily stand-up for the teams doing the work
- A weekly update for leadership
The daily meeting was for decisions and blockers. The leadership update was for progress, risks, and anything that needed help outside the delivery team.
When a blocker appeared, we treated it in a NOC-like manner: get the right people involved, make the decision, and move it as quickly as possible. A three-month delivery does not survive blockers sitting in someone’s queue for a week.
The first performance warning
By the time the major pieces were built, we had not proved that the system would deliver 15 Tbps. Burn-in and performance validation were still uncertain.
Early testing using WARP showed packet drops. The MTU was set to 1500, which was not right for the path we were building. We pulled network architects into the debugging effort and moved the MTU to 9100.
That fixed a real problem, but it also made the larger point clear: capacity on paper is not performance in production.
A network diagram can look complete. Server counts can look sufficient. The real answer only comes when customer GPUs push traffic through the whole path.
That was the next phase, and it was the hardest one.
Session 3 will cover the load test: validating from customer GPUs, finding the remaining performance issues, and deciding when the system was actually ready to hand over.