Early in my career, I spent a lot of time tuning two numbers.
The first number lived on the load-balancer. Amazon gets far more traffic than any one computer could handle, so behind the website there’s a fleet of them. And something has to decide which computer gets which incoming request, to balance the load across the computers. The first number that needed tuning was a setting for how many requests the load-balancer was allowed to send to any one computer at the same time.
The second number lived on the computers themselves. Each computer had a fixed number of threads. Think of a thread as a worker at a counter taking an order. It takes one request, does the work to answer it, hands back the result, and only then can it take the next order. A computer with twenty threads has twenty of these workers that can do jobs like rendering part of the website or calculating the taxes on your order.
Those two numbers had to marry.
If the load-balancer sent more requ…
