Averages hide the thing you actually care about. In a cluster of three or four nodes, one slow disk or one noisy neighbour is enough to move p99 by an order of magnitude while p50 stays flat.
What has helped most is measuring the tail directly and keeping the number of hops in the hot path small. Every extra hop adds a queue, and queues are where tails come from.
The other half is making the slow path boring: bounded retries, a deadline that is enforced rather than hoped for, and a way to shed load before the queue grows without bound.