Building High-Speed Systems – Performance, Latency & Throughput

Across every high-scale system I've worked on, one truth consistently holds: "users never complain that a system is "too fast." They notice, often subconsciously, when the interface lags, a request takes longer than it should, or the system shows signs of strain under load.

This is why understanding Performance, Latency, and Throughput is foundational to building high-speed systems.

These three terms are often used interchangeably in meetings, but in practice they represent different constraints and engineering trade-offs.

1. Understanding Performance

Performance describes how efficiently a system uses compute, network, memory, and storage resources to complete work. It is not merely "fast." Instead, performance is a balance of:

- how much work the system can do,
- how quickly it can do it,
- and how consistently it delivers results under varying load.

A system that responds quickly at low load but collapses under moderate traffic is not performing well. Similarly, a system that handles large throughput but consumes excessive CPU or memory is not efficient.

True performance comes from eliminating unnecessary work, optimizing hot paths, and respecting the fundamentals of hardware — CPU caches, disk I/O costs, network hops, and memory access patterns.

Example: Optimizing a Product Search API

Imagine you run an e-commerce platform with a Search API. Initially, each search request does the following:

- Performs a database query scanning millions of rows
- Sorts results in memory
- Joins multiple tables to fetch product metadata
- Sends the response back to the user

At low traffic, this might take 150 ms, which looks fine. But at high load, CPU usage spikes, queries slow down, and response time jumps to 1–2 seconds.

To improve performance, you might:

- Add a search index (e.g., Elasticsearch) so queries become O(log n) instead of scanning everything.
- Cache the top 1000 products in memory to eliminate redundant computation
- Move sorting into the database or index engine
- Reduce unnecessary fields in the response (less serialization work)
- Avoid repeated joins by precomputing product metadata

After these optimizations:

- The API now responds in 40 ms under normal load
- Under 10× traffic, it still delivers consistent responses
- CPU usage drops by ~60%
- The system requires fewer servers, reducing cost

This improvement did not come from buying faster hardware. It came from eliminating unnecessary work and optimizing the hot path — the portion of the code executed most often.

2. Understanding Latency

Latency is the time it takes to complete a single request from start to finish. Even a system with enormous computing power can have unacceptable latency if operations wait on slow I/O, locks, GC pauses, or overloaded dependencies.

Response Time = Latency + Processing Time

Latency is rarely uniform. Some requests are fast, some are slower, and a few are extremely slow. Percentiles help us understand the distribution of latency instead of relying on averages (which hide outliers).

What Percentiles Mean?

p50 latency: 50% of requests are faster than this value, and 50% are slower. Often called the median.

p90 latency: 90% of requests are faster, and only 10% are slower.
p95 latency: 95% of requests are faster; the slowest 5% take longer.

p99 latency: 99% of requests are faster; the worst 1% are extremely slow.

Percentiles tell you how your system behaves not only normally but also under stress or edge conditions.

Across many teams, we treat p95 latency as the customer-experience metric, and p99 latency as the fire-alarm metric. The former tells us how fast the system feels for most users, while the latter reveals how the system behaves under stress or rare worst-case conditions.

A system that performs well on average but collapses at the tail latencies cannot scale gracefully.

Latency Example

The following pseudocode highlights a common source of high latency in real-world systems — a blocking database call placed directly on the request path.

public User getUser(String id) {
    // Adds ~120ms latency under load
    return database.query("SELECT * FROM users WHERE id = ?", id);
}

At low traffic, this call may feel harmless, but under load it introduces significant waiting time. Every request is forced to pause until the database returns its result. If the DB is even mildly congested—due to slow disks, locking, or increased queue depth—the 120 ms delay can easily stretch into hundreds of milliseconds.

Since this function sits on a hot path, the added latency compounds, slowing down every upstream service and ultimately degrading the end-user experience.

To optimize this, engineers typically redesign the flow so the request does not depend on an expensive synchronous query. This can be done by serving frequently accessed records from an in-memory cache, reducing round-trips entirely.

In situations where multiple lookups are required, batching queries into a single request dramatically reduces overhead.

Some systems replace blocking calls with asynchronous I/O, allowing work to continue while waiting for data.

In cases where read patterns are predictable, data may be denormalized or precomputed so the service can fetch everything it needs in one lightweight operation.

Each of these approaches reduces the time spent waiting on the database, allowing the system to deliver consistently lower latency under load.

Latency vs Response Time

Although people often use the terms interchangeably, they describe different aspects of performance.

Latency is the time spent waiting for an operation to occur.
Response time is the total time a system takes to complete a request, including all delays and processing.

In short:

Latency = waiting time
Response time = latency + processing time + any additional overhead

Example: Imagine a user requests their profile from a service.

The request reaches the server.
The server must fetch data from a database.
The database takes 40 ms before it even starts returning bytes — this is latency.
The service processes the data, renders a JSON response, applies security checks, and sends the result back.
Total time from request → response: 85 ms — this is response time.

So in this example:

Database latency: 40 ms
Total response time: 85 ms

Latency is just one component of total response time.

3. Understanding Throughput

While latency describes the experience of a single request, throughput refers to the total volume of work a system can process per unit of time.

A system with high throughput is able to handle a large number of operations—such as requests served per second, messages consumed from a queue, or records written to storage—without collapsing under load.

In practice, engineers increase throughput by adding more concurrency—spinning up additional threads, workers, or replicas so the system can process many tasks in parallel rather than waiting on slow operations.

However, this increase is not free. As concurrency grows, so does contention—multiple workers now compete for CPU, memory, locks, or I/O bandwidth. When contention rises, latency often rises with it. This tension between throughput and latency is the core balancing act of performance engineering.

Mathematically, we describe throughput as:

Throughput = Total Work / Time Taken

A system may show impressive throughput numbers and still deliver a poor user experience. Consider a queue processing service that consumes 1,000 messages per second. On paper, this looks efficient.

But if each message spends two seconds in the queue before being picked up, the end-to-end latency for the workload is still high. Users or downstream systems will observe the delay, regardless of how fast the worker consumes items once it gets to them.

This highlights a critical point: throughput without corresponding attention to latency can mask real performance problems.

4. Performance vs Latency vs Throughput

These three concepts are tightly interdependent:

Sometimes you optimize for one at the cost of the other. For example:

1. Increasing throughput via batching increases latency for individual requests.
2. Reducing latency by serving from cache reduces durability and increases memory pressure.

One of the most important formulas I apply as an engineer is:

Little's Law: L = λ × W

Where:
L = Number of items in system
λ (lambda) = arrival rate (throughput)
W = response time (latency)

This law predicts queue buildup, load behavior, and system collapse points. When your latency increases, queue depth rises, throughput falls, and users experience cascading slowdowns.

5. Designing High-Speed Systems

After years of building and debugging systems, I've learned a few truths:

1. You cannot improve what you cannot measure: If you don't track metrics like p95/p99 latency, RPS, CPU, or queue depth, you're guessing, not optimizing. Good performance starts with visibility—measure first, tune second.

2. Latency is the most user-visible metric: Users don't see your throughput; they only feel how long their request takes. Even a powerful system feels "slow" if latency spikes.

3. The fastest code is the code that doesn't run: Skipping unnecessary work often produces bigger gains than micro-optimizations. Eliminate steps, reduce calls, and simplify logic before trying to "speed up" code.

4. Bottlenecks move as throughput increases: When your system grows 10×, the slowest part shifts—today's optimization may not matter tomorrow. Performance tuning is an ongoing process, not a one-time fix.

5. Caching is a superpower—but a dangerous one: Caching can make systems incredibly fast, but bad invalidation or cache stampedes can cause outages. Every mature system eventually learns to handle cache failures and fallback logic.

6. Performance problems are usually architecture problems, not developer mistakes: Slow systems often result from design choices—chatty services, blocking I/O, wrong data models—not poorly written code.

Fixing structure beats tweaking individual lines. A well-architected system makes slow code visible and easy to improve.