How Cloudflare could save petabytes of cache storage with Zstandard and Pingora

Storage problems often disguise themselves as purchasing problems. A cache fills, capacity forecasts climb, and the natural response is to add disks. Cloudflare's cache transcoding experiment starts from a more interesting premise: before buying more storage, change what the cache stores.

The prototype uses Zstandard inside Pingora to encode eligible objects before they reach disk. Those objects remain compressed while they move through internal cache layers and data centers, then are decoded before the response continues through the normal client-facing path. Cloudflare reported that eligible objects were reduced to roughly one third of their original size. At the company's scale, that turns a representation detail into a potential petabyte-level capacity decision.

This is not a simple recommendation to compress everything. It is a case study in systems design: choose a narrow workload, place the transformation at the right boundary, preserve semantics, and prove that the CPU bill is smaller than the storage and bandwidth benefit.

What cache transcoding changes

A conventional cache stores the response body it received from the origin or upstream cache. If that response arrived without content encoding, an HTML document, JSON payload, JavaScript bundle, or stylesheet may be written to disk as plain bytes. The edge can compress it later for a browser, but the internal cache has already paid the full storage cost.

Cache transcoding inserts a private representation between the upstream response and the cache device:

Pingora identifies an eligible response during cache fill.
It encodes the body with Zstandard.
The cache records that the stored object uses the private encoding.
The object remains compact on disk and across supported internal transfers.
A reader decodes it before the ordinary HTTP response logic continues.

That last point matters. This is not the same as adding Content-Encoding: zstd to every public response. The internal storage format and the format negotiated with a browser are separate concerns. Keeping them separate gives the cache freedom to optimize its own data path without changing the public contract.

Why the object mix matters more than the headline ratio

Compression ratios sound universal, but workloads are not. Cloudflare's analysis found that text-like content such as HTML, JSON, CSS, and JavaScript represented 67.3 percent of requests but only 22.3 percent of bytes. About 71 percent of that text arrived without compression. That is a large pool of objects that are both frequently requested and structurally compressible.

Media told the opposite story. It represented 21.4 percent of requests and 63.3 percent of bytes, but formats such as images and video are commonly compressed already. Running a general-purpose text compressor over those bytes would consume CPU for little benefit and could occasionally make the result larger.

The lesson is broader than CDN caching. Capacity models should use the real distribution of content types, sizes, hit counts, and existing encodings. A global average can hide the segment that creates the opportunity and the segment that destroys it.

A useful first inventory looks like this:

content family     request share   byte share   uncompressed share   p50 size
HTML and JSON      ...             ...          ...                  ...
CSS and JS         ...             ...          ...                  ...
images             ...             ...          ...                  ...
video              ...             ...          ...                  ...
other binaries     ...             ...          ...                  ...

If uncompressed text is a meaningful share of stored bytes, transcoding deserves a benchmark. If the cache is dominated by JPEG, AVIF, MP4, ZIP, and other compressed formats, it probably does not.

Why Zstandard fits an internal cache format

An internal cache representation has different priorities from an archival format. A cache object is encoded once on fill but may be decoded many times on hits. Decompression speed is therefore unusually important. Compression ratio still matters, but the highest possible ratio is not automatically the best choice.

Cloudflare tested Zstandard at level 3. Its published measurements reported encoding around 4.31 nanoseconds per byte, about 232 MB/s, and decoding around 1.56 nanoseconds per byte, about 641 MB/s. On a controlled corpus, the compression ratio was 2.834 to 1. Those figures describe the prototype and test conditions, not a promise for every processor or payload, but they explain the choice.

Zstandard also has practical engineering advantages. It is mature, widely deployed, supports streaming, and provides explicit compression levels. A team can select a fast level for live traffic and measure the effect instead of treating compression as one fixed algorithmic cost.

The asymmetric workload suggests a clear optimization target:

fill path:  one encode, latency is important
hit path:   many decodes, latency is critical
disk path:  fewer bytes, capacity and I/O improve
network:    fewer internal bytes when the representation is preserved

An algorithm that saves a few more percent but doubles hit-path decode time could be a poor trade. The objective is system efficiency, not the best number in a standalone compression contest.

The eligibility gate is the real product

The compressor is only one component. The eligibility function decides whether the system is safe and economical.

Cloudflare's prototype targeted responses that were successful, currently unencoded, compressible, large enough to justify the work, and known to have a complete length. It excluded partial and range-oriented flows, existing encodings, unknown lengths, and binary types that were unlikely to benefit.

A simplified policy might look like this:

fn can_transcode(meta: &ResponseMeta) -> bool {
    meta.status == 200
        && meta.content_encoding.is_none()
        && meta.content_length.is_some_and(|n| n >= 4 * 1024)
        && meta.content_type.is_compressible_text()
        && !meta.is_range_response
        && !meta.is_slice_subrequest
        && !meta.is_partial
}

The 4 KiB threshold in Cloudflare's experiment is a good example of evidence-based filtering. Very small objects create fixed metadata and CPU overhead while contributing little capacity. Cloudflare found that excluding them removed only about 1 percent of otherwise eligible bytes. The system avoided a large population of low-value operations without giving up much storage benefit.

Production policy would need more nuance. Content type can be missing or incorrect. Transfer encoding can obscure length. Some responses are transformed by later filters. A safe implementation defaults to ineligible whenever the system cannot prove that the object fits the contract.

Why Pingora is the right control point

Pingora is Cloudflare's Rust-based framework for building programmable network services. In the cache transcoding design, it has access to response metadata and the streaming body while also participating in the cache lifecycle. That makes it a natural place to decide, encode, mark, and later decode the stored representation.

Placement matters because a transformation implemented too high in an application stack may not survive internal cache transfers. Implemented too low, it may lack enough HTTP context to handle validators, ranges, or content types correctly. The cache framework sits at the boundary where both representations are visible.

Streaming is essential. Buffering a multi-megabyte object in memory just to compress it would shift the capacity problem from disk to RAM and amplify latency under concurrency. A streaming encoder can consume body chunks, emit compressed chunks to storage, and maintain bounded memory.

The read path needs the inverse operation with the same discipline. It detects the private marker, streams bytes through a decoder, removes or translates private metadata, and presents the original logical body to the rest of the proxy.

Preserving compression across cache tiers

The storage win becomes more valuable when the representation remains compact through Tiered Cache. In a tiered architecture, lower cache layers can fetch from upper tiers rather than every location returning to the origin. If the object is decoded and re-encoded at every boundary, the system pays repeated CPU costs and increases internal bandwidth.

Preserving the private representation allows one successful fill to produce compact bytes that can be stored and transported through compatible internal layers. The decoding work is delayed until the point where the unencoded logical response is required.

This creates a protocol requirement inside the cache fleet. Every participating reader must understand the marker and format version. Mixed-version rollouts need an explicit compatibility plan. Common options include:

Writers emit the new format only after all readers can decode it.
The marker includes an encoding version and parameters.
Unsupported readers treat the object as a miss rather than serving corrupt bytes.
Rollback disables new writes while readers retain decode support until old entries expire.

That sequence is more important than the compression code itself. Distributed storage formats should be rolled out like protocols, not local implementation details.

The economics: CPU once, bytes saved repeatedly

Cloudflare modeled compression CPU at only a few percent for the eligible workload. The attractive shape comes from how costs repeat. Encoding happens once per cache fill. Storage savings persist for the lifetime of the object. Internal transfer savings may occur across tiers. Decoding repeats on hits, so it must remain fast.

A first-order model for one object is:

benefit = stored_bytes_saved
        + internal_transfer_bytes_saved
        + disk_io_bytes_saved

cost = encode_cpu_on_fill
     + decode_cpu_per_hit * hit_count
     + metadata_and_operational_cost

The model should be evaluated across the whole workload, not only the average object. A large JSON response with a high ratio and moderate hit count may be excellent. A tiny HTML fragment with a short lifetime may never repay the fixed work. An extremely hot object could make decode CPU dominate even though it compresses well.

Teams should measure by cohorts: size bucket, MIME family, cache tier, region, hit frequency, and object age. A single global ratio is useful for a headline but not for a rollout controller.

Correctness risks at the representation boundary

Changing stored bytes without changing logical content sounds straightforward. HTTP contains enough edge cases to make it dangerous.

Content length

The stored length and logical response length are different. The cache needs both or must be able to reconstruct the logical value. A private compressed length must never leak into a decoded public response.

Validators

ETags and other validators describe a representation. If an origin ETag is preserved, the system must ensure its public semantics still correspond to the decoded logical body. Internal compression metadata should not accidentally become part of public cache validation.

Range requests

A byte range refers to offsets in a representation. Random access into a compressed stream does not map trivially to offsets in the original body. Excluding ranges and partial responses is safer than inventing semantics during an early experiment.

Partial writes and corruption

A failed encoder must not leave an object that is marked complete. Writes should commit metadata only after the compressed stream finishes and its integrity checks pass. Readers should fail closed on truncated or invalid frames.

Double encoding

An object that is already privately encoded must not pass through the encoder again. Cloudflare described using a storage marker so the state travels with the object. The marker is a simple mechanism with a critical invariant.

Observability before optimization

A feature like this needs operational evidence at three levels: selection, performance, and correctness.

Selection metrics explain what the policy is doing:

objects considered and accepted
rejection reason by category
original and encoded bytes
compression ratio by MIME type and size bucket
percentage of cache bytes eligible

Performance metrics explain its cost:

encode and decode CPU time
first-byte and total response latency
encoder throughput and queueing
disk read and write bytes
internal cache-transfer bytes

Correctness metrics protect users:

decode failures
checksum or frame errors
unexpected marker versions
content-length mismatches
fallbacks to cache miss

Dashboards should compare enabled and control populations. A storage chart alone can make a broken feature look successful.

A rollout that can be reversed

The safest rollout begins in shadow mode. Evaluate eligibility and estimate saved bytes without changing the stored object. This validates workload assumptions and shows whether the policy selects the expected content.

Next, enable writes for a small traffic cohort while all readers already support decoding. Start with one content family and conservative size bounds. Measure hit latency, CPU, compression ratio, and errors against a control group. Increase exposure only when the joint budget remains healthy.

A practical progression is:

Instrument the existing cache object mix.
Run offline compression experiments on representative bodies.
Deploy decode support with no encoded writes.
Shadow the eligibility policy.
Enable encoded writes for a tiny cohort.
Expand by region, content family, and size bucket.
Keep a kill switch for new writes.
Retain decoder compatibility until encoded objects age out.

The rollback path is deliberately asymmetric. Stopping writers is immediate. Removing readers is delayed. That lets the cache drain the new format safely.

When this pattern belongs in your architecture

Cache transcoding is promising when storage is material, uncompressed text is common, and CPU has predictable headroom. It is especially interesting in multi-tier systems where compact objects can reduce both disk and internal network pressure.

It is less attractive when most bytes are already compressed, objects are too small or short-lived, cache hit rates are low, or the latency budget cannot tolerate decode work. Operational maturity matters too. A team without representation-versioning, fine-grained telemetry, and safe fleet rollouts may create more risk than capacity.

The design question is not, "Can Zstandard compress our objects?" It almost certainly can. The useful questions are:

Which objects create most of the recoverable bytes?
Where can one encoding operation benefit multiple storage and transfer steps?
How many times will each object be decoded?
What public semantics must remain unchanged?
Can every reader survive a mixed-version deployment?
Is the rollback path proven before the first write?

What engineering leaders should take from the experiment

Cloudflare's prototype is valuable even for teams that never build cache transcoding. It demonstrates a repeatable way to attack infrastructure cost.

First, inspect the physical representation behind an expensive logical service. Second, segment the workload instead of optimizing everything. Third, move the transformation to the boundary where context and lifecycle meet. Fourth, model recurring and one-time costs separately. Finally, treat format changes as compatibility protocols with observability and rollback.

The petabyte-scale headline comes from Cloudflare's size. The engineering insight is universal: before scaling the container, reconsider the bytes inside it.

Designing a representative compression experiment

Before touching a live cache, build a corpus from production observations. The corpus should preserve the dimensions that influence compression and cost: MIME type, original size, response frequency, cache lifetime, geographic distribution, and whether the origin already applies an encoding. Remove sensitive payloads or generate structurally equivalent samples when policy prevents retaining bodies.

Do not select only large, obviously compressible files. That produces an impressive ratio and a misleading business case. Use stratified sampling so small HTML responses, API JSON, generated JavaScript, configuration documents, and uncommon text types appear in the same proportions seen by the cache.

Run each sample through several Zstandard levels on the processors used in the fleet. Record compressed size, encode time, decode time, maximum memory, and output variance. Warm and cold measurements are both useful because cache services can be sensitive to instruction and data locality.

zstd -3 --no-progress sample.json -o sample.json.zst
zstd --test sample.json.zst
zstd -d --no-progress sample.json.zst -o decoded.json
cmp sample.json decoded.json

The round-trip comparison is non-negotiable. Ratio and throughput tests prove economics; byte-for-byte validation proves correctness. Extend the same principle to a streaming harness that feeds unusual chunk boundaries, empty final chunks, truncated frames, cancellations, and simulated storage failures.

Next, replay the measured object distribution through a capacity model. Apply actual cache lifetimes and hit counts. A body that compresses from 300 KiB to 100 KiB saves 200 KiB of resident storage, but only while it remains cached. A similar object retained for days can contribute much more than one evicted after minutes.

The experiment should output ranges, not one estimate. Model a conservative case using lower ratios and higher CPU, an expected case using medians, and a stress case for hot objects. Capacity planning improves when uncertainty is visible.

Internal encoding and browser compression are separate decisions

It is easy to confuse cache transcoding with HTTP response compression because both reduce byte counts. Their contracts are different.

Browser compression is negotiated using request headers such as Accept-Encoding and communicated through response metadata such as Content-Encoding and Vary. It changes the representation sent across the public connection. An edge might choose Brotli for one browser, gzip for an older client, and no compression for another response.

Internal cache transcoding is private. The cache chooses a storage representation optimized for disk and internal movement, then restores the logical body before ordinary public negotiation. The browser does not need to know which format was used on disk.

This separation avoids tying storage policy to the current client population. It also permits the best algorithm to differ at each layer. Zstandard may be attractive internally for fast decoding, while Brotli remains useful for client-facing static assets where network size has greater weight.

The separation does add work. If the cache decodes Zstandard and immediately encodes Brotli for a client, the CPU path contains two transformations. Implementations should measure whether a frequently requested client encoding deserves its own stored variant or a secondary encoded cache. That decision depends on variant explosion, hit rate, available capacity, and CPU.

There is no universally correct arrangement. The valuable architectural move is making each representation boundary explicit so its purpose, metadata, and cost can be reasoned about independently.

Failure injection for the write and read paths

Happy-path benchmarks will not expose the riskiest states. Cache transcoding changes data while it is being persisted, so failure injection belongs in the design phase.

Interrupt the encoder after every possible chunk boundary. Simulate a full disk, a connection reset from an upper cache tier, a process termination before metadata commit, an invalid marker, and a decoder that rejects the frame. Verify that the system either serves the original logical object or records a miss. It must never serve partial decoded data as a successful response.

Test mixed fleets explicitly. Send new-format objects to old readers and unknown future markers to current readers. The required behavior should be deterministic and visible in metrics. A cache miss is expensive but safe; silent interpretation is neither.

Finally, verify cancellation. If a downstream client disconnects during decoding, the decoder, buffers, and upstream cache stream should be released promptly. Saving disk bytes is not a win if abandoned decode tasks create memory or CPU leaks.

These tests turn a promising compression benchmark into a storage feature that operators can trust.

One final check belongs outside the cache service. Fetch a sample of enabled objects from a client test harness and compare headers and decoded bodies with a control path that bypasses transcoding. Include conditional requests, cache revalidation, stale serving, purge, and origin failure. Internal metrics can show that encoders and decoders agree with each other; an end-to-end comparison shows that the public HTTP contract still agrees with the origin. That distinction is what separates an internally consistent format from a correct user-facing system.

Sources and further reading

Let's build something together

We build fast, modern websites and applications using Next.js, React, WordPress, Rust, and more. If you have a project in mind or just want to talk through an idea, we'd love to hear from you.

Start a Project →

How Cloudflare could save petabytes of cache storage with Zstandard and Pingora