Success Was the Incident

I ended the last essay with a promise that the caching and cost story behind SFDS deserved its own write-up. This is that story, and it starts where the last one ended: with adoption. Every team that migrated added pages, and every page added customers pulling components from our CDN. The line we had worked eighteen months to bend upward had a twin that bent upward with it: the delivery bill.

Our browser caching strategy at the time was a five-minute TTL. That was a defensible default in the early days, when consumers were few and the ability to push a fix and see it everywhere within minutes was worth almost any cost. As migration progressed it got exposed fast. Any customer whose visit outlasted five minutes was re-downloading files that had not changed, and most of the files never changed between releases at all.

The architecture multiplied the problem. Components pull in shared libraries and dependencies as they need them, which is good engineering and terrible arithmetic: a single page could fan out into hundreds of CDN requests per customer. Multiply by pages, by customers, and by a cache that forgets everything on a five-minute timer, and request volume climbed with every team we onboarded. Success was the incident.

The way it surfaced is worth recording. We track request volume through firewall metrics in Elastic, and that chart moved first: traffic climbing with every migrated team, well past what page counts alone would suggest. We took the finding to the finops team and they confirmed the same story from the cost side. The dashboards we had built for the platform itself tracked adoption, quality, and accessibility, and none of them caught it. The firewall did.

The first lever was already partly in place. The npm package was in production with our server-rendered apps, where bundling is mandatory anyway, and a bundled consumer gets tree shaking, version pinning, and no runtime CDN dependency at all. The cost findings prompted the next step: making npm consumption broadly available and actively encouraged, because every team that moved was a team whose customers stopped generating CDN requests entirely.

It does not fit everywhere, and pretending otherwise would have stalled the whole effort. Legacy stacks without modern build pipelines cannot bundle. Shared shells and pages that depend on live propagation need the CDN’s update model, which is the reason we offer it in the first place. Teams mid-migration were not going to re-platform their build to save us bandwidth. We took the npm wins available and accepted the real constraint: the CDN path serves a large share of the estate permanently, so the CDN path had to get cheap on its own merits.

The phrase “the cache” hides the structure of the problem. There are three caches in this system with three different owners: the browser cache that lives on customers’ devices, the edge cache the CDN operates, and the origin behind both. They expire for different reasons, they fail in different directions, and they need different levers. Splitting them apart was the design step that made everything after it tractable.

The edge tier went first because we control it directly. We raised the CDN to its maximum TTL and wired automated cache invalidation into the CI/CD pipeline, scoped to what a release actually changed. The edge cache stopped expiring on a timer and started expiring on intent. Deploys became the invalidation event, the origin stopped absorbing traffic it had already answered, and we kept the property the five-minute TTL had been protecting all along: when we ship, the new content goes out.

The browser tier was the interesting one, and where the volume actually lived. The majority of requests were never for the component entry files. They were for the layer underneath: shared libraries, dependencies, common chunks, the files components pull in as needed. Those files change on our schedule, at release time. Customers’ browsers were treating them as if they might change any minute, because a five-minute TTL says exactly that.

One constraint shaped the design. Consumers need a stable URL for each component they use. A team references files such as sf-button.js and sf-modal.js, and those URLs cannot move between releases, or every adopter has to touch its pages every time we ship. Everything those component files reference, though, is ours to restructure.

So I wrote a script into our Rollup build that keeps each component entry file stable and content-hashes the dependency files it references. A dependency’s filename changes exactly when its contents do, which makes the hashed file immutable by construction. If it exists, it is correct.

before                        after
------                        -----
sf-button.js                  sf-button.js              (component entry, stable)
sf-modal.js                   sf-modal.js               (component entry, stable)
shared-utils.js               shared-utils.f3a91c.js    (hashed)
base-styles.js                base-styles.7d20be.js     (hashed)
chunk-form-field.js           chunk-form-field.c54e12.js (hashed)

Cache metadata per file class does the rest. Hashed files carry a one-year TTL, which in practice means indefinite: the browser holds the file until we ship a change, the hash moves, and the new filename busts the cache on its own. No invalidation call, no timer, no coordination with anyone. Component entry files keep their stable URLs and a far more lenient TTL, tuned to what usage data and testing showed about a typical visit. The specific number stays internal, but the principle travels fine: a component entry file’s TTL is a product decision about how fast changes reach customers, and data should set it rather than a default.

What falls out of this is that each component entry file acts as a manifest for its own dependency graph. The component file keeps its stable URL and references the current hashes for the shared utilities, styles, and chunks it needs. Those hashed dependencies stay immutable and can be reused across component entry files. The build system promises that a filename identifies its contents. The cache metadata promises a TTL that matches each file’s real change frequency. Neither tier needs to know anything else about the other.

The final optimization had no code in it. With the engineering levers in place, our traffic became predictable, and predictable traffic is a negotiating asset. We partnered with our cloud provider’s finops organization and our internal cloud teams to move CDN spend off a pure pay-as-you-go model and onto a planned model. The optimized traffic made the commitment safe to size, and the commitment made the remaining traffic cheaper per request. Engineering work and procurement work compounding like that is rarer than it should be, mostly because the two conversations seldom happen in the same room.

The whole effort ran about a month end to end. Together the levers cut request traffic, and the costs that follow it, to roughly a third of where they started. Two thirds of what our CDN had been serving turned out to be the same bytes, re-downloaded on a five-minute timer.

The order mattered. npm adoption shrank the population of CDN consumers where it could. Edge TTLs and deploy-scoped invalidation made the tier we own efficient. Content hashing made the browser tier self-invalidating. And the pricing model captured the predictability the first three levers created. Each one made the next one stronger.

Reduced to a sentence: hash what can move, stabilize what cannot, and make every TTL tell the truth about how often its file actually changes. The longer version:

• Defaults are decisions. A five-minute TTL was harmless at a handful of consumers and a budget line at a hundred teams. Revisit defaults whenever adoption changes the math underneath them.
• Split the cache by owner before optimizing. Browser, edge, and origin expire for different reasons, and a lever that helps one tier can be irrelevant to another.
• Content hashing turns invalidation from coordination into arithmetic. The filename is the cache key, and the build system maintains it for free.
• Protect the stable component URLs consumers reference. The dependency files behind them can be immutable, and immutable files are the cheapest files there are.
• Take the finops meeting. Predictable traffic is negotiating power, and engineering teams almost never cash it in.

Most of this is one build script, some cache headers, and a pricing conversation. The gain came from running them in the right order.

Platform cost work is adoption stewardship. Every team that chose SFDS handed us a slice of their page performance and, indirectly, a slice of the company’s bill. A platform that gets more expensive per consumer as it succeeds has a design flaw, no matter how good the components are.

The caching architecture holds for the same reason the design system does. The cheap path and the correct path are the same path: browsers do the right thing by default, deploys invalidate exactly what they change, and the stable component URLs are the only promises we keep by hand.