The Bot That Opened Merge Requests: Push vs Pull for Config, Credentials, and License Distribution

At 04:12 the bot opened its three-hundred-and-fortieth merge request of the night. Each one was identical in shape: a single-file diff replacing license.jwt with a freshly signed copy, a templated description, an auto-merge label. By morning, 314 had merged and deployed. Eleven were stuck behind failing CI on unrelated main-branch breakage. Eight hit repos whose branch protection had changed and now required a human approval nobody was awake to give. Six went to repositories that had been archived, where the bot’s push triggered a webhook that paged an on-call engineer for a service that no longer existed. And one — the one that mattered — failed because the bot’s access token had been scoped out of that repo during a security review three weeks earlier, so the service ran on its old license until the license expired on a Saturday and the service refused to boot.

The system that produced this night was not badly built. It was built by good engineers following the strongest current in modern infrastructure: everything through git, everything reviewed, everything deployed by pipeline. A license renews, so a bot commits the renewal, so every consumer’s normal deploy machinery ships it. GitOps for artifacts. It demos beautifully with three consumer repos. It decays combinatorially with three hundred.

This post is about that architectural fork: when a renewable artifact — certificate, license, key set, configuration blob — must reach N consumers on a schedule, do you push it into each consumer’s world, or do consumers pull it from yours? The industry has run this experiment at enormous scale in several independent domains, and the results are strikingly consistent. We will do the availability math, take the bot’s credential seriously as an attack surface, work through jitter and thundering herds, and end with an honest account of what still belongs in the push column — because some things genuinely do.

Two topologies, precisely stated

Push: the producer holds credentials for every consumer’s boundary and delivers renewed artifacts through it — a bot that commits to consumer repos, a deploy job that writes into consumer buckets, an agent that SSHes artifacts onto consumer hosts. Delivery is initiated by the producer, at renewal time, once per consumer.

Pull: each consumer holds a reference to the producer — a URL, plus a public key to verify what it fetches — and refreshes on its own schedule: fetch daily (jittered), verify offline, atomically swap the local copy, and keep serving the cached copy if the fetch fails, within a defined grace window.

The models differ along five axes that the rest of this post walks in turn: who holds credentials for whom, whose availability matters when, how failures distribute, how load distributes, and where human judgment sits. A useful reflex before we start: notice that the flagship “GitOps” tools are themselves pull systems. Argo CD and Flux do not have a CI job that pushes manifests into clusters; an agent inside each cluster pulls desired state from git and reconciles, and the OpenGitOps principles codify this — state is “pulled” by software agents that continuously reconcile toward it¹. The push-bot pattern borrows GitOps aesthetics while inverting its actual data flow.

The precedents, because none of this is hypothetical

ACME and cert-manager: pull, industrialized. The Web PKI is the largest renewable-artifact distribution system in existence, and it is pull all the way down. Under ACME (RFC 8555, March 2019), the certificate consumer initiates: it proves control of its identifiers, requests issuance, and downloads the result². cert-manager packages the loop for Kubernetes — each cluster runs a controller that watches certificate lifetimes and re-initiates issuance when a configurable renewBefore threshold is reached, by default renewing when roughly two-thirds of the lifetime has elapsed³. Let’s Encrypt does not hold a credential for any of the hundreds of millions of sites it serves; every one of them holds an account key and comes to the CA. Imagine the alternative for one second — a CA-operated bot with commit rights to the deployment repo of every website on earth — and you have imagined a single credential whose compromise ends the Web PKI. Nobody would build that. The push-bot for licenses is that design at smaller scale.

ARI: even the schedule is pulled. The one thing a CA legitimately wants to tell consumers is “renew earlier than planned” (impending revocation, shortened lifetimes). The IETF’s answer, ACME Renewal Information (RFC 9773, June 2025), is instructive: the CA does not push a notification — it publishes, per certificate, a suggested renewal window that clients poll alongside their normal loop, letting the CA both trigger early renewal and spread renewal load by handing different clients different windows⁴. Let’s Encrypt shipped ARI and recommends clients adopt it precisely because it turns emergency revocation events and load spreading into ordinary pull-schedule adjustments⁵. Even the “push-shaped” requirement got solved with pull.

JWKS: pull with cache is the whole design. Every OpenID Connect deployment on earth rotates signing keys without pushing anything anywhere. The issuer publishes its keys at a jwks_uri; relying parties cache the key set and re-fetch — on TTL, or on encountering a token signed by an unfamiliar kid, exactly as OpenID Connect Core’s key-rotation section prescribes⁶. Millions of relying parties, continuous rotation, no bot, no credential from issuer to consumer, and outage tolerance for free: a relying party with a cached JWKS validates tokens happily while the issuer’s metadata endpoint is down.

Renovate and Dependabot: push, deliberately. The strongest real-world push-bots are the dependency updaters. Renovate and Dependabot open pull requests into thousands of repos to bump dependency versions⁷⁸. And they are correct to push — which is what makes them the perfect control group. A dependency bump is not a self-verifiable artifact; it is a proposed code change whose safety is a judgment call requiring the consumer’s tests and often a human. The PR is not an unfortunate delivery mechanism, it is the product: a reviewable diff parked in the one place reviews happen. Note also what these bots relentlessly accumulate: merge-conflict handling, per-repo config dialects, rate-limit management, stale-PR grooming — an entire product category dedicated to the operational weight of writing into other people’s repositories. That weight is intrinsic to push, and a license-renewal bot inherits all of it while producing diffs no human will ever meaningfully review.

The availability math

Here is where pull’s advantage stops being aesthetic and becomes arithmetic.

Pull with cached fallback. Let the artifact be valid for V days, let consumers fetch daily, and let the producer additionally honor a grace window G during which a consumer may keep using its cached artifact past nominal refresh (or, equivalently, let artifacts be issued with V comfortably longer than the refresh cadence — the certificate pattern, where V = 90 days and renewal starts at day 60). A consumer breaks only if the producer is unreachable for that consumer continuously for the entire slack period — every daily attempt failing for, say, 30 consecutive days. If daily-fetch attempts fail independently with probability p, the chance a given consumer exhausts 30 days of slack is p³⁰: even a dismal p = 0.1 (the producer is down for your fetch one day in ten) gives 10⁻³⁰ — the failure mode has been pushed out of the realm of plausible events and into the realm of “the producer ceased to exist,” which no distribution scheme survives. The real risk consolidates into one correlated scenario — a producer outage longer than the slack — which is exactly the quantity you provision: slack of days-to-weeks means the producer can suffer any realistic outage, planned or not, and no consumer notices. The producer’s uptime requirement decouples from consumer availability almost entirely.

Push through git and CI. Now count what must all be true, per consumer, in the renewal window: the bot is up; the bot’s token for this repo is valid now; the repo accepts the push (branch protection, required reviews, merge queues — all drifting independently under 340 different teams’ control); the consumer’s CI is green on unrelated code at that moment; the deploy pipeline completes. Call it five links at 99% each and delivery-on-first-attempt is 0.99⁵ ≈ 95% — per consumer, per renewal. At 340 consumers, that is ~17 failures every cycle, each requiring investigation, and the failure list is a fresh random sample each time. The night described in the opening was not bad luck; it was the expected value.

The deep difference is when the dependency chain must hold. Pull front-loads its requirements (consumer needs the producer reachable sometime in a long window) and holds a local asset through trouble. Push demands the whole chain be healthy at the producer’s chosen moment, holds nothing at the consumer beyond what last deployed, and — the subtle one — couples artifact freshness to the consumer’s code health: a repo with a broken main branch can’t take a license renewal, which is an absurd sentence to have to say out loud.

One more asymmetry, this time in pull’s favor on the monitoring side, because push advocates raise a fair point here: “at least the bot knows who failed.” True — push failures are producer-visible by default, pull failures are consumer-local. But the fix for pull is one endpoint: consumers report artifact age/expiry as a metric (or the producer simply logs who has fetched — it terminates every fetch, after all), and the producer alerts on consumers approaching slack exhaustion. Observability is a feature you add to pull in an afternoon; the delivery-chain fragility of push is not removable at any price, because it is the topology.

Blast radius, or: what mass events look like in each model

Renewable-artifact systems have a characteristic disaster: the mass event, where many consumers must change at once — a producer signing-key compromise, a systemic mis-issuance, a format migration.

The Web PKI has run this drill in production. In February 2020, Let’s Encrypt discovered a CAA-rechecking bug affecting 3,048,289 live certificates and announced revocation on days’ notice; the ecosystem strain was severe enough that they ultimately let most affected certificates age out rather than break ~30% of holders who hadn’t managed to replace in time⁹. Two lessons. First: mass events are real and you must design for them, not hope. Second: even under a forced mass event, pull’s failure mode was graceful — sites kept serving on their existing certificates while the long tail caught up. The mechanism absorbed an emergency by degrading a security property temporarily and visibly, rather than by going down. ARI exists in part so the next such event is a polled renewal-window update instead of a blog post and a deadline⁴⁵. And the pressure is structural now: with CA/Browser Forum ballot SC-081v3 (April 2025) stepping maximum TLS certificate lifetimes down to 47 days by March 2029, renewal frequency rises ~8× and only fully automated pull survives at all¹⁰.

Now run a mass event through the push-bot: 340 merge requests in one burst — every consumer’s CI fires simultaneously (the producer has just DoS’d its own organization’s build farm), the failure tail is 17+ repos dark for reasons distributed across 340 teams’ local configuration, and rollback means force-pushing reverts through the same fragile chain. Worse is the bad-artifact case: a push-bot that signs and ships a corrupt artifact has deployed it everywhere by the time the first consumer notices, at producer speed, synchronously. Pull consumers, fetching at jittered times over a day, give you a canary period for free: the first few consumers to pick up a bad artifact fail (loudly, locally, while serving their cached previous artifact if you built the swap right — verify then swap, never swap then verify), and you pull the artifact before the herd arrives. Push distributes instantly, which is precisely the property you do not want for anything that can be wrong.

The credential surface of a git-writing bot

Security architecture reduces to a blunt question: who can write to what? Put the two models side by side.

Push-bot: the producer system holds a credential — commonly one token, because per-repo tokens at N = 340 are an operational program of their own — with write access to every consumer repository. Those repos are not config buckets; they are the source of truth that CI builds and ships to production. The bot’s token is therefore a code-execution credential for the entire consumer fleet, sitting in the environment of a service whose actual job (signing licenses) needed no such power. Compromise the license service — or its CI, or one dependency in its build (the supply-chain literature of the last half-decade is a catalog of exactly this move) — and you don’t get a signing key; you get commit access to everything, wearing the identity of a bot whose commits are, by design, auto-merged and boring. The push-bot is a watering hole you built for yourself. Every mitigation exists (fine-grained tokens, mandatory review of bot PRs, commit signing with verification) and every mitigation is either an O(N) standing cost or a reintroduction of the human latency the bot existed to remove.

Pull consumers hold… a public key. Invert the flow and the credential story nearly vanishes. Each consumer stores the producer’s URL and an Ed25519 public key baked into its deployment. The fetch endpoint can require a per-consumer read token if artifacts are confidential (a read credential, per consumer, low blast radius) or can be public if they’re merely signed. The producer holds no credentials for any consumer, has no inbound path to any consumer, and cannot execute code anywhere but its own house. The signing key — now the only crown jewel — lives in one place, ideally an HSM/KMS, guarding a signature operation rather than a fleet-wide write path.

Offline verification is what makes this safe rather than merely convenient. Because the artifact is signed, the consumer trusts the artifact, not the channel: a compromised CDN, a hijacked DNS name, a malicious insider at the storage layer can at worst serve garbage (rejected at verify, cached copy retained) or serve stale (bounded by the grace window plus expiry embedded in the signed artifact — which is why licenses and certs carry their own exp, so replay of an old artifact buys an attacker only the time until its embedded expiry). This is the JWKS/X.509 trust model, and it composes with the availability story: the same signature that defeats the malicious CDN lets you serve the cached copy through the honest outage.

Thundering herds, jitter, and the schedule as a contract

Pull’s classic failure mode is self-synchronization: ship “fetch daily” as cron: 0 0 * * * and you have built a distributed system whose entire demand arrives in the same second — the thundering herd. Three disciplines dissolve it:

Jitter at birth. Each consumer picks a persistent random offset within the cadence (fetch at a uniformly random minute of the day, chosen once and kept — or derived as hash(consumer_id) mod 86400, which is stable across restarts and needs no state). Load flattens to N/86400 requests per second. Marc Brooker’s canonical analysis of backoff-and-jitter is about retries, but its core observation — that independent clients given the same deterministic schedule are not independent, and randomness is what buys you the independence you assumed — is exactly this problem¹¹.

Jitter on retry. Failed fetches retry with exponential backoff plus full jitter, or a producer blip converts the flat schedule back into synchronized waves of exactly the clients that failed together¹¹.

Producer-steered windows. The ARI trick generalizes: let the producer embed a suggested next-fetch window in each response (per-consumer, randomized server-side). Now the producer can spread load, accelerate rotation before a planned migration, or slow an overloaded tier — all without any push channel, because the schedule itself rides the pull⁴.

Here is the whole consumer, compressed — jitter, verify-then-swap, cached fallback, grace accounting — in ~40 lines of Python:

import hashlib, json, os, random, time, urllib.request
from nacl.signing import VerifyKey   # PyNaCl, Ed25519

PRODUCER_URL = "https://artifacts.example.com/license/v1/tenant-42"
PUBKEY       = VerifyKey(bytes.fromhex(os.environ["PRODUCER_PUBKEY_HEX"]))
CACHE_PATH   = "/var/lib/myapp/license.signed"
CADENCE      = 86_400                      # nominal: daily
GRACE        = 14 * 86_400                 # serve stale up to 14 days

def stable_jitter(consumer_id: str, cadence: int) -> int:
    h = hashlib.sha256(consumer_id.encode()).digest()
    return int.from_bytes(h[:4], "big") % cadence   # same offset every boot

def fetch_and_verify() -> dict:
    raw = urllib.request.urlopen(PRODUCER_URL, timeout=10).read()
    blob = json.loads(raw)
    payload = bytes.fromhex(blob["payload_hex"])
    PUBKEY.verify(payload, bytes.fromhex(blob["sig_hex"]))   # throws if bad
    artifact = json.loads(payload)
    if artifact["exp"] < time.time():
        raise ValueError("producer served an expired artifact")
    tmp = CACHE_PATH + ".tmp"                # verify THEN swap, atomically
    with open(tmp, "wb") as f:
        f.write(raw)
    os.replace(tmp, CACHE_PATH)
    return artifact

def load_cached() -> dict | None:
    try:
        blob = json.loads(open(CACHE_PATH, "rb").read())
        payload = bytes.fromhex(blob["payload_hex"])
        PUBKEY.verify(payload, bytes.fromhex(blob["sig_hex"]))
        return json.loads(payload)
    except Exception:
        return None

def refresh_loop(consumer_id: str):
    offset = stable_jitter(consumer_id, CADENCE)
    while True:
        time.sleep(seconds_until(offset))            # jittered daily slot
        for attempt in range(6):                     # backoff + full jitter
            try:
                fetch_and_verify(); break
            except Exception:
                time.sleep(random.uniform(0, min(3600, 30 * 2**attempt)))
        art = load_cached()
        if art is None or art["exp"] + GRACE < time.time():
            alert("license unrecoverable: cache empty or beyond grace")

Every load-bearing property of the pull model is visible in that snippet: the producer appears only as a URL and a public key; a network failure costs nothing until cache-plus-grace is exhausted; a malicious response costs nothing ever; and the fleet’s aggregate load is flat by construction. The equivalent push system is not a snippet — it is a bot, a token-management regime, N repos’ worth of merge policy, and a runbook.

What actually belongs in each column

The preceding sections read like a prosecution of push, so let me state the defense honestly, because push is the right model for a real class of things. The discriminating questions:

1. Does a human need to judge this change? If the change’s safety depends on consumer-side context — a dependency bump, a config schema migration, generated client code — then it is a proposal, and a PR into the consumer’s repo is the correct artifact. This is Renovate’s column, and nothing in this post argues against it⁷⁸.
2. Is the artifact self-verifiable? If the consumer can check it mechanically (signature valid, not expired, schema-conformant), human review adds nothing but latency and rubber-stamping — pull it. Certificates, licenses, JWKS, revocation lists, feature-flag payloads, model weights with checksums.
3. Does freshness have a deadline measured in renewals? Anything that expires belongs in pull, because push couples the deadline to the delivery chain’s worst link. The opening incident is this rule violated.
4. Can the consumer run the loop? Pull presumes a consumer that executes code on a schedule. A static artifact bucket, a partner who receives files, a firmware image on a device that only boots occasionally — sometimes there is no loop to own, and push (or push-then-poll) is what remains.
5. Is N small and stable? At N = 3 with one owning team, the push-bot’s chain is short and its failures are legible. The analysis in this post is about what happens as N grows and consumer boundaries drift. Push does not scale organizationally; that is its actual limit.

And the hybrid that resolves most remaining cases: push the signal, pull the data. A webhook or message saying “artifact updated, fetch now” costs nothing to lose (the daily pull is the backstop), carries no payload to verify, and needs no write access to anything — it just advances the schedule. This is the standard maturation path: systems that begin as pure push tend to evolve into notify-then-pull, and systems that begin as pure pull add notification for latency. Both converge on the same place: the transfer of record is a verified pull; push is at most an optimization hint.

Closing: the bot was a symptom

The 04:12 bot was built because git-plus-CI was the only deployment path the organization trusted, so every moving thing was forced through it — including things that are not code changes and never wanted review. That is the actual error: mistaking “our delivery pipeline runs through git” for “therefore everything must be delivered through git.” A license renewal is not a change to a system; it is the system continuing to be itself. Routing it through merge requests gave it all of code’s friction and none of code’s safeguards, and attached a fleet-wide write credential to a service that only ever needed to sign.

The pull loop — jittered fetch, offline verification, cached fallback, grace window — is not merely the alternative; it is the pattern that every renewable-artifact ecosystem that operates at real scale has independently converged on. ACME made the consumer initiate²; cert-manager put the loop in every cluster³; ARI showed that even the producer’s need to steer can ride the pull channel⁴; JWKS showed that key rotation needs nothing more than a cache and a re-fetch rule⁶; and the 2020 Let’s Encrypt event showed the model bending gracefully under a genuine emergency instead of shattering⁹. The convergence is the evidence.

So the judgment: push proposals, pull artifacts. If a machine can verify it, no human should deliver it; if it expires, its freshness must not depend on anyone else’s CI being green; and if your renewal system includes a credential that can write to every consumer, you have not automated distribution — you have centralized compromise. The bot that opens merge requests at 04:12 is not infrastructure. It is a finding.

References