Billing meters that survive service failure

Opening

A billing API can return a successful response while the usage event that should fund next month's invoice is already at risk of being lost.

The product action has been committed, the tenant sees a success state, and the application appears to be working. The missing meter may only become visible later — when finance reconciles Stripe against your ledger and a chunk of billable activity is simply gone.

I hit this while building Ledgerline, a multi-tenant usage-based billing platform. The system needed more than the latest balance. It needed a chronological meter trail that remained recoverable when Stripe, the network path, or a publishing worker became unavailable — without charging a tenant twice when delivery retried.

My responsibility was the metering path end to end: data model, Stripe Meter Events integration, operator dashboard, and the failure drills that proved recovery. This article is the reliability slice of that work.

A successful API call can still underbill

The failure window was narrow, but it was real. UsageService could record a billable action, commit the database transaction, and only then attempt to publish a Stripe meter event. The finance dashboard would reflect billed usage only after that publication succeeded.

That left a gap between two true statements: The tenant used the feature. The meter event was not recorded at Stripe.

If Stripe was slow or unavailable, the worker crashed, or the network failed after the commit, retrying the whole product API would risk repeating a side effect customers already saw. Ignoring the publication error would leave product state and billed usage out of sync.

This mattered more than a delayed notification. The meter existed to reconstruct commercial truth — who consumed what, when, and at which price point. Losing that event meant losing revenue quietly.

Preserve first, deliver later

The design became clearer once I separated two responsibilities: preserve the usage event and deliver it to Stripe.

Preservation had to share the business transaction. Delivery could happen later, outside the request path, with retry. Stripe availability should determine when the event is delivered, not whether it survives.

Commit the product change and its pending meter event together. Attempt delivery to Stripe only after both are safe.

Store the event in the same transaction

Services that generated billable activity kept a local outbox. In Ledgerline that lived beside the tenants that already mutated product state — API metering, seat changes, and overage writes.

The transaction shape was simple: write the business change, write the pending meter event, then commit both together. If either write failed, both rolled back. Stripe was not part of the atomic boundary.

The TypeScript below is simplified. The important property is the shared transaction, not the ORM API.

await db.transaction(async (tx) => {
  await tx.usage.insert({
    tenantId,
    metric: "api_calls",
    quantity: 1,
    occurredAt,
  });

  await tx.meterOutbox.insert({
    eventId: crypto.randomUUID(),
    tenantId,
    stripeCustomerId,
    eventName: "api_calls",
    payload: { quantity: 1, occurredAt },
    status: "pending",
  });
});

Publish outside the request path

After commit, a background MeterPublisher read pending rows, sent them to Stripe Meter Events, waited for acknowledgement, and marked the row published only after Stripe accepted the event.

If publication failed, the row stayed pending for a later retry. That pending row was not an error by itself. It was an observable intermediate state: the product transaction succeeded, but commercial delivery had not completed yet.

const pending = await outbox.getPending({ limit: 100 });

for (const row of pending) {
  try {
    await stripe.billing.meterEvents.create(
      {
        event_name: row.eventName,
        payload: {
          stripe_customer_id: row.stripeCustomerId,
          value: String(row.payload.quantity),
        },
        identifier: row.eventId,
      },
      { idempotencyKey: row.eventId },
    );

    await outbox.markPublished(row.eventId, new Date());
  } catch (error) {
    await outbox.recordFailure(row.eventId, String(error));
  }
}

Retry creates a double-charge problem

Retry was necessary, but it did not solve the full problem by itself.

A publisher could send an event successfully and then crash before marking the outbox row published. After restart, the row would still look pending, so the same usage would be published again. Redelivery could also happen at the worker or queue level.

I did not treat the path to Stripe as exactly once. The safer model was simpler: transport may deliver more than once. The final metered total must still count the usage once.

Four layers of idempotency

The duplicate problem was handled in layers because no single layer covered every replay path.

Layer 1 — Transactional outbox

The outbox preserved one retryable event alongside the business change. A retry reused the same stored event instead of minting a new meter for each attempt.

Layer 2 — Stable event identity

Each event carried a stable eventId. Retries reused that identity, which let Stripe and our own ledger distinguish a replay from new billable activity.

Layer 3 — Stripe idempotency keys

The publisher sent the stable identity as both Stripe's meter identifier and the request idempotencyKey. Stripe could reject or safely collapse repeated publication inside its idempotency window.

Layer 4 — Persistence constraint

The final guard lived in our own ledger. The meter log enforced uniqueness on event identity, because API-level deduplication windows are bounded and workers can still replay after they expire.

An application-level exists check was not enough under concurrent publishers. The database constraint had to remain authoritative.

• Example: CREATE UNIQUE INDEX ux_meter_log_event_id ON meter_log (event_id);

Test the failure path

A recovery mechanism should be tested while the failure is still present.

That meant observing pending outbox entries before recovery. Reaching zero pending rows at the end was not sufficient evidence unless the failed state had been visible first.

The Ledgerline recovery drills covered three scenarios: Stripe outage, publisher-to-Stripe network interruption, and a worker crash after pending rows had already been written.

In those drills, the Stripe-outage case recovered 24 pending entries, the network-interruption scenario recovered 12 events in 0.31 seconds, and the worker-crash scenario recovered 12 events in 0.29 seconds. All three ended with 0 pending entries, 0% duplicate metering, and 0% event loss in the documented results.

Verify integrity under normal load

Recovery testing was only one side. The meter path also needed to stay complete and duplicate-free when the system was operating normally.

The verification workload walked the core Ledgerline path: create tenant, attach Stripe customer, emit API usage, apply overage, close a billing window, and reconcile the operator dashboard against Stripe.

That run recorded 1,180 billing windows and 8,420 meter events, with a Meter Completeness Rate of 100%, a Duplicate Processing Rate of 0%, and a final Outbox Pending Rate of 0%. Those checks mattered because a recovery mechanism that only works during isolated failure drills is not enough for finance.

What this design does not solve

The outbox improved recoverability, but it did not remove trade-offs.

Eventual consistency

Product state can commit before the meter appears in Stripe. Readers — including the operator dashboard — need to tolerate that temporary gap.

Additional operational components

The design adds outbox tables, background publishers, retry logic, consumer idempotency, cleanup or archival work, and monitoring.

Polling delay

A shorter polling interval reduces delivery latency, but it also increases database activity.

Outbox growth

Published rows need a retention, archival, or deletion policy so the outbox does not become a permanent accumulation point.

Observability

Useful signals include pending event count, age of the oldest pending event, publication failure count, retry count, and duplicate-constraint violations.

Guarantee boundary

The event becomes recoverable only after the database transaction commits it to the outbox. A failure before commit should roll back both the product change and the event.

What changed in my understanding of reliability

This project changed one assumption I used to carry too casually: a successful API response means the operation is commercially complete.

A more accurate model is narrower. A successful response confirms only the synchronous part of the operation.

Reliability is not the absence of failure. It is the ability to preserve intent, recover incomplete work, and verify the result after failure has occurred.

• Was the product change committed?
• Was its meter event preserved?
• Can publication be retried?
• Can a retry charge the tenant only once?
• Can finance verify the final state?

Summary

The database transaction preserved the product change and the pending meter event together. The publisher retried delivery to Stripe independently, and layered idempotency prevented those retries from becoming double charges.

Request success and billing completion were different checkpoints. The meter trail survived partial failure because event preservation did not depend on immediate Stripe availability.

A successful response is one checkpoint. Recoverability — and honest invoices — begin after it.