Skip to content

feat(runway): dlq reconciler for merge topics - #459

Merged
behinddwalls merged 1 commit into
mainfrom
preetam/runway-dlq-reconciler
Jul 30, 2026
Merged

feat(runway): dlq reconciler for merge topics#459
behinddwalls merged 1 commit into
mainfrom
preetam/runway-dlq-reconciler

Conversation

@behinddwalls

@behinddwalls behinddwalls commented Jul 29, 2026

Copy link
Copy Markdown
Collaborator

Summary

Why?

SubmitQueue records in-flight merge work before publishing and then waits for exactly one MergeResult echoing its correlation id. Runway is stateless and the sole responder on that id, so every request must resolve to a result — or the client waits forever.

The primary controllers resolve what they can name: conflicts and invalid requests become a FAILED result, infrastructure faults are nacked for retry. But a fault that never recovers exhausts the retry budget and dead-letters. Nothing consumed those dead-letter topics, so the request produced no signal at all and the client's correlation id hung indefinitely.

What?

Adds runway/controller/dlq, a reconciler that subscribes to an inbound topic's _dlq queue and, for each dead-lettered MergeRequest, republishes a FAILED MergeResult echoing the correlation id to the corresponding signal topic. dlq.TopicKey derives the DLQ topic key from the primary one so the two stay in lockstep.

Unlike the SubmitQueue and Stovepipe DLQ reconcilers this one writes no entity state — Runway has none, and the signal is the resolution. A payload that cannot be decoded carries no correlation id and is dropped rather than retried forever.

Wires two instances in the server (one per inbound topic) on a dedicated consumer running under errs.AlwaysRetryableProcessor, so a transient publish failure retries indefinitely rather than dead-lettering the dead-letter. The DLQ consumer is started alongside the primary one and stopped with the same 30s drain on shutdown; both stop errors are joined into the exit status.

Test Plan

bazel test //runway/... — 5/5 pass, including new //runway/controller/dlq coverage for republish-on-dead-letter, the drop-undecodable-payload path, and publish-failure propagation
bazel build //service/runway/... — wiring compiles
make gazelle, make fmt

Stack

  1. @ feat(runway): dlq reconciler for merge topics #459
  2. feat(runway): git-backed merger with REBASE #460
  3. feat(runway): git merger SQUASH_REBASE and MERGE #461
  4. feat(runway): git merger PROMOTE #462
  5. feat(runway): wire the git merger into the server #463
  6. feat(runway): reject changes that disagree on provider #476
  7. docs: define the URI as the unit of change #477

@behinddwalls
behinddwalls force-pushed the preetam/runway-terminal-errors branch from 8ee23c0 to 133a045 Compare July 30, 2026 17:55
@behinddwalls
behinddwalls force-pushed the preetam/runway-dlq-reconciler branch from b358333 to d976bb9 Compare July 30, 2026 17:55
Base automatically changed from preetam/runway-terminal-errors to main July 30, 2026 17:58
## Summary

### Why?

SubmitQueue records in-flight merge work before publishing and then waits for exactly one `MergeResult` echoing its correlation id. Runway is stateless and the sole responder on that id, so every request must resolve to a result — or the client waits forever.

The primary controllers resolve what they can name: conflicts and invalid requests become a `FAILED` result, infrastructure faults are nacked for retry. But a fault that never recovers exhausts the retry budget and dead-letters. Nothing consumed those dead-letter topics, so the request produced no signal at all and the client's correlation id hung indefinitely.

### What?

Adds `runway/controller/dlq`, a reconciler that subscribes to an inbound topic's `_dlq` queue and, for each dead-lettered `MergeRequest`, republishes a `FAILED` `MergeResult` echoing the correlation id to the corresponding signal topic. `dlq.TopicKey` derives the DLQ topic key from the primary one so the two stay in lockstep.

Unlike the SubmitQueue and Stovepipe DLQ reconcilers this one writes no entity state — Runway has none, and the signal *is* the resolution. A payload that cannot be decoded carries no correlation id and is dropped rather than retried forever.

Wires two instances in the server (one per inbound topic) on a dedicated consumer running under `errs.AlwaysRetryableProcessor`, so a transient publish failure retries indefinitely rather than dead-lettering the dead-letter. The DLQ consumer is started alongside the primary one and stopped with the same 30s drain on shutdown; both stop errors are joined into the exit status.

## Test Plan

✅ `bazel test //runway/...` — 5/5 pass, including new `//runway/controller/dlq` coverage for republish-on-dead-letter, the drop-undecodable-payload path, and publish-failure propagation
✅ `bazel build //service/runway/...` — wiring compiles
✅ `make gazelle`, `make fmt`
@behinddwalls
behinddwalls force-pushed the preetam/runway-dlq-reconciler branch from d976bb9 to c766cf8 Compare July 30, 2026 18:05
@behinddwalls
behinddwalls merged commit dfcf52f into main Jul 30, 2026
15 checks passed
@behinddwalls
behinddwalls deleted the preetam/runway-dlq-reconciler branch July 30, 2026 20:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants