# Merge Queue Academy > Educational documentation about merge queues—systems that serialize and validate pull requests before they land on main. Learn how merge queues keep your main branch stable while enabling high-velocity development. ## What is a Merge Queue? Source: /introduction/what-is-a-merge-queue/ A merge queue sits between "PR approved" and "PR merged." It validates changes before they land on main. Simple merge queues serialize merges. Modern ones do more: **test PRs against future main state**, **batch PRs together**, **run dedicated CI**, and **parallelize testing across independent code paths**. ## The Core Problem Without a merge queue, two PRs can each pass CI individually, yet break main when combined. Each PR is tested against an outdated snapshot of main—not against each other. This happens constantly: renamed modules, deleted functions, changed signatures, conflicting config changes. No merge conflict, but broken code. :::tip[Want the full breakdown?] See [What Happens Without a Merge Queue](/decision/failure-scenarios/) for detailed failure scenarios and their cascading costs. ::: ## The Paradigm Shift: CI Before Merge, Not After A merge queue tests each PR against its **actual merge target**—including all PRs ahead of it. The key insight: **test the PR against what main will look like after the merge, not what it looked like when the PR was created.** Traditional workflows run final CI **after** merging to main: [diagram] The problem: by the time you discover the failure, **main is already broken**. The damage cascades: 1. **Deployments stop** - you can't ship until main is fixed 2. **All developers are blocked** - no one can merge until the fix lands 3. **New PRs inherit the breakage** - any PR based on broken main will also fail CI 4. **Everyone must rebase** - after the fix, every in-flight PR needs to rebase 5. **The cycle can repeat** - rebasing and re-merging might break main again A merge queue flips this: final CI runs **before** the merge: [diagram] The queue tests the PR **as if it were already merged** with current main. Pass? Merge succeeds. Fail? PR rejected, main untouched. This means: - **Main stays green** — broken code never lands - **Deploy anytime** — main is always releasable - **Failures stay contained** — only the PR author fixes their code A merge queue makes "broken main" impossible by construction. ## Core Capabilities Modern merge queues offer more than serialization: - **[Two-Step CI](/features/two-step-ci/)** — Separate PR validation from queue validation - **[Batching](/features/batching/)** — Test multiple PRs together - **[Speculative Checks](/features/speculative-merging/)** — Parallelize testing by assuming success - **[Parallel Queues](/features/parallel-queues/)** — Independent queues for non-conflicting changes - **[Freshness Policies](/features/freshness-policies/)** — Balance safety and throughput - **[Priority Management](/features/priority-management/)** — Let urgent PRs jump the queue ## Summary A merge queue does more than automate merges. It: 1. **Keeps main stable** by testing PRs against their actual merge target 2. **Validates before merging** — CI runs before code lands, not after 3. **Increases throughput** through batching, speculation, and parallelism 4. **Scales to large codebases** with selective testing 5. **Handles exceptions** with priority queues Next: [How merge queues work](/introduction/how-merge-queues-work/) covers the mechanics. --- ## How Merge Queues Work Source: /introduction/how-merge-queues-work/ This page explains the internal mechanics of a merge queue — how PRs flow through it, how test branches are created, and how the queue coordinates with your CI system. ## The PR Lifecycle A pull request goes through several stages in a merge queue: [diagram] ### Stage 1: Entering the Queue When a PR is added to the merge queue, the queue: 1. **Validates eligibility** — Is the PR approved? Are required checks passing? 2. **Assigns position** — Based on priority and arrival time 3. **Records the base** — Captures the current state of the target branch [diagram] ### Stage 2: Creating the Test Branch The merge queue creates a **temporary branch** that represents "what main will look like after this PR merges." This is the key insight that makes merge queues work. [diagram] The test branch contains: - All commits from `main` - All commits from PRs ahead in the queue (if using [speculative checks](/features/speculative-merging/)) - The PR's changes, merged in ### Stage 3: Running CI The merge queue triggers CI on the test branch. This is often called "queue CI" to distinguish it from the CI that runs on the PR branch itself. [diagram] The queue monitors CI status and waits for all required checks to complete. ### Stage 4: Merging or Failing **If CI passes:** [diagram] **If CI fails:** [diagram] ## Queue Dynamics Understanding how the queue manages multiple PRs is crucial. ### Queue State At any moment, the queue contains PRs in various states: ``` Queue State: ┌─────────────────────────────────────────────┐ │ Position │ PR │ Status │ Base │ ├─────────────────────────────────────────────┤ │ 1 │ #101 │ Testing │ abc123 │ │ 2 │ #102 │ Testing │ +#101 │ │ 3 │ #103 │ Pending │ +#102 │ │ 4 │ #104 │ Pending │ +#103 │ └─────────────────────────────────────────────┘ ``` Each PR's "base" includes all PRs ahead of it — this is how the queue ensures PRs are tested against the future state of main. ### What Happens When a PR Fails Mid-Queue When PR #102 fails, the queue must re-evaluate everything behind it: [diagram] PR #103's test is now invalid — it was tested against a world where #102 existed, but #102 is gone. The queue automatically re-tests #103 with a new base. ## Merge Strategies When a PR passes CI, the merge queue must integrate it into main. There are several strategies: ### Merge Commit Creates a merge commit, preserving the full branch history. [diagram] **Pros:** Full history, easy to revert **Cons:** Cluttered history with merge commits ### Squash and Merge Combines all PR commits into a single commit. [diagram] **Pros:** Clean linear history **Cons:** Loses individual commit granularity ### Rebase and Merge Replays PR commits on top of main. [diagram] **Pros:** Linear history, preserves commits **Cons:** Changes commit hashes ### Fast-Forward Only possible when the test branch is already based on current main. Moves the main pointer forward. **Pros:** No extra commits, preserves exact SHA tested in queue **Cons:** Not always possible ## Coordination with CI The merge queue needs tight integration with your CI system. ### Required Checks You configure which CI checks must pass before a PR can merge: ```yaml required_checks: - build - test - lint ``` ### CI Triggers The queue must be able to: 1. **Trigger CI** on the test branch 2. **Receive status updates** when checks complete 3. **Cancel CI** if a PR is removed from the queue [diagram] ### Handling Flaky Tests If CI fails due to a flaky test: - Some queues offer **automatic retry** (1-2 times) - Some require manual re-queue - Some track flake rates and adjust behavior ## The Test Branch Lifecycle Test branches are temporary. Here's their lifecycle: 1. **Created** when PR starts testing 2. **Updated** if base changes (PR ahead merges/fails) 3. **Used for merge** if CI passes 4. **Deleted** after merge or failure Most merge queues clean up test branches automatically. You'll see branches like: - `mq/main/pr-123` - `gh-readonly-queue/main/pr-123-abc1234` - `mergify/merge-queue/main/pr-123` ## Race Conditions and Edge Cases ### The "ABA" Problem What if main changes while CI is running? ``` 1. PR #1 starts testing against main@A 2. Someone pushes directly to main (now main@B) 3. PR #1's CI passes 4. Should PR #1 merge? ``` Different queues handle this differently: - **Strict:** Require re-test against new main - **Optimistic:** Allow merge if no conflicts - **Configurable:** Based on [freshness policy](/features/freshness-policies/) ### Merge Conflicts If the test branch has merge conflicts: - The queue cannot create the test branch - The PR is removed from the queue - Developer must resolve conflicts and re-queue ### Force Pushes If someone force-pushes to a PR while it's in the queue: - Most queues detect this and re-start testing - Some queues remove the PR and require re-queue ## Summary A merge queue works by: 1. **Queuing PRs** in order of priority and arrival 2. **Creating test branches** that represent the future state of main 3. **Running CI** on these test branches 4. **Merging** only if CI passes 5. **Re-testing** downstream PRs when failures occur This ensures that every commit on main has been tested against the exact state it will merge into — eliminating the "two green PRs make a red main" problem. ## Next Steps - [Do You Need One?](/decision/failure-scenarios/) — See the real cost of not having a merge queue - [Companies Using Merge Queues](/introduction/companies-using-merge-queues/) — How Uber, Shopify, and GitHub implement these mechanics - **Feature deep-dives:** [Two-Step CI](/features/two-step-ci/) · [Batching](/features/batching/) · [Speculative Checks](/features/speculative-merging/) · [Parallel Queues](/features/parallel-queues/) · [Priority Management](/features/priority-management/) · [Freshness Policies](/features/freshness-policies/) --- ## Companies Using Merge Queues Source: /introduction/companies-using-merge-queues/ Merge queues aren't theoretical—they're battle-tested at some of the largest engineering organizations in the world. Here's how real companies use them. ## Uber Uber built **SubmitQueue**, a custom merge queue for their massive monorepos. **The problem:** Before SubmitQueue, Uber's mainlines would often break due to developers racing to commit changes. On the worst days, 10% of commits had to be reverted, resulting in hours of lost developer time. Their iOS mainline was green only 52% of the time. **The solution:** SubmitQueue validates changes before merging using [speculative builds](/features/speculative-merging/). It employs machine learning to predict change success and optimize build scheduling. **Results:** - Main branch success rate jumped to **99%** - **74% improvement** in wait time to land code - Mainlines have remained green at all times since adoption Sources: [Uber Engineering: iOS Monorepo](https://www.uber.com/blog/ios-monorepo/), [Bypassing Large Diffs in SubmitQueue](https://www.uber.com/blog/bypassing-large-diffs-in-submitqueue/), [Building Uber's Go Monorepo with Bazel](https://www.uber.com/blog/go-monorepo-bazel/) ## Shopify Shopify integrated a merge queue into **Shipit**, their open-source deployment tool. **Scale:** Shopify's core monolith has over 2.8 million lines of Ruby code and 500,000 commits. They merge ~400 commits to master daily across 40+ deployments. Over 1,000 developers contribute to their codebase. **How it works:** Instead of merging directly to master, developers add PRs to the merge queue with a `/shipit` comment. The queue merges on their behalf after validating against a "predictive branch" — an approach similar to [two-step CI](/features/two-step-ci/). **Results:** Over 90% of pull requests to Shopify's core application use Shipit with the merge queue, making it the largest contributor to their monolith. Sources: [Introducing the Merge Queue](https://shopify.engineering/introducing-the-merge-queue), [Successfully Merging the Work of 1000+ Developers](https://shopify.engineering/successfully-merging-work-1000-developers) ## GitHub GitHub uses their own merge queue product internally. **Background:** In 2020, GitHub engineers set out to improve how they deploy and merge PRs in their largest monorepo. Their old process required special GitHub-only logic and external tools that weren't the same experience as their customers. **Scale:** Every month, 500+ engineers merge 2,500 pull requests into GitHub's large monorepo. The merge queue has processed over 30,000 PRs with 4.5 million CI executions. **Results:** - **33% reduction** in average time to deploy a change - Engineers called it "one of the most significant quality-of-life enhancements for deploying changes" Source: [How GitHub uses merge queue to ship hundreds of changes every day](https://github.blog/engineering/engineering-principles/how-github-uses-merge-queue-to-ship-hundreds-of-changes-every-day/) ## Google Google operates at a scale that required inventing their own approach. **Scale:** Google's monorepo (Piper) contains billions of lines of code. 95% of engineers contribute to the same codebase, averaging changes per second. They handle **100,000+ commits per day**. **How it works:** Google uses a "two-pointer" system: 1. One pointer to the latest commit on trunk 2. One pointer to the latest **verified green** commit Developers can pull from either pointer. Many default to "last green," treating the gap between pointers as an optimistic-but-unverified queue. Changes pass pretests before merging, then a continuous verification system validates the trunk. Source: [Not Rocket Science: How Bors and Google's TAP inspired modern merge queues](https://graphite.com/blog/bors-google-tap-merge-queue) ## Rust The Rust project pioneered modern merge queues in open source. **2013:** Graydon Hoare (Rust's creator) built **Bors** to enforce the "Not Rocket Science Rule": automatically maintain a repository that always passes all tests. **2014:** Contributor Barosl Lee created **Homu**, a more extensible reimplementation. Instead of merging then testing, Homu tested PRs before they landed by combining them with an up-to-date main branch. **2015:** Homu launched as a service (homu.io), letting other open-source projects use a hosted merge queue. **Legacy:** Homu's approach directly inspired GitHub's merge queue, GitLab's merge trains, Mergify, and Bors-NG. Source: [Rust Forge: Bors](https://forge.rust-lang.org/infra/docs/bors.html) ## Strava Strava built **Butler**, a CI-integrated merge queue. **Scale:** In 2016, 36 developers submitted 3,100 pull requests to their Ruby on Rails front-end application—their most active codebase. **How it works:** Developers submit PRs to the queue via Slack: `/butler merge android feature-a`. Butler squashes commits, runs all tests, merges the PR, and cleans up the branch. Fire-and-forget. **Results:** For close to a year, their repositories maintained a fast-forward-only master branch guaranteed to be green with only peer-reviewed code. Source: [Butler Merge Queue — How Strava Merges Code](https://medium.com/strava-engineering/butler-merge-queue-how-strava-merges-code-7095a3310930) ## Back Market Back Market switched to Mergify after outgrowing their homegrown solution. **The problem:** With ~300 engineers, their internal merge queue couldn't keep up. CI checks took 25 minutes per PR, creating a hard limit on daily merges. During peak hours, engineers waited **4-5 hours** to merge. Their goal was 150 PRs/day—they were nowhere close. **The solution:** Mergify's merge queue with [speculative checks](/features/speculative-merging/). No manual merging—developers create PRs and label them, Mergify handles rebasing, testing, and merging automatically. **Results:** Eliminated the bottleneck. Linear git history with fast-forward merges, fully automated workflow. Source: [Back Market × Mergify Case Study](https://mergify.com/case-studies/back-market) ## Common Patterns Looking across these implementations: | Company | Scale | Key Feature | |---------|-------|-------------| | Uber | Thousands of merges/day | [Speculative checks](/features/speculative-merging/) | | Shopify | 400 commits/day | [Two-step CI](/features/two-step-ci/) | | GitHub | 2,500 PRs/month | [High-velocity workflow](/use-cases/high-velocity-teams/) | | Google | 100,000+ commits/day | [Freshness policies](/features/freshness-policies/) | | Strava | 3,100 PRs/year | Slack-triggered queue | | Back Market | 150 PRs/day target | [Speculative checks](/features/speculative-merging/) | | Rust | Open source | Origin of the concept | **What they share:** - All operate monorepos or large codebases - All prioritize keeping main green over individual PR speed - All invested significant engineering effort before commercial tools existed ## History From Rust's Bors bot in 2013 to Shopify's Shipit, Uber's SubmitQueue, and eventually native features in GitHub and GitLab—merge queues evolved from side-project scripts into essential infrastructure. For the full story: [The Origin Story of Merge Queues](https://mergify.com/blog/the-origin-story-of-merge-queues) ## The Takeaway These companies built merge queues out of necessity. At scale, the cost of a broken main branch—blocked developers, constant reverts, deployment delays—exceeds the cost of implementing proper serialization. The pattern emerged independently at multiple organizations facing the same fundamental problem. ## Explore Further - [What is a Merge Queue?](/introduction/what-is-a-merge-queue/) — Understand the core concept behind these implementations - [Do You Need One?](/decision/failure-scenarios/) — Find out if your team is facing the same problems - [Monorepos](/use-cases/monorepos/) — How parallel queues solve the monorepo challenge - [High-Velocity Teams](/use-cases/high-velocity-teams/) — Strategies for scaling to 100+ PRs/day --- ## What Happens Without a Merge Queue Source: /decision/failure-scenarios/ When main breaks, failures cascade. This page shows what happens and why the costs exceed what most teams expect. ## The Anatomy of a Broken Main Two developers, Alice and Bob, work on separate features. [diagram] Both PRs were tested against M1 and passed. But Alice's changes (M2) and Bob's changes (M3) conflict in ways that neither CI run could detect. Now main is broken. ### A Concrete Example Your codebase has a utility module: ```python # utils.py def calculate_tax(amount): return amount * 0.2 ``` **Alice** is refactoring. She renames `utils.py` to `helpers.py` and updates all existing imports: ```python # helpers.py (renamed from utils.py) def calculate_tax(amount): return amount * 0.2 ``` **Bob** is building a new feature. He adds code that imports from `utils`: ```python # checkout.py (new file) from utils import calculate_tax def process_order(total): tax = calculate_tax(total) return total + tax ``` Both PRs pass CI: - Alice's PR: All tests pass—she updated every import - Bob's PR: All tests pass—`utils.py` still exists on his branch There's no merge conflict—they touched different files. Git happily merges both. But now `checkout.py` imports from `utils`, which no longer exists. **Main is broken.** ``` ModuleNotFoundError: No module named 'utils' ``` This pattern repeats with renamed functions, deleted code, changed signatures, and modified configuration. No merge conflict to warn you—just broken code on main. ## The Cascade Begins ### Stage 1: Discovery (Minutes to Hours) Someone notices main is broken. Maybe it's the post-merge CI. Maybe it's a developer who just pulled latest. Maybe it's a failed deployment. [diagram] **Time lost:** 15 minutes to several hours, depending on how obvious the failure is. ### Stage 2: Blocked Developers While main is broken: [diagram] - **Developers with ready PRs** can't merge—the policy is "don't merge to a broken main" - **Developers starting new work** base their branches on broken code - **Deployments are blocked** until main is fixed - **Hotfixes become complex** because you can't deploy the fix without also deploying the broken code **Cost:** If you have 10 developers and main is broken for 2 hours, that's 20 developer-hours of disruption. ### Stage 3: The Fix Someone identifies the problem and creates a fix PR. [diagram] **Problem:** It's hard to verify the fix works because CI is running against broken main. You might: - Think you fixed it, but you didn't - Fix one issue but introduce another - Have to iterate multiple times **Time lost:** 30 minutes to several hours for the fix itself. ### Stage 4: The Rebase Avalanche The fix lands. Main is green again. But now: [diagram] Every developer with an in-flight PR must: 1. Rebase onto the fixed main 2. Resolve any conflicts with the fix 3. Re-run CI 4. Wait in line to merge again **Cost:** If 8 PRs were in flight, and each rebase + CI takes 30 minutes, that's 4 hours of additional wait time across the team. ### Stage 5: The Risk of Recurrence The worst part: the same conditions that caused the break still exist. [diagram] Without a merge queue, you're right back where you started. The PRs were rebased, but they were only tested individually, not against each other. The cycle can repeat. ## The Hidden Costs ### Developer Context Switching Every time a developer is blocked, they have to: 1. Stop what they're doing 2. Investigate or wait 3. Resume their original work (losing context) [Research by Gloria Mark at UC Irvine](https://www.ics.uci.edu/~gmark/chi08-mark.pdf) shows it takes **23 minutes** to regain focus after an interruption. A broken main interrupts everyone. ### Compound Delays If main breaks once per week and takes 2 hours to fix, the direct cost is 2 hours. But the indirect cost is: - 2 hours × N developers blocked - Time to rebase all in-flight PRs - CI resources wasted on broken runs - Deployment delays - Possible customer impact if caught in a release cycle ### Trust Erosion Teams that experience frequent broken mains develop defensive behaviors: - Hesitation to merge ("let someone else go first") - Over-reliance on manual testing - Slower release cycles - Reduced confidence in CI ## How a Merge Queue Prevents This With a merge queue: [diagram] The queue catches the conflict **before** it breaks main. Alice's changes land. Bob gets notified. Main stays green. No one else is affected. ## Summary | Without Merge Queue | With Merge Queue | |---------------------|------------------| | Main can break | Main cannot break | | Everyone is blocked | Only failing PR author is affected | | Cascading rebases | No rebases needed | | CI waste on broken main | CI only runs on valid states | | Trust erodes over time | Confidence in main stays high | | Cycle can repeat | Problem is contained | The cost of a broken main is not the time to fix it—it's the compound disruption across your team. A merge queue eliminates this class of problem. ## Next Steps - [What is a Merge Queue?](/introduction/what-is-a-merge-queue/) — Understand the solution in depth - [Prerequisites](/decision/prerequisites/) — What to fix before adopting one - [Making the Case](/decision/making-the-case/) — How to convince your team or leadership - [Companies Using Merge Queues](/introduction/companies-using-merge-queues/) — See who's already solved this problem --- ## Do You Need a Merge Queue? Source: /decision/quiz/ Answer a few questions about your team and select the pain points you've experienced. We'll calculate your merge queue readiness score. --- ## The Signals ### Technical signals - **PR CI passes, post-merge CI fails** — The classic symptom. PRs test against stale main. - **CI takes 20+ minutes** — Long CI means stale branches and wasted re-runs. - **Frequent reverts** — "Revert 'Revert 'Add feature X''" is a red flag. - **Flaky tests appear on main but not PRs** — Integration issues only surface after merge. ### Organizational signals - **Merge races** — Two devs refreshing GitHub, waiting to click merge first. - **Informal merge coordination** — Slack messages like "can I merge?" or "go ahead, I'll wait." - **Friday freeze** — Unwritten rule to avoid merging before the weekend. - **Blame games** — Time spent figuring out who broke main instead of fixing it. ### Scale tipping points | Factor | Might not need | Great to have | Essential | |--------|----------------|---------------|-----------| | Team size | Under 20 engineers | 20-50 engineers | 50+ engineers | | Merge frequency | 5-10 PRs/day | 20-50 PRs/day | 50+ PRs/day | | CI duration | 5-10 minutes | 20-30 minutes | 45+ minutes | --- ## What's Next? - **Need to convince others?** → [Making the Case](/decision/making-the-case/) - **Want to understand features?** → [How Merge Queues Work](/introduction/how-merge-queues-work/) --- ## When to Skip It Source: /decision/when-to-skip-it/ A merge queue solves real problems—but it also adds complexity. If you don't have those problems, the queue is overhead without benefit. Here's when to skip it. ## Small Teams **Under 20 engineers on the same repo?** You probably don't need one. With a small team: - Merge conflicts are rare - Informal coordination works ("hey, I'm about to merge") - Someone notices a broken main quickly - The cost of fixing main is low (few people affected) A merge queue adds process and wait time that a small team doesn't need. :::tip[Exception: Long CI] Small team + very long CI (45+ minutes)? A merge queue alone won't help much—but [two-step CI](/features/two-step-ci/) can. Run fast checks on PRs, full suite only in the queue. ::: --- ## Low Merge Volume **Fewer than 5-10 PRs per day?** The math doesn't work. The probability of two PRs conflicting depends on them landing close together. With only a few merges per day, the window for conflict is small. If main breaks once a month, the cost of fixing it occasionally is lower than the daily overhead of a queue. --- ## Fast CI **CI under 5 minutes?** Rebasing is cheap. The main pain of "PR passed CI, but main broke" is the rebase-and-retest cycle. If your CI is fast: - Rebasing costs 5 minutes, not 30 - You can afford to test against latest main every time - The queue's value proposition shrinks Fast CI + small team = just enforce "rebase before merge" and you're fine. --- ## Main Rarely Breaks **Main hasn't broken in months?** You've solved the problem another way. Maybe you have: - Excellent test coverage - Strong code review catching integration issues - Developers who coordinate naturally - A codebase with few cross-cutting changes If it's not broken, don't add complexity to fix it. --- ## Monorepo with Independent Projects **Multiple independent projects, different teams, no shared code?** Consider separate repos instead. A merge queue helps when changes interact. If your "monorepo" is really just co-located independent projects: - They don't conflict with each other - A queue adds cross-team dependencies where none exist - Separate repos (or separate queues per project) might be simpler --- ## Early-Stage Startup **Moving fast and breaking things intentionally?** A queue slows you down. In early stages: - Speed of iteration matters more than stability - You're changing everything constantly - The cost of bugs is low (few users) - Process overhead hurts more than broken main Add a merge queue when you have enough users that broken main = real pain. --- ## The Overhead is Real A merge queue isn't free: | Overhead | Impact | |----------|--------| | Wait time | PRs wait in queue instead of merging immediately | | Learning curve | Team needs to understand queue behavior | | Failure handling | Queue failures need investigation | | Tool maintenance | Another system to configure and monitor | | Mental model | "Why didn't my PR merge?" becomes a question | If you don't have merge conflicts, broken main, or very long CI, this overhead buys you nothing. --- ## Revisit When Things Change Skip the merge queue now, but revisit when: - Team grows past 20 engineers — see [High-Velocity Teams](/use-cases/high-velocity-teams/) - Merge volume exceeds 10 PRs/day - CI slows down past 15-20 minutes — see [Long CI Pipelines](/use-cases/long-ci-pipelines/) - Main starts breaking regularly - Developers start complaining about rebase loops The signals in [What Happens Without a Merge Queue?](/decision/failure-scenarios/) will tell you when it's time. Ready to evaluate? Check the [Prerequisites](/decision/prerequisites/) or learn how to [make the case](/decision/making-the-case/) to your team. --- ## Summary Skip a merge queue if: - ✓ Small team (under 20 engineers) - ✓ Low volume (under 5-10 PRs/day) - ✓ Fast CI (under 5 minutes) - ✓ Main rarely breaks - ✓ Early-stage, speed over stability Consider one if any of these change. --- ## Prerequisites Source: /decision/prerequisites/ A merge queue is not a silver bullet. It amplifies your existing CI practices—both the good and the bad. If your tests are flaky, the queue will surface that pain constantly. If your CI is slow, the queue becomes a bottleneck. Fix these issues first, or the merge queue will be more frustrating than helpful. ## Flaky Tests This is the most critical prerequisite. A flaky test is one that sometimes passes and sometimes fails for the same code. [diagram] **The math is brutal:** - A 5% flake rate means 1 in 20 test runs fails randomly - With 20 PRs/day, that's 1 false failure every single day - With batching, a flake fails the entire batch—ejecting innocent PRs **With a merge queue**, flaky tests cause: - PRs ejected from the queue for no real reason - Developers re-queuing and waiting again - Lost trust in the system ("the queue is broken") - Wasted CI resources on retries ### Target: <2% flake rate Before adopting a merge queue, your test suite should have a flake rate under 2%. That means fewer than 1 in 50 runs fails randomly. [Google's testing guidelines](https://testing.googleblog.com/2016/05/flaky-tests-at-google-and-how-we.html) discuss how they tackled this at scale. ### How to measure Run your test suite 100+ times on the same commit. Count failures. ```bash # Run tests N times, track pass/fail rate runs=100; fails=0 for i in $(seq 1 $runs); do npm test &>/dev/null || ((fails++)) printf "\rRun %d/%d (failures: %d)" "$i" "$runs" "$fails" done echo -e "\n\nFlake rate: $fails/$runs ($(echo "scale=1; $fails*100/$runs" | bc)%)" ``` If your flake rate exceeds 2%, you have work to do before adopting a merge queue. ### How to fix 1. **Quarantine flaky tests** — Move them to a separate suite that doesn't block merges 2. **Fix the root cause** — Usually: timing issues, shared state, or external dependencies 3. **Delete tests that can't be fixed** — A test that fails randomly provides negative value --- ## CI Reliability A merge queue trusts your CI completely. If CI says "pass," the PR merges. If CI says "fail," the PR is ejected. There's no human in the loop second-guessing the result. This means CI must give a reliable signal. When CI fails, it should mean the code is actually broken—not that a runner crashed or the network hiccuped. **Problems to fix:** - Runners that crash or timeout randomly - Network issues causing spurious failures - Resource contention (out of memory, disk full) - Non-deterministic builds (different results for same code) ### Target: >99% infrastructure reliability CI failures should almost always be real test failures, not infrastructure problems. ### Red flags - "CI was flaky, re-running" is a common phrase on your team - Developers retry failed jobs without looking at logs - Same test passes on retry without code changes - CI failures correlate with time of day (resource contention) --- ## CI Speed CI speed matters because of the **feedback loop**. When a PR fails in the queue, the developer needs to know quickly so they can fix it and re-queue. A 45-minute CI means 45 minutes of waiting before learning something went wrong—then another 45 minutes after the fix. ### Ideal: <20 minutes Under 20 minutes keeps the feedback loop tight. Developers can fix issues and re-queue within the same focus session. [Research on developer productivity](https://queue.acm.org/detail.cfm?id=3595878) shows that fast feedback cycles significantly improve developer experience. But not everyone can achieve this—and that's okay. If your CI is slower, merge queue features can help: - **[Batching](/features/batching/)** — Test multiple PRs together, amortizing CI time across the batch - **[Two-step CI](/features/two-step-ci/)** — Run fast checks on PRs, full suite only in the queue - **[Speculative checks](/features/speculative-merging/)** — Test PRs in parallel, assuming earlier ones will pass - **[Parallel queues](/features/parallel-queues/)** — Separate queues for independent parts of the codebase ### The real question Can your developers get feedback and iterate within a reasonable time? If a PR takes 3 CI cycles to merge (common for complex changes), that's 3× your CI duration in wait time. Make sure that's acceptable for your team. --- ## Test Coverage A merge queue validates that tests pass—nothing more. If your tests don't catch bugs, the queue won't either. **Merge queue guarantees:** - ✅ Tests that exist will pass on main - ❌ Bugs not covered by tests will still reach main ### Minimum bar Before adopting a merge queue, ensure: - Critical user paths have integration tests - Core business logic has unit tests - API contracts are tested - Database migrations are tested ### Warning sign If you frequently hear "tests passed but the feature is broken," your test coverage is the problem—not your merge process. --- ## Readiness Checklist | Prerequisite | Target | How to Measure | |--------------|--------|----------------| | Flaky test rate | <2% | Run tests 100x on same commit | | CI reliability | >99% | Track infra failures vs test failures | | CI duration | <20 min ideal | Average pipeline run time | | Test coverage | Critical paths covered | Code review, coverage reports | --- ## What If You're Not Ready? If you don't meet these prerequisites, you have options: ### Fix flaky tests first This is almost always the right answer. Flaky tests hurt you with or without a merge queue—the queue just makes the pain visible. ### Start with a subset Some merge queue tools let you enable the queue for specific paths or teams. Start with the most stable part of your codebase. ### Use "dry run" mode Some tools offer a mode where the queue runs but doesn't block merges. Use this to measure your flake rate and CI reliability before committing. ### Optimize CI in parallel You can work on CI speed while fixing flaky tests. Both improvements pay off independently. --- ## Next Steps Once you meet these prerequisites: - [When to Skip It](/decision/when-to-skip-it/) — Make sure a merge queue is right for your situation - [Making the Case](/decision/making-the-case/) — Convince your team or leadership --- ## Making the Case Source: /decision/making-the-case/ You know you need a merge queue. Now you need to convince others. This page gives you the talking points, data, and framing for different audiences. ## Know Your Audience Different stakeholders care about different things. ### For Engineering Managers **They care about:** Team velocity, predictability, developer happiness **Lead with:** - "We're losing X hours per week to broken main incidents" - "Developers are blocked Y times per month waiting for main to be fixed" - "We can ship faster with less coordination overhead" **Frame it as:** Risk reduction + velocity improvement ### For Individual Contributors **They care about:** Less friction, fewer interruptions, shipping their code **Lead with:** - "No more rebasing 3 times to merge one PR" - "No more 'who broke main?' investigations" - "Your PR gets tested against what main will actually look like" **Frame it as:** Quality of life improvement ### For Engineering Leadership / VPs **They care about:** Reliability, incident reduction, team scalability **Lead with:** - "Main breakages are preventable incidents" - "As we scale, coordination costs grow quadratically without automation" - "Teams at our scale ([Google, Shopify, Uber](/introduction/companies-using-merge-queues/)) treat this as infrastructure" **Frame it as:** Scaling investment + incident prevention ### For Product / Business Stakeholders **They care about:** Ship velocity, reliability, predictability **Lead with:** - "We can deploy with confidence more frequently" - "Fewer rollbacks means more stable releases" - "Engineering time goes to features, not firefighting" **Frame it as:** Faster, safer delivery --- ## The ROI Calculation ### Quantify the current cost Start by measuring what broken main actually costs: ``` Weekly cost = (breaks per week) × (avg fix time) × (devs blocked) × (hourly rate) ``` **Example:** - 2 breaks per week - 1.5 hours average to fix - 8 developers blocked - $75/hour loaded cost ``` 2 × 1.5 × 8 × $75 = $1,800/week = $93,600/year ``` That's just the direct cost. Add: - CI re-runs from rebasing: +20-30% - Context switching cost: +20-30% - Deployment delays: variable but significant ### Compare to merge queue cost Most merge queue solutions cost: - **SaaS options:** $50-500/month depending on scale - **Self-hosted:** Engineering time to set up + maintain - **Platform-native:** Free, but often limited features **The math usually works out to 10-50x ROI.** ### Time savings breakdown | Activity | Without MQ | With MQ | Savings | |----------|-----------|---------|---------| | Rebasing before merge | 15 min × 3 PRs/dev/week | 0 | 45 min/dev/week | | Investigating broken main | 30 min × 2/week shared | 0 | 1 hour/week | | Waiting for main to be fixed | 1 hour × 2/week × team | 0 | N hours/week | | CI re-runs | 20 min × 2/PR | 20 min × 1/PR | 50% CI cost | --- ## Common Objections (and Responses) ### "We don't break main that often" > Track it for two weeks. Most teams underestimate because small incidents don't get reported. Count every time someone says "CI is red" or "wait to merge." ### "We can just require rebasing before merge" > That doesn't scale. With 10 PRs/day and 20-minute CI: > - PR merges, main advances > - 9 PRs must rebase and re-run CI > - While they're running, another PR merges > - The cycle never ends > > A merge queue tests PRs against *future* main, breaking the cycle. ### "GitHub has branch protection, isn't that enough?" > Branch protection requires PRs to be up-to-date, but doesn't coordinate merges. Two PRs can both be "up to date," both pass CI, and still conflict when combined. ### "It's another tool to maintain" > True. But compare: > - Tool maintenance: a few hours/quarter > - Broken main incidents: hours/week > > The math favors the tool. ### "Our CI is fast, we don't need it" > Fast CI helps, but doesn't solve the coordination problem. If two PRs merge within seconds of each other, CI speed doesn't matter—they weren't tested together. ### "We're too small" > This is valid. Under 5 engineers with few daily merges, manual coordination works. Consider setting a threshold: when you exceed N merges per day or experience your first semantic conflict, revisit the decision. --- ## Building Internal Support ### Start with data Before proposing anything, collect numbers: - How often does main break? (Check CI history) - How long to fix? (Check incident timelines or Slack) - How many PRs merge daily? (Check GitHub) - How long is CI? (Check average run times) ### Pilot with one team If you can't get org-wide buy-in, propose a pilot: - "Let's try this on [low-risk repo] for one month" - "We'll measure before/after and share results" - Success makes the next conversation easier. ### Connect to existing pain Reference recent incidents: - "Remember when main was broken for 3 hours last Tuesday?" - "This would have caught that before merge" ### Propose, don't demand Frame it as an experiment: - "I'd like to try X for 30 days and measure the impact" - "If it doesn't help, we turn it off" --- ## Next Steps 1. **Measure current state** — Track broken main incidents for 2 weeks 2. **Calculate the cost** — Use the ROI formula above 3. **Propose a pilot** — Start small, measure results 4. **Share the wins** — Document time saved, incidents prevented --- Still not sure if you need one? Take the [interactive quiz](/decision/final-quiz/) or review [what happens without a merge queue](/decision/failure-scenarios/). --- ## Final Quiz Source: /decision/final-quiz/ You've learned the concepts. You've seen the diagrams. You've read about Uber, Shopify, and Google. **Now it's time to prove your mastery.** 🎯 This 12-question quiz covers everything from the basics to advanced strategies. Get 10+ right and you'll earn bragging rights (and a shareable diploma). --- ## Want to Review First? Quick links to brush up on key topics: - [What is a Merge Queue?](/introduction/what-is-a-merge-queue/) — The fundamentals - [How Merge Queues Work](/introduction/how-merge-queues-work/) — PR lifecycle, test branches, merge strategies - [Speculative Checks](/features/speculative-merging/) — Parallelizing the queue - [Batching](/features/batching/) — Testing multiple PRs together - [Parallel Queues](/features/parallel-queues/) — Independent lanes for different scopes - [Companies Using Merge Queues](/introduction/companies-using-merge-queues/) — Real-world examples Good luck! 🍀 --- ## Two-Step CI Source: /features/two-step-ci/ Most teams already run CI on pull requests. A merge queue adds a **second validation step** that runs when the PR enters the queue: [diagram] ## Why Two Steps? - **PR CI** catches obvious issues quickly, giving fast feedback to developers - **Queue CI** validates against the true merge target, catching integration issues The queue CI can run a different (often more comprehensive) test suite than PR CI. Some teams run fast unit tests on PRs and full integration tests in the queue. ## Common Configurations | PR CI | Queue CI | Use Case | |-------|----------|----------| | Unit tests only | Full test suite | Fast PR feedback, thorough queue validation | | Lint + type check | All tests + E2E | Catch formatting issues early, integration last | | Affected tests only | Full test suite | Scale PR CI for monorepos | | Same as queue | Same as PR | Simple setup, consistent validation | ## Catching What PR CI Misses Consider two PRs that both pass PR CI independently: - **PR #1** adds a new required parameter to a shared API endpoint - **PR #2** calls that same endpoint without the new parameter Both PRs pass their own CI — they were tested against the current `main` where neither change existed yet. But when PR #1 merges first, PR #2 is now broken. Queue CI catches this. When PR #2 enters the queue, it's tested against `main + PR #1`, where the API signature has already changed. The test fails *before* reaching main. This is the kind of [semantic conflict](/introduction/what-is-a-merge-queue/) that only two-step CI can reliably prevent. PR CI tells developers "your change works in isolation." Queue CI answers a different question: "will your change work when it actually lands?" ## Optimizing CI Costs Two-step CI isn't just about safety — it's a cost optimization strategy. Expensive tests (E2E, browser tests, load tests) only run when a PR has already passed review and is ready to merge. For a team with 50 PRs per week where only 30 pass review: - **Without two-step:** 50 full CI runs - **With two-step:** 50 lightweight PR CI runs + 30 full queue CI runs The savings compound when full CI involves spinning up infrastructure like databases, browser farms, or staging environments. ## Benefits 1. **Faster PR feedback** — developers get quick signal on obvious issues 2. **Comprehensive merge validation** — full testing before code lands 3. **Resource optimization** — expensive tests only run when PR is ready to merge 4. **Separation of concerns** — different test suites for different purposes ## Related Features - **[Batching](/features/batching/)** — combine queue CI runs for even more efficiency - **[Freshness Policies](/features/freshness-policies/)** — control how up-to-date queue tests must be - **[Speculative Checks](/features/speculative-merging/)** — run queue CI for multiple PRs in parallel --- ## Batching Source: /features/batching/ Testing each PR individually means one CI run per PR. With 10 PRs, that's 10 CI runs. Batching groups multiple PRs into a single CI run, dramatically cutting CI cost and resource usage. | | Without Batching | With Batching | |---|---|---| | **PRs** | 4 | 4 | | **CI runs** | 4 | 1 | | **Cost** | 4x | 1x | [diagram] ## How It Works 1. Multiple PRs enter the queue 2. The merge queue combines them into a single test branch 3. CI runs once against the combined changes 4. If it passes, all PRs merge together ## Handling Failures If the batch fails, the merge queue needs to identify which PR caused the failure. Common strategies: ### Speculative Bisection Test overlapping subsets in parallel. This allows partial merges while identifying failures. ``` Batch [1,2,3,4] fails → Test [1,2] and [1,2,3] in parallel → [1,2] passes → merge PR #1 and #2 → [1,2,3] fails → PR #3 is the problem → Remove PR #3 → Put PR #4 back in queue ``` ## In Practice Consider a team merging 20 PRs per day with a 30-minute CI pipeline: - **Without batching:** 20 CI runs × 30 min = 10 hours of CI time - **With batches of 4:** 5 CI runs × 30 min = 2.5 hours of CI time That's a 75% reduction in CI resource usage. For teams paying for CI by the minute, this directly reduces infrastructure costs. The trade-off appears when a batch fails. If batch [1,2,3,4] fails, bisection adds 1-2 extra CI runs to isolate the culprit. But with a healthy codebase (failure rate under 5%), the savings far outweigh the occasional bisection cost. ## Choosing Your Batch Size The right batch size depends on your failure rate and CI duration: - **Low failure rate (<2%)** — larger batches (5-10 PRs) work well since bisection is rare - **Medium failure rate (2-5%)** — moderate batches (3-5 PRs) balance efficiency and recovery time - **High failure rate (>5%)** — small batches (2-3 PRs), or invest in fixing your flaky tests first A useful rule of thumb: if bisection happens more than once per day, your batch size is too large or your test stability needs work. ## Configuration Options | Setting | Description | |---------|-------------| | **Batch size** | Maximum PRs per batch (e.g., 5, 10, unlimited) | | **Batch wait time** | Time to wait for more PRs before starting CI | ## Combining with Speculative Checks Batching and [speculative checks](/features/speculative-merging/) are complementary strategies that can be used together: :::tip[Best of both worlds] - **Batching** reduces CI cost by testing multiple PRs per run - **Speculative checks** reduce latency by testing batches in parallel A queue might test batch [1-3] while speculatively testing batch [4-6], achieving both efficiency and speed. ::: ## Trade-offs **Pros:** - Dramatically reduces CI cost and resource usage - Fewer CI runs means less infrastructure load - Combines well with speculative checks for both speed and efficiency **Cons:** - One failure affects the whole batch - Bisection adds latency when failures occur - May need larger CI runners for combined changes ## Related Features - **[Speculative Checks](/features/speculative-merging/)** — test batches in parallel for maximum throughput - **[Two-Step CI](/features/two-step-ci/)** — ensure PRs pass lightweight checks before entering a batch - **[Priority Management](/features/priority-management/)** — urgent PRs can bypass batch waiting --- ## Speculative Checks Source: /features/speculative-merging/ Instead of waiting for each PR to complete, the queue can **test multiple PRs in parallel by assuming earlier PRs will pass**. ## Sequential vs Speculative **Sequential approach** — each PR waits for the previous one to finish: [diagram] **Speculative approach** — all PRs test in parallel, assuming earlier ones will pass: [diagram] **3x faster!** ## How It Works 1. PR #1 enters the queue → test against `main` 2. PR #2 enters the queue → test against `main + PR #1` (assuming #1 will pass) 3. PR #3 enters the queue → test against `main + PR #1 + PR #2` (assuming both will pass) All three tests run simultaneously. ## When Speculation Fails If PR #1 fails, the speculation for PRs #2 and #3 was wrong. They were tested against a state that will never exist. **What happens:** - PR #1 is removed from the queue - PRs #2 and #3 are automatically re-queued - PR #2 now tests against `main` (not `main + PR #1`) - PR #3 tests against `main + PR #2` The speculation was wrong, but we only lost the time for one CI run. On average, this is still much faster than sequential testing. ## Speculation Depth You can limit how far ahead the queue speculates: | Depth | Behavior | |-------|----------| | 1 | No speculation (sequential) | | 3 | Test up to 3 PRs ahead | | Unlimited | Test all PRs in parallel | Higher depth = more parallelism but more wasted CI if early PRs fail. ## Choosing Your Depth The right speculation depth depends on two factors: your queue failure rate and your CI resource budget. **Low failure rate (<2%):** Use unlimited or high depth. Speculations almost always succeed, so you get maximum parallelism with minimal waste. **Medium failure rate (2-5%):** Depth of 3-5 works well. You get significant speedup while limiting cascade failures to a manageable scope. **High failure rate (>5%):** Keep depth at 2-3, and invest in stabilizing your test suite first. High failure rates cause frequent cascade restarts that can actually make the queue *slower* than sequential processing. A useful metric to track: **speculation hit rate** — the percentage of speculative runs that don't need to restart. If your hit rate drops below 80%, consider reducing depth or fixing test stability. ## Cascade Failures When speculation fails, the impact depends on *which* PR fails: - **PR #1 fails** → PRs #2, #3, #4 all restart (tested against a state that will never exist) - **PR #3 fails** → PRs #1 and #2 merge normally, only PR #4 restarts A failure early in the queue causes more wasted work than a failure late in the queue. This is why most merge queues run a [lightweight PR CI check](/features/two-step-ci/) before PRs even enter the queue — catching obvious failures early reduces costly cascade restarts. ## Combining with Batching Speculative checks and [batching](/features/batching/) work well together: :::tip[Maximize throughput] - **Speculative checks** reduce latency by testing in parallel - **Batching** reduces CI cost by grouping PRs per run Example: speculatively test batch [1-3] and batch [4-6] in parallel. You get the speed of speculation with the efficiency of batching. ::: ## Best For - Teams with high PR volume — see [High-Velocity Teams](/use-cases/high-velocity-teams/) - Codebases with low failure rates in the queue - When CI resources are not a constraint — see [Long CI Pipelines](/use-cases/long-ci-pipelines/) for constrained environments ## Related Features - **[Batching](/features/batching/)** — combine with speculation for maximum throughput - **[Two-Step CI](/features/two-step-ci/)** — pre-validate PRs to improve speculation hit rate - **[Priority Management](/features/priority-management/)** — high-priority PRs speculate from the front of the queue --- ## Parallel Queues Source: /features/parallel-queues/ Not all changes conflict with each other. A frontend CSS change and a backend API change can often be tested and merged independently. **Parallel queues** (sometimes called "partitions" or "scopes") allow this: **Single queue:** all PRs wait in one line, even if they don't conflict. [diagram] **Parallel queues:** independent scopes merge simultaneously. [diagram] ## How It Works With parallel queues: - PRs in the **same scope** are tested against each other (strict ordering) - PRs in **different scopes** can merge independently (parallel) - You define scopes based on your codebase (by directory, by team, by project) This dramatically increases throughput for large monorepos where most changes don't interact. ## In Practice Consider a monorepo with three teams: Frontend, Backend, and Platform. Without parallel queues, all 30 daily PRs wait in a single queue. With a 20-minute CI pipeline, a serial queue processes ~24 PRs per 8-hour day. At 30 PRs/day, the queue falls behind. With parallel queues scoped by directory: | Scope | PRs/day | Queue capacity | Status | |-------|---------|----------------|--------| | `src/frontend/` | 12 | 24/day | Comfortable | | `src/backend/` | 15 | 24/day | Comfortable | | `src/platform/` | 3 | 24/day | Nearly empty | Each scope operates independently. Combined throughput is 72 PRs/day — 3x the single-queue capacity. Teams merge at their own pace without blocking each other. ## Defining Scopes Common approaches to defining parallel queues: - **By directory** — group by file paths (`src/frontend/`, `src/backend/`, `docs/`) - **By team** — each team owns their scope and merge pace - **By build target** — particularly useful with monorepo build tools like Bazel, Rush, Nx, Turborepo, or Pants that understand dependency graphs ## Handling Cross-Scope Changes What happens when a PR touches multiple scopes? | Strategy | Behavior | |----------|----------| | **Union** | PR joins all affected queues, must pass all | | **Primary scope** | PR joins only its primary/largest scope | | **Global queue** | Cross-scope PRs go to a single global queue | ## Benefits 1. **Higher throughput** - independent changes don't block each other 2. **Faster merges** - smaller queues = shorter wait times 3. **Team autonomy** - teams control their own merge pace 4. **Fault isolation** - one team's failures don't affect others ## Scope Design Principles Good scope boundaries share three characteristics: 1. **Low coupling** — changes rarely cross scope boundaries. If 30% of PRs touch multiple scopes, the boundaries are too fine-grained. 2. **Independent testing** — each scope has its own test suite that can validate changes without running unrelated tests. 3. **Clear ownership** — teams know which scope their changes belong to, reducing ambiguity. Start with coarse scopes (2-3 partitions) and refine over time. Overly granular scopes create overhead and make cross-scope changes painful. ## Considerations - Scope definitions need maintenance as codebase evolves - Cross-cutting changes may still create bottlenecks - Requires clear code ownership boundaries ## Related Features - **[Speculative Checks](/features/speculative-merging/)** — each parallel queue can speculate independently - **[Freshness Policies](/features/freshness-policies/)** — per-scope freshness for maximum throughput - **[Batching](/features/batching/)** — batch within each scope for CI efficiency See also: [Monorepos](/use-cases/monorepos/) for a deep dive on parallel queues in practice. --- ## Freshness Policies Source: /features/freshness-policies/ The strictest policy requires every PR to be tested against the **exact current state of main**. But this isn't always necessary. ## Freshness Options | Freshness Policy | Description | Use Case | |-----------------|-------------|----------| | **Strict** | Must be tested against current HEAD | Critical systems, regulated industries | | **Within N commits** | Can be up to N commits behind | Balance between safety and throughput | | **Within scope** | Only up-to-date within its partition | Monorepos with independent components | | **Time-based** | Valid for N minutes after queue CI passes | High-velocity teams with fast CI | ## Why Relax Freshness? Strict freshness means every merge invalidates all in-flight queue tests. With high PR volume, this creates a race condition where PRs struggle to merge. **Example:** - PR #1 passes queue CI at 10:00 - PR #2 merges at 10:01 - PR #1's test is now "stale" and must re-run With relaxed freshness (e.g., "within 2 commits"), PR #1 can still merge if it's only 1 commit behind. ## Risk vs Throughput [diagram] **The risk is usually theoretical** because: - Most PRs don't conflict semantically - Type systems catch many integration issues - Runtime failures are rare for unrelated changes ## Choosing a Policy | Team Profile | Recommended Policy | |--------------|-------------------| | Regulated/compliance | Strict | | High-velocity startup | Time-based (5-10 min) | | Monorepo with scopes | Within scope | | Medium velocity | Within 2-3 commits | ## Strict Freshness Under Load Consider a team merging 30 PRs per day with a 15-minute CI pipeline. With strict freshness, every merge invalidates all in-flight tests. If 3 PRs are testing and one merges, the other 2 must restart. During peak hours, PRs can cycle through 3-4 restarts before finally merging — turning a 15-minute pipeline into a 60-minute wait. The math is unforgiving: | PRs in queue | Avg restarts per PR (strict) | Effective merge time | |---|---|---| | 2 | 0.5 | ~22 min | | 5 | 1.5 | ~37 min | | 10 | 3+ | ~60+ min | Relaxing freshness to "within 2 commits" eliminates most restarts while maintaining strong safety guarantees. The probability of a semantic conflict between two unrelated PRs is typically well under 1%. ## Per-Scope Freshness When using [parallel queues](/features/parallel-queues/), freshness can be applied per scope. A frontend PR only needs to be fresh relative to other frontend changes — a backend merge doesn't invalidate it. This is the most powerful freshness optimization for monorepos. It combines the safety of strict freshness within each domain with the throughput of relaxed policies across domains. ## Related Features - **[Parallel Queues](/features/parallel-queues/)** — per-scope freshness for monorepos - **[Speculative Checks](/features/speculative-merging/)** — reduce the impact of restarts from freshness invalidation - **[Batching](/features/batching/)** — atomic batch merges count as a single freshness event See also: [High-Velocity Teams](/use-cases/high-velocity-teams/) for guidance on choosing freshness policies at scale. --- ## Priority Management Source: /features/priority-management/ Not all PRs are equally urgent. A critical security fix can jump ahead of routine changes: [diagram] ## Priority Levels Typical priority tiers: | Priority | Use Case | Example | |----------|----------|---------| | **Urgent/Critical** | Production incidents, security fixes | Hotfix for data breach | | **High** | Time-sensitive features, blockers | Release deadline feature | | **Normal** | Regular development work | Most PRs | | **Low** | Non-urgent improvements | Refactoring, tech debt | ## How Priority Affects the Queue When a high-priority PR enters: 1. It's placed ahead of lower-priority PRs 2. Lower-priority PRs may be re-queued to test behind it 3. The high-priority PR gets tested first [diagram] ## Setting Priority Priority can be set via: - **Rules** — automatically based on labels, files changed, or PR author - **Commands** — manually via PR comments ## In Practice It's 3 PM on a Wednesday. Your merge queue has 8 PRs testing. A security vulnerability is reported and an engineer has a hotfix ready in 15 minutes. Without priority management, the hotfix waits behind all 8 PRs. With a 20-minute CI pipeline, that's potentially 40+ minutes before the fix even starts testing. With priority management, the hotfix enters the queue at position 1. The queue pauses lower-priority testing, tests the hotfix against `main`, and merges it within 20 minutes. The other 8 PRs re-queue behind the fix and continue normally. The difference: 20 minutes to production vs. 60+ minutes. For a security fix, that gap matters. ## Avoiding Queue Starvation A common failure mode: teams start using "high priority" for routine work, and normal-priority PRs never merge. This is **queue starvation**. Mitigations: - **Priority decay** — PRs waiting longer than a threshold automatically promote. A normal PR waiting 2 hours becomes high priority. - **Reserved slots** — ensure at least N% of queue capacity serves normal-priority PRs, regardless of how many high-priority PRs are waiting. - **Priority budgets** — each team gets a limited number of high-priority slots per week. Once exhausted, all PRs enter at normal priority. - **Monitoring** — track the distribution of priority levels over time. If more than 20% of PRs are "high priority," the system is being gamed. ## Best Practices 1. **Reserve urgent for true emergencies** — overuse defeats the purpose 2. **Document what qualifies for each level** — avoid priority inflation 3. **Monitor priority distribution** — too many high-priority PRs indicates a problem 4. **Consider priority decay** — PRs waiting too long could auto-promote ## Related Features - **[Speculative Checks](/features/speculative-merging/)** — high-priority PRs speculate from the front - **[Freshness Policies](/features/freshness-policies/)** — urgent PRs may need stricter freshness - **[Two-Step CI](/features/two-step-ci/)** — consider fast-tracking queue CI for emergency hotfixes --- ## Monorepos Source: /use-cases/monorepos/ Monorepos amplify every problem a merge queue solves. More developers, more PRs, more potential conflicts, more CI runs. A single queue becomes a bottleneck. Without one, main breaks constantly. ## Why Monorepos Need Merge Queues In a monorepo, the math works against you: | Metric | Polyrepo (10 repos) | Monorepo | |--------|---------------------|----------| | PRs per day | 5 per repo = 50 total | 50 in one repo | | Conflict potential | Low (isolated repos) | High (shared codebase) | | "Main is broken" impact | One team blocked | Everyone blocked | | CI cost | Scales per repo | Full suite every PR? | A broken main in a monorepo blocks *everyone*. The pressure to keep it green is much higher. ## The Single Queue Problem A naive merge queue serializes all PRs: [diagram] With 20-minute CI and 50 PRs/day, you're underwater. PRs queue for hours. Engineers context-switch. Frustration mounts. ## The Solution: Parallel Queues [Parallel queues](/features/parallel-queues/) let independent changes merge simultaneously: [diagram] Each team gets their own queue. Frontend changes don't block backend. iOS doesn't wait for Android. ### Defining Queue Boundaries Common approaches: | Approach | Works well when | Watch out for | |----------|-----------------|---------------| | **By directory** | Clear folder structure (`apps/`, `libs/`) | Shared code in `/common` | | **By team** | Strong code ownership | Cross-team changes | | **By build target** | Using Bazel, Nx, Turborepo | Requires build tool setup | | **By CI config** | Different test suites per area | Config drift | Most teams start with directory-based rules and evolve from there. ## Handling Shared Code The tricky part: what happens when a PR touches code used by multiple areas? ``` src/ ├── apps/ │ ├── frontend/ → Frontend queue │ ├── backend/ → Backend queue │ └── mobile/ → Mobile queue └── libs/ └── common/ → ??? which queue ??? ``` The standard approach: **the PR joins all affected queues and must pass all of them**. A change to `libs/common` that's used by frontend and backend joins both queues, runs both CI suites, and only merges when both pass. [diagram] If either queue fails, the PR is removed from both: [diagram] ## CI Optimization Running the full test suite for every PR defeats the purpose. Monorepo-aware CI helps: - **Build tools** (Bazel, Nx, Turborepo, Pants) can determine affected tests - **Per-queue CI** — Each queue runs only its relevant tests - **Two-step CI** — Fast checks on PR, thorough checks in queue Example queue CI strategy: | Queue | CI Runs | |-------|---------| | Frontend | Frontend unit tests, E2E, build | | Backend | Backend unit tests, API tests, migrations | | Mobile | Mobile unit tests, build for simulators | | Global | Full integration suite | ## Scaling Considerations ### Queue Depth Monitoring Track how long PRs wait in each queue. If one queue consistently backs up: - Split it into sub-queues - Investigate slow CI - Add [batching](/features/batching/) to reduce CI runs ### Cross-Queue Dependencies Some changes genuinely need coordination. A breaking API change in backend needs frontend updates. Options: 1. **Stack PRs** — Land backend first, then frontend 2. **Feature flags** — Ship both independently, flag controls activation 3. **Atomic PR** — Single PR touching both (goes to global queue) ### Team Autonomy vs. Consistency Parallel queues give teams autonomy—each controls their merge pace. But this can lead to: - Different merge standards per team - Confusion about which queue a PR belongs to - Orphaned queues when teams restructure Document queue ownership clearly. Review quarterly. ## Example: Growing Into Parallel Queues **Stage 1: Single queue** - Small team, fast CI - One queue works fine **Stage 2: CI becomes slow** - Add [two-step CI](/features/two-step-ci/) - Add [batching](/features/batching/) - Still one queue **Stage 3: Queue backs up despite optimization** - Split into 2-3 parallel queues by major area - Keep a global queue for shared code **Stage 4: Multiple teams, clear ownership** - Queue per team/domain - Build-tool integration for automatic assignment - Metrics dashboard per queue ## Key Takeaways 1. **Monorepos need merge queues** — The alternative is constant broken main 2. **Single queues don't scale** — Parallel queues are essential for large monorepos 3. **Start simple, evolve** — Begin with directories, add sophistication as needed 4. **Shared code is the hard part** — Have a clear strategy before it becomes a bottleneck 5. **Optimize CI per queue** — Don't run everything for every PR 6. **Use [freshness policies](/features/freshness-policies/)** — Per-scope freshness avoids unnecessary re-testing across unrelated queues See also: [Companies Using Merge Queues](/introduction/companies-using-merge-queues/) for how Uber, Shopify, and others handle monorepos at scale. --- ## High-Velocity Teams Source: /use-cases/high-velocity-teams/ When you're merging 20, 50, or 100+ PRs per day, a merge queue isn't optional—it's infrastructure. At this velocity, manual coordination breaks down and broken main becomes a daily occurrence. ## The Math of High Velocity With 15-minute CI and a simple serial queue: | PRs/day | Queue capacity | Result | |---------|----------------|--------| | 20 | 32 (8hr ÷ 15min) | Manageable | | 50 | 32 | PRs back up | | 100 | 32 | Impossible | Without optimization, high-velocity teams hit a ceiling. PRs wait hours. Engineers lose context. Frustration spikes. ## Breaking the Bottleneck High-velocity teams need every optimization available: ### 1. Speculative Checks [Speculative checks](/features/speculative-merging/) test multiple PRs in parallel by assuming earlier ones will pass: [diagram] Three PRs merge in the time of one. Throughput triples. ### 2. Batching [Batching](/features/batching/) groups PRs into a single CI run: ``` Batch: PR #1, #2, #3, #4, #5 → One CI run for 5 PRs → 80% CI cost reduction ``` Fewer CI runs means lower costs and faster feedback for everyone. ### 3. Two-Step CI [Two-step CI](/features/two-step-ci/) splits validation: - **PR CI**: Fast checks (lint, unit tests, build) - **Queue CI**: Thorough checks (integration, E2E) This keeps PR feedback fast while the queue handles comprehensive testing. ### 4. Priority Lanes Not all PRs are equal. [Priority management](/features/priority-management/) lets urgent changes skip ahead: - Hotfixes merge immediately - Feature work follows normal flow - Docs and chores can wait ## Measuring Queue Health Track these metrics to know if your queue is keeping up. These thresholds are based on patterns observed at [high-performing teams like GitHub, Shopify, and Uber](/introduction/companies-using-merge-queues/)—adjust for your context: | Metric | Healthy | Warning | Critical | |--------|---------|---------|----------| | **Median wait time** | < 30 min | 30-60 min | > 60 min | | **P95 wait time** | < 2 hr | 2-4 hr | > 4 hr | | **Queue depth** | < 10 PRs | 10-20 PRs | > 20 PRs | | **Failure rate** | < 5% | 5-15% | > 15% | When metrics trend toward warning, investigate before they hit critical. ## Common Patterns ### Small, Focused PRs High-velocity teams ship small PRs: - Easier to review - Faster CI - Lower conflict risk - Simpler rollback A team shipping 50 small PRs moves faster than one shipping 10 large PRs. ### Trunk-Based Development Most high-velocity teams practice trunk-based development: - Short-lived branches (hours, not days) - Frequent integration - Feature flags for incomplete work The merge queue makes this safe by catching integration issues before they land. ### Automated Everything At high velocity, manual steps become bottlenecks: - Auto-merge when approved and CI passes - Auto-assign reviewers - Auto-label by file paths - Auto-add to queue on approval The merge queue is one piece of a fully automated pipeline. ## Scaling Signals You need to optimize when: - **Wait times creep up** — PRs taking longer to merge each month - **Engineers complain** — "The queue is always full" - **Workarounds appear** — People merging directly to main "just this once" - **CI costs spike** — Full test suite running for every PR Address early. Small delays compound into large productivity losses. ## Example: Scaling from 20 to 100 PRs/day **At 20 PRs/day:** - Serial queue, 15-minute CI - Comfortable margin **At 40 PRs/day:** - Add batching (batch size 3-4) - Add speculative checks (depth 2) - Still one queue **At 70 PRs/day:** - Split into parallel queues by area - Increase speculation depth - Add two-step CI **At 100+ PRs/day:** - Per-team queues - Aggressive batching - Priority lanes - Real-time queue monitoring ## Key Takeaways 1. **Do the math** — Know your theoretical throughput vs. actual PR volume 2. **Use every lever** — Batching, speculation, parallelism, priorities 3. **Measure constantly** — Wait time is your key metric 4. **Optimize proactively** — Fix slowdowns before engineers feel them 5. **Automate the pipeline** — The queue is just one piece --- ## Long CI Pipelines Source: /use-cases/long-ci-pipelines/ When CI takes 30, 60, or 90+ minutes, a naive merge queue becomes a bottleneck. With 60-minute CI, you can only merge 8 PRs in an 8-hour day. That's not sustainable. ## The Problem Long CI limits queue throughput: | CI Duration | Max PRs/day (serial) | Reality | |-------------|----------------------|---------| | 15 min | 32 | Comfortable | | 30 min | 16 | Tight | | 60 min | 8 | Bottleneck | | 90 min | 5 | Unusable | Teams with long CI need strategies beyond "make CI faster" (though that helps too). ## Strategy 1: Two-Step CI [Two-step CI](/features/two-step-ci/) splits validation into fast and thorough phases: [diagram] - **PR CI**: Lint, unit tests, build — catches most issues quickly (typically 80-90%) - **Queue CI**: Integration tests, E2E, full validation — runs only for approved PRs Engineers get fast feedback. The queue runs thorough checks. Both needs met. ## Strategy 2: Batching [Batching](/features/batching/) groups PRs into single CI runs: ``` Without batching: 5 PRs × 60 min = 300 min CI time With batching: 5 PRs × 1 run = 60 min CI time ``` Batching reduces CI cost dramatically. The tradeoff: if the batch fails, you need to identify which PR caused it. ## When You Can't Speed Up CI Some CI is genuinely slow: - Hardware-in-the-loop tests - Full E2E suites against real services - Compliance scans - Performance benchmarks If you can't make it faster, use queue strategies: 1. **Split required vs. optional** — Only block on critical tests 2. **Run slow tests post-merge** — Validate before deploy, not before merge 3. **Parallelize test suites** — More runners, same wall time 4. **Cache aggressively** — Dependencies, build artifacts, test fixtures ## Configuration Example For 60-minute CI with ~30 PRs/day: | Setting | Value | Rationale | |---------|-------|-----------| | Batch size | 3-5 | Reduce CI runs, manageable failure isolation | | PR CI | 5-10 min | Fast feedback for developers | | Queue CI | 60 min | Full validation | | Priority lanes | Yes | Hotfixes skip the queue | ## Warning Signs Your queue strategy isn't working if: - **Wait times exceed CI time** — PRs waiting longer to enter queue than to run CI - **Engineers bypass the queue** — "Just this once" direct merges appear - **Batch failures are common** — Batches fail often, bisection adds delay Monitor and adjust. ## Strategy 3: Speculative Checks [Speculative checks](/features/speculative-merging/) test multiple PRs (or batches) in parallel, assuming earlier ones will pass. Combined with batching, this is the most effective way to maximize throughput with slow CI: ``` Without optimization: 5 PRs × 60 min = 300 min With batching + speculation: 2 batches × 60 min in parallel = 60 min ``` ## Key Takeaways 1. **Two-step CI is essential** — Fast PR feedback, thorough queue validation 2. **Batching reduces cost** — Fewer CI runs for same PRs 3. **Speculative checks reduce latency** — Test in parallel instead of waiting 4. **[Priority lanes](/features/priority-management/)** — Ensure hotfixes don't wait behind long-running batches 5. **Monitor queue health** — Wait time is your key metric See also: [High-Velocity Teams](/use-cases/high-velocity-teams/) for strategies beyond CI optimization. --- ## Glossary Source: /glossary/ Quick reference for merge queue terminology. ## Core Concepts ### Merge Queue A system that validates and serializes PR merges, testing each PR against its actual merge target before integration. Prevents broken main by ensuring every commit is tested against what main will look like when it lands. ### Test Branch Temporary branch created by the merge queue representing "what main will look like after this PR merges." PRs are rebased onto this branch and tested there—not against the current main. ### Queue CI Validation run when a PR enters the queue. Unlike PR CI (which tests against current main), queue CI tests against the test branch. This is the critical difference that prevents integration failures. ### Broken Main When the main branch fails tests or doesn't build. Blocks all development, deploys, and releases. The primary problem merge queues solve. --- ## Features ### Speculative Checks Testing multiple PRs in parallel by assuming earlier PRs will pass. If PR #1 is testing, PR #2 starts testing against "main + PR #1" immediately. Reduces latency by 3x or more. See [Speculative Checks](/features/speculative-merging/). ### Speculation Depth How many PRs ahead the queue will speculatively test. Depth of 3 means PR #4 waits for PR #1 to finish before starting. Balances parallelism against wasted CI when speculations fail. ### Batching Grouping multiple PRs into a single CI run. Instead of testing PRs individually, test "main + PR #1 + PR #2 + PR #3" together. Reduces CI cost but requires bisection on failure. See [Batching](/features/batching/). ### Bisection When a batch fails, the process of identifying which PR caused the failure. Often done by testing overlapping subsets: if batch [A,B,C] fails, test [A,B] and [B,C] to isolate the culprit. ### Parallel Queues Independent queues for non-conflicting changes (by directory, service, or team). Allows simultaneous merges when changes don't overlap. See [Parallel Queues](/features/parallel-queues/). ### Scope The boundary defining a parallel queue. Can be a directory path, build target, team ownership, or other logical boundary. PRs in different scopes can merge concurrently. ### Priority Management System allowing urgent PRs to jump ahead in the queue based on priority levels. Ensures hotfixes and critical changes aren't blocked behind routine PRs. See [Priority Management](/features/priority-management/). ### Freshness Policy Rules controlling how up-to-date a PR must be against main before merging. Strict freshness requires testing against the very latest main; relaxed policies allow some staleness. See [Freshness Policies](/features/freshness-policies/). ### Two-Step CI Separation of PR CI (fast feedback for developers) and queue CI (comprehensive validation before merge). Allows different test suites for each stage. See [Two-Step CI](/features/two-step-ci/). --- ## Git & CI Terms ### Stale Branch A PR branch that hasn't been tested against the current state of main. The gap between what was tested and what exists on main creates risk of integration failures. ### Rebase Replaying commits from one branch onto another. When your PR is "stale," you rebase it onto the latest main. Merge queues automate this—you don't manually rebase. ### Merge Strategies How code is integrated into main: - **Merge commit**: Creates a merge commit preserving branch history - **Squash and merge**: Combines all PR commits into one - **Rebase and merge**: Replays commits linearly - **Fast-forward**: Moves main pointer directly (no merge commit) ### Required Checks CI jobs that must pass before a PR can merge. Configured in GitHub branch protection or similar. The merge queue respects these when deciding if a PR passes. ### Flaky Test A test that passes and fails intermittently without code changes. Particularly problematic in merge queues where a flaky failure can block the entire queue. ### Post-merge CI CI that runs after code lands on main (the traditional approach). By the time it fails, the damage is done—main is already broken. --- ## Failure Modes ### Integration Failure A failure that only surfaces when multiple changes are combined. PR #1 passes alone, PR #2 passes alone, but PR #1 + PR #2 together fail. The core problem merge queues prevent. ### Semantic Conflict When two PRs are textually compatible (no git conflicts) but logically incompatible. Example: PR #1 adds a required function parameter, PR #2 calls the function without it. Git merges cleanly; tests fail. ### Cascade Failure When one failed speculation causes multiple PRs to need re-testing. If PR #2 fails, PRs #3, #4, #5 (which were speculating on #2's success) must restart. ### Queue Starvation When high-priority PRs continuously preempt normal PRs, causing regular work to never merge. Mitigated with priority decay or reserved queue slots. --- ## Metrics ### Time to Merge (TTM) Time from PR approval to code landing on main. Merge queues typically reduce this by eliminating manual coordination and rebase cycles. ### Queue Depth Number of PRs currently waiting in the merge queue. High queue depth indicates either high velocity or slow CI. ### Speculation Hit Rate Percentage of speculative checks that don't need to restart due to earlier failures. Higher is better—indicates stable queue throughput. ### Merge Throughput Number of PRs successfully merged per unit time. The primary measure of merge queue efficiency. --- ## Related Concepts ### Trunk-Based Development Development practice where all engineers commit to a single main branch with short-lived feature branches. Merge queues support this by keeping main stable despite high merge frequency. ### Feature Flags Runtime toggles that enable/disable features without code changes. Often used alongside merge queues to allow partially-complete work to merge safely. ### Monorepo Single repository containing multiple projects or services. Merge queues are especially valuable for monorepos due to high PR volume and complex interdependencies. See [Monorepos](/use-cases/monorepos/). ---