Free lesson · Git Mastery

Production Workflows Overview

Hard7 min readgit

Production Workflows Overview

A note from Swati Bajpai, Engineering Manager at Google Search Infrastructure.

When Git Operations Touch Production

Module 19 covers Git operations specific to operating production systems. The transition from development to production changes the stakes of Git operations. A mistake in a feature branch is recoverable through rebasing and force-pushing. A mistake on a production branch - a bad merge, an accidental force-push, a wrong revert - can mean an outage, data loss, or compliance issues that take weeks to recover from.

The patterns in this module aren't fundamentally different operations. The branches, commits, merges, and tags are the same as elsewhere. What changes is the discipline around them - the additional checks, the coordination requirements, the audit trails, the safety mechanisms. For teams operating systems where downtime has costs, the production discipline isn't optional.

I manage a fourteen-person team at Google Search Infrastructure. Our work touches systems that millions of users depend on; production discipline is part of how we operate. This topic frames what production-grade Git workflows mean and the categories of concern. Topics 19.2 through 19.8 cover specific patterns: hotfixes, rollbacks, releases, deployment coordination, incident response, and long-term branch management.

What Makes a Workflow "Production"

A workflow is production-grade when it accounts for the following concerns:

Reversibility. Any change that reaches production can be reversed quickly when needed. The team has tested rollback paths, not just rollforward paths.

Traceability. Every change to production code can be traced to who made it, when, why, and with what review. The Git history serves as an audit trail.

Stability. Changes don't reach production until they've passed appropriate quality gates. The bar is higher than for feature branches.

Coordination. Changes don't conflict with other ongoing work. Deployments are coordinated across teams.

Predictability. The state of production at any moment matches what the team expects. Surprises are minimized.

These concerns shape the entire production workflow. Reversibility shapes how branches are structured. Traceability shapes commit conventions. Stability shapes review and testing requirements. Coordination shapes deployment timing. Predictability shapes branch protection. The patterns in this module address each.

For teams without production-grade discipline, these concerns are aspirational. For teams with mature practices, they're encoded in tooling and process.

The Major Workflow Patterns

Three patterns dominate production Git workflows:

Trunk-based development. One main branch (often main); all changes merge there; production deploys from main. Feature branches are short-lived. Releases are tag-based.

Release branches. Main branch holds active development; long-lived release branches (e.g., release/2.4.x) hold stable production code. Hotfixes go to release branches; features go to main.

GitFlow (or variants). Multiple long-lived branches: develop for integration, main for production, release/* for stabilization, hotfix/* for urgent fixes. More complex but explicit about each branch's purpose.

Each pattern has tradeoffs. Trunk-based development is the simplest and works for teams with strong CI and frequent deployment. Release branches add complexity but support multiple supported versions in parallel. GitFlow is more structured but adds operational overhead. The choice depends on the deployment frequency, the number of versions supported, and the team's discipline.

The patterns aren't mutually exclusive. A team may use trunk-based development for most work but spin up release branches for major versions.

A Visualization

flowchart TD
    subgraph "Trunk-based"
        T1[main branch]
        T2[feature branches<br/>short-lived]
        T3[tags for releases]
        T2 -->|merge quickly| T1
        T1 -.tag.-> T3
    end
    
    subgraph "Release branches"
        R1[main: active development]
        R2[release/2.4.x: stable]
        R3[release/2.3.x: maintenance]
        R1 -->|cut release| R2
        R2 -.hotfix backport.-> R1
    end
    
    subgraph "GitFlow"
        G1[main: production]
        G2[develop: integration]
        G3[release/*: stabilization]
        G4[feature/*: in progress]
        G5[hotfix/*: urgent]
        G4 -->|merge| G2
        G2 -->|cut| G3
        G3 -->|merge| G1
        G1 -.tag.-> G3
        G5 -->|merge| G1
        G5 -->|merge| G2
    end
    
    style T1 fill:#dcfce7
    style R1 fill:#dbeafe
    style G1 fill:#fef3c7

The diagram shows the structural difference. Trunk-based has one trunk; everything flows through it. Release branches have a trunk plus stable parallel branches. GitFlow has multiple long-lived branches with defined roles. The complexity grows; the operational expressiveness grows with it.

Choosing a Pattern

The decision factors:

Deployment frequency. Daily deployments fit trunk-based; quarterly releases may benefit from release branches.

Number of supported versions. Supporting one current version fits trunk-based; supporting multiple parallel versions (common in enterprise software) needs release branches.

Team size and coordination overhead. Small teams can sustain GitFlow; larger teams benefit from simpler patterns to reduce process tax.

CI/CD maturity. Trunk-based requires strong CI to keep main always deployable; without it, regressions slip in.

Risk tolerance for production. Higher risk tolerance fits simpler patterns; lower tolerance benefits from explicit stabilization branches.

For my team at Google, we use a variant of trunk-based development with explicit release tags. Our CI is mature enough to keep main deployable; we tag specific commits for deployment. Hotfixes happen by cherry-picking to release tags when needed.

What Production Demands Beyond Standard Git

The standard Git operations work for production, but additional disciplines apply:

Required code review. Production-touching commits require review; the requirement is enforced via branch protection (Topic 20.2).

Required status checks. Tests, security scans, and other checks must pass before merge.

Signed commits or signed merges. Some teams require signing for production commits (Topic 18.6).

Restricted force-push. Force-push to production branches is forbidden or restricted to specific roles.

Audit logs. Beyond Git's commit history, organizational audit logs may track who did what.

Documentation requirements. Production changes may need linked design documents, change-control tickets, or compliance reviews.

These disciplines turn standard Git into production-grade Git. None of them change what Git does; they change what's allowed and what's required. The enforcement happens at the team level via branch protection, hooks, and review processes.

A Pattern at Google Search Infrastructure

The team's production workflow:

• Main branch (main) is the only branch that gets deployed
• All changes go through code review (minimum 2 reviewers for production-touching changes)
• All commits must pass CI (build, tests, security scan, lint)
• Direct push to main is forbidden; everything goes through PR
• Specific commits are tagged for deployment (prod-2024-11-15-abc123)
• Deployments use tags, not branch tips
• Rollback uses earlier tags
• Long-running feature work uses feature branches that rebase against main

The workflow is deliberately simple. One protected branch, one path to production, one set of conventions. Engineers don't have to track multiple long-lived branches. New engineers learn the workflow within their first week.

The simplicity has costs - we can't support multiple parallel versions easily. For our use case (a single production system), that's acceptable. Teams supporting multiple versions need more complex patterns.

A Specific Story From My Stripe Days

Around 2018 at Stripe, I worked on a team that used a complex variant of GitFlow. We had main, develop, release branches, feature branches, hotfix branches - all with specific rules about when each was active.

The complexity had grown organically. Each branch type had been added to solve a specific problem. By the time I joined, the workflow had eight kinds of branches and a 20-page wiki explaining when to use each.

The cost was visible: new engineers spent weeks learning the workflow. Mistakes were common ("I merged to develop instead of main"). The process took disproportionate engineering time.

We simplified over about six months. Eliminated develop (merged into main). Eliminated release branches (used tags instead). Kept hotfix branches for urgent production fixes. The workflow shrank to three branch types with clear rules.

The simplified workflow was just as effective. We deployed reliably; we tracked production state precisely; we recovered from issues quickly. The complexity hadn't been buying us anything; it had been a process tax.

The lesson: production workflows benefit from simplicity. Add complexity when there's a specific problem to solve, not because more complexity seems more "professional." Each branch type, each rule, each process step has an ongoing cost.

Common Failure Modes

The patterns that signal production workflow problems:

Engineers bypassing the workflow. "I had to push directly because the process was too slow." Signals the process needs streamlining.

Production state unclear. Nobody knows exactly what's deployed. Signals missing tooling or convention.

Frequent hotfixes. Indicates regressions making it to production. Signals quality gate problems.

Long lead time from commit to production. Slows feedback. May or may not be appropriate depending on context.

Frequent rollbacks. Indicates production receiving changes that aren't ready. Signals testing gaps.

Inconsistent enforcement. Some branches protected, some not; some commits reviewed, some not. Signals governance gaps.

Each signal prompts investigation. The fix is usually local (better tooling for one specific issue), not a wholesale workflow change.

What Comes Next

Topic 19.2 covers hotfix patterns in detail. Topic 19.3 covers rollback strategies. Topic 19.4 covers tag-based releases. Topic 19.5 covers release branches. Topic 19.6 covers deployment coordination. Topic 19.7 covers incident response using Git. Topic 19.8 covers long-term production branch management.

What Goes Wrong

Warning: Common production workflow mistakes:

Choosing complex patterns without need. GitFlow when trunk-based would suffice.

No branch protection on production branches. Direct pushes; unreviewed changes.

No tooling for rollback. Forward-only; can't recover quickly from issues.

Unclear production state. Nobody knows what's deployed.

Process bypass becomes routine. Process exists but isn't followed.

No deployment coordination. Multiple teams stepping on each other.

Try It

Exercise: Inventory your production workflow:

1. List the long-lived branches in your team's main repository. What's each one's role?
2. Identify which workflow pattern (trunk-based, release branches, GitFlow) most closely matches yours.
3. List the safeguards on production-touching changes. What's enforced; what's convention?
4. Time how long it takes a change to go from commit to production. Note the bottlenecks.
5. Identify one place where the workflow has complexity without clear value.
6. Discuss with your team: what would simplify the workflow without losing safety?

Back to Git Mastery