This is part of our ongoing series about Agile in practice. You can get up to speed on where this post by starting [where we began](/blog/agile-where-we-started). 

## Issue

Nuve was fresh off a win [fixing the vast majority of our release quality issues](/blog/release-quality-issues). Our QA environment was a great end-to-end testing ground.

But one day we had a bug slip through that end-to-end QA process. On top of that, a developer on the team had already merged a new feature int our QA environment that wasn’t fully tested. That meant we couldn’t make the fix on QA and release immediately to production - there was too much risk that the new code on QA could lead to new bugs.

We needed a way to release a fix to production bypassing our normal QA process.

## Solution

### Agile Brainstorming: Options for Solving

**Possibility 1: Make a fix on QA, Test, and Pluck that Commit into Production**

This is a promising idea. Theoretically you can make a fix and move that fix into QA and then pluck just that change from QA into production, ignoring all other changes on QA.

In practice, though, it’s kind of messy to do that. Not a lot of developers - even very good ones - know how to pluck just one or two commits from one branch into another. It’s a non-standard workflow with lots of room for error. Also, the more attempts there are at a fix (i.e. the more commits there are) the more complicated this pattern becomes.

We wanted something a little bit more foolproof when working with production so we passed on this idea.

**Possibility 2: Hotfix Workflow - Fix on Production, then Move those Changes into QA**

This idea is exactly what it sounds like. We create a feature branch off of `main` (production - like we used to when we used the GitFlow workflow). Then make the changes we needed to make to fix the bug. Then we merge the feature branch right back into our `main` branch AND into the `develop` (QA) branch when we prove the fix works.

We only do that for speed when we know that there is a bug on production and we can’t do a full release from QA to production. It’s like a safety release valve for making fast fixes.

This was the process we chose to do.

### Implementation: The Workflow

Because we had separated our QA (`develop`) and production (`main`) branches, we could do the following:

- Create a branch `hotfix-xyz` off of `main`
- Make the fixes we need to make
- Merge `hotfix-xyz` back into `main`
- Deploy to production and test that it works (repeat if not)
- Merge `hotfix-xyz` into `develop` once it works as expected

At the end of that process, both `develop` and `main` (QA and Production) have the fix and we didn’t risk merging anything from QA to production that wasn’t fully tested. We are confident about this because `develop` never merged into `main` !

Further, we always were doing the same simple deploy process

- On production - pull code from `main`
- On QA - pull code from `develop`

Just like any other release. No magical git commands required.

## Results

Because there was little overhead associated with fixing production and the flow was simple, everybody on the team could do a hotfix and we knew we could do it quickly. We saw this in results. Our average time from production bug discovery to fix dropped from several hours to 30 minutes after this change was implemented.

But arguably the biggest win with this process was that it inspired confidence. We knew that our QA environment (`develop` branch) was open for new features as soon as we completed a full release to production because we could always hotfix production if necessary.

We rarely hotfix because our QA process is strong - there have been 3 or 4 hotfixes in the last several months - but knowing that we can hotfix enables us to work quickly in QA.


Inspiring confidence by allowing quick production fixes.

Adding a hotfix workflow to our toolkit inspried confidence in both our development workflow as well as our release procedures.

Peace of mind with hotfixes

Related posts

Empowering releases

Peace of mind with hotfixes

Release big stuff in chunks