Post-Production Editing using Git

Production

Pre-Post-Production

After you have produced the commits representing the code you wish to perfect, there are a few things you need to do before moving into post-production.

Record important SHA

There are two SHA which are useful to the post-production process.

The SHA before the first commit you want to rewrite

Knowing a ref to the commit before the first commit you want to rewrite is absolutely critical. This ref must be passed in to several of the commands you will be using. You can find this commit using the normal techniques, including git log --oneline and gitk (remembering that if you know the SHA of the first commit you want to rewrite, you can find the SHA of the commit before that with git rev-parse SHA^—replacing the string "SHA" with the SHA of the first commit you want to rewrite, leaving "^" as literal). While you can sometimes get away with leaving SHA^ there instead of dereferencing that commit, you absolutely cannot safely and generally use a symbolic reference (like HEAD or master) in place of SHA in "SHA^" in the commands listed below, so it is just safer all around if you find the true SHA of the first commit's parent.

However, if you have followed the production guidelines above and you are working with a standard central repository distributed repository model, then there is already a convenient ref pointing to the very commit you are interested in. Specifically I am talking about "@{u}" (added in git v1.7.0) which is the pointer to the SHA of the upstream branch. If you have set your upstream branch, pulled with "--rebase", and not done any merge commits, then this ref will always point to a commit before all of the commits you want to/is safe to rewrite.

In my examples below I will be using "@{u}". So if you need to use a different SHA or ref, please substitute it as needed.

The SHA of the your last commit

If you decide that you made a horrible mistake during the post-production process, your original work is still around and accessible (with enough effort). However, recording the SHA of your most recent commit SHA might reduce that required effort. git rev-parse HEAD

Again, it isn't like this data isn't stored in git, but having to grope through the reflog can be tedious. You will have to decide whether recording these SHA is more tedious than the odd reflog dive.

Backups

Production of test repository

In order to create an example to work through, we will be creating a test configuration which we will be sending through post-production. There is no doubt that this example is extremely synthetic, but for all of that, it should more or less replicate the processes of what you will do in real life. Please feel free to following along by copy-pasting (not retyping, copy-pasting) these commands into a convenient shell.

cd /tmp; rm -rf gpp-[yz]; git init --bare gpp-z; git clone gpp-z gpp-y;
cd gpp-y; for f in A B C D; do echo A > $f; done; git add .
git commit -m "Initial structure"; git push -u origin master

Here we have an upstream repository "gpp-z" and a local repository "gpp-y" and we have created an initial commit with the letter A in four different files named "A" "B" "C" and "D". This represents the prior existing state of the universe before a commit series. Now we will go ahead and create the commit series that we will be putting through post-production.

for f in A B C D; do echo $f$f >> $f; done; git commit -am "Add doubles"

echo AAA>>A; echo AAAA>>A; git commit -am "Fill out A series"

echo BBB>>B; echo BBBB>>B; for f in A B C D; do sed -i "s/A/$f/g" $f; done
git commit -am "Fill out B series plus bugfix to initial commits"

echo CCC>C; echo CCCC>C; git commit -am "Fill out C series"

echo DDD>>D; echo DDDD>>D; (echo С; echo СС; echo ССС; echo СССС)>C;
git commit -am "Fill out D series and fix C series"

After this series of five commits, we end up with A AA AAA AAAA in file A, B BB BBB BBBB in file B, etc. However we got here by a somewhat convoluted path where in some commits we made changes breadth first, in others depth first, we fixed a bug made in the commits before this commit series, and we fixed a bug made during this commit series.

After reviewing the changes we made, we have decided that our goal for this commit series is to separate out the bugfix to previous commits in a commit by itself and choose to make one commit to fill out each letter in their own commit without any (known) bugs. We will end up with the same number of commits, but the commits will be much simpler to understand, without bugs which might confuse git-bisect, and cherry-pickable.

Post-Production

Now you should be finally ready to start performing post-production editing of your unpushed commits in your repository. The technique I will be espousing here is either DIVINE (DIvide, INtegrate, and Evaluate) or SATAN (Split, Arrange, Team, Analyze, and Narrate) depending on your point of view.

Contrived acronyms aside, the process is fairly straightforward.

Splitting or DIviding commits

First we need to divide the commits representing multiple concepts (or a bugfix and a commit or other containing multiple things you want in separate commits).

The command you execute to begin the process of splitting your commits is very simple: git rebase -i @{u} (remember you can replace @{u} with the SHA of the commit before the first commit you want to edit in post-production if necessary).

This will bring you in an editor showing you a list of commits (with parents above children) available for modification, something like the following if you were using the provided example configuration (note the 7 character abbreviated SHA will be different for you):

pick d6d2951 Add doubles
pick d4803e2 Fill out A series
pick aacc018 Fill out B series plus bugfix to initial commits
pick ff2b343 Fill out C series
pick ba253fb Fill out D series and fix C series

There will also be some additional information commented out. You need to change the word "pick" to "edit" for each commit which (might) need to be split. In the example above, we can easily see that the third and fifth lines need to be split, so we could change those lines from "pick" to "edit". Since we want to separate each file into its own commit, we also need to edit the first commit. However, if we happened to not remember whether the second commit needed to be split or not, we can go ahead and edit that as well. The resulting lines would look something like:

edit d6d2951 Add doubles
edit d4803e2 Fill out A series
edit aacc018 Fill out B series plus bugfix to initial commits
pick ff2b343 Fill out C series
edit ba253fb Fill out D series and fix C series

After saving the file and exiting your editor, the system will bring you to the first commit you asked to edit:

Stopped at d6d2951... Add doubles
You can amend the commit now, with

    git commit --amend

Once you are satisfied with your changes, run

    git rebase --continue

Since we want to make the modification to each file a commit by itself, we need to peel off the "Add doubles" commit (leaving the working directory with the results of the commit in it) and turn the single commit into four distinct commits:

Special Warning! We use git reset --mixed HEAD^ a lot in this document to peel off the last commit so that you can split the commit. However, please remember that if you added a file during the commit you just removed, it will NOT be left as staged so a git commit -am "foo" will not cause that file to be re-added. Similarly, if a file was deleted in the commit you just peeled off, the deletion will also not be staged. In both cases, you would need to manually git add filename or (more dangerously and only when you know what the status is) git add -A .. You should always inspect git status after your reset and when you think you are done to validate that the system is in the state you think it is.

git reset --mixed HEAD^
git status
git add A
git commit -m "Add double As"
git add B
git commit -m "Add double Bs"
git add C
git commit -m "Add double Cs"
git add D
git commit -m "Add double Ds"
git status

Everything looks good, so we can tell git to proceed to the next commit with git rebase --continue

Stopped at d4803e2... Fill out A series
You can amend the commit now, with

    git commit --amend

Once you are satisfied with your changes, run

    git rebase --continue

At this point, you can git show HEAD or git diff HEAD^ or otherwise investigate the commit to see if you need to split it or not. In this case, not so much, so we go ahead and tell git we are satisfied with git rebase --continue.

It will now bring us to the third commit we asked to edit:

Stopped at aacc018... Fill out B series plus bugfix to initial commits
You can amend the commit now, with

    git commit --amend

Once you are satisfied with your changes, run

    git rebase --continue

This time, investigation shows that we have a bugfix to prior commits and new features intermingled in the same commit. You can fix this any way you want, but this is one simple method: first we peel off the current commit, leaving the working directory the same. Now running git status or git diff shows all of the changes from that commit in our working directory, ready to be added. In this example, we'll make the bugfix commit first and the new feature commit second (though it could be done in the other order).

git reset --mixed HEAD^
git status
git add C D
git add -p B

This may be the first time you have heard of the git add -p option, but it is a very powerful method of dividing independent changes made to the same file. If you are following along, you should see something like:

diff --git a/B b/B
index 0dc9441..921f5d2 100644
--- a/B
+++ b/B
@@ -1,2 +1,4 @@
-A
+B
 BB
+BBB
+BBBB
Stage this hunk [y,n,q,a,d,/,s,e,?]?

You can go ahead and type ? for more information, but in this case we want to split this particular "hunk" (or change component) into multiple changes, so we will go ahead and type "s" here. In your own endeavors, you might need to use any one of those options, but please be very careful with "e" since editing diff files is typically extremely tricky. After typing "s" we should see something like:

Split into 2 hunks.
@@ -1,2 +1,2 @@
-A
+B
 BB
Stage this hunk [y,n,q,a,d,/,j,J,g,e,?]?

This is the bugfix change we want, so we go ahead and type y here. We then see the other half of the change, which in this case represents the new feature, so we can type "n" or "q" here. In any case, we can review our git status and our staged change git diff --cached.

# Not currently on any branch.
# Changes to be committed:
#   (use "git reset HEAD <file>..." to unstage)
#
#       modified:   B
#       modified:   C
#       modified:   D
#
# Changes not staged for commit:
#   (use "git add <file>..." to update what will be committed)
#   (use "git checkout -- <file>..." to discard changes in working directory)
#
#       modified:   B
#

diff --git a/B b/B
index 0dc9441..0b1bb71 100644
--- a/B
+++ b/B
@@ -1,2 +1,2 @@
-A
+B
 BB
diff --git a/C b/C
index edc7dd2..21c9dd3 100644
--- a/C
+++ b/C
@@ -1,2 +1,2 @@
-A
+C
 CC
diff --git a/D b/D
index 71dbedb..7f07dc1 100644
--- a/D
+++ b/D
@@ -1,2 +1,2 @@
-A
+D
 DD

Everything looks good for the bugfix, so we can go ahead and commit. Be sure not to accidentally add the "-a" flag to git-commit here! git commit -m "Bugfix to initial commits"

Now, looking at git status shows that we still have uncommitted changes left. Investigating a git diff shows that we don't need to further split the commit and can go ahead and commit the remainder of the code, git commit -am "Fill out B series", and then tell git to proceed to the next patch: git rebase --continue

Stopped at ba253fb... Fill out D series and fix C series
You can amend the commit now, with

    git commit --amend

Once you are satisfied with your changes, run

    git rebase --continue

Investigation shows that we need to split this into two commits, one for Ds, and the other for Cs. We can go ahead and do that.

git reset --mixed HEAD^
git status
git add D
git commit -m "Fill out D series"
git add C
git commit -m "Fix truncation problem in C series"
git status

Yes, I know --mixed is the default for git-reset. I like being precise with git-reset since it is one of the more dangerous git commands. Anyway, this looks all good, so we can tell git to proceed: git rebase --continue

Successfully rebased and updated refs/heads/master.

So now we can review where we currently are with git log --oneline @{u}..

64a7ac6 Fix truncation problem in C series
bee967d Fill out D series
c4a1c0d Fill out C series
2186302 Fill out B series
3ee2f5e Bugfix to initial commits
3453b2f Fill out A series
3a946cc Add double Ds
c1ee9e3 Add double Cs
b9c39f0 Add double Bs
f45017e Add double As

Well, we certainly seem to have successfully divided. Our five commits have been turned into ten. But this is certainly not what we want the public to see, so we press on.

Please note if you commit early and often enough, you get to skip this step entirely since your commits would never have multiple things in them which need separating. This is a special bonus for those who follow that practice.

Arrange commits

Next we reorder the commits so that commits you want to merge a next to each other; for example, a bugfix made to a bug you introduced in this series of commits is next to the commit which introduced the bug.

Unlike the previous step, this step can be a bit tricky since reordering the commits may introduce dependency problems which will cause rebase conflicts. Sometimes you can do some creative ordering to fix this problem, sometimes you are just out of luck.

Unfortunately, the example we have put together is going to run into this problem since the bugfix to the initial commits occurred after the commits to add the doubled letters and we want to assemble all of the new features for a specific letter to be in a distinct commit. Thus no matter how we order those commits, we are going to get a (well more than one honestly) conflict. Perhaps we can think of it as a teachable happenstance.

To start the reordering process we once again run git rebase -i @{u}

pick f45017e Add double As
pick b9c39f0 Add double Bs
pick c1ee9e3 Add double Cs
pick 3a946cc Add double Ds
pick 3453b2f Fill out A series
pick 3ee2f5e Bugfix to initial commits
pick 2186302 Fill out B series
pick c4a1c0d Fill out C series
pick bee967d Fill out D series
pick 64a7ac6 Fix truncation problem in C series

Now we get to sort things around to make everything line up better. Remember that lines appearing first in the file get applied before lines appearing subsequently. We've decided that the following commit series makes the most sense and is the most useful to potential future users (since being able to cherry-pick the bugfix without getting a merge conflict due to the new features is very convenient).

pick 3ee2f5e Bugfix to initial commits
pick f45017e Add double As
pick 3453b2f Fill out A series
pick b9c39f0 Add double Bs
pick 2186302 Fill out B series
pick c1ee9e3 Add double Cs
pick c4a1c0d Fill out C series
pick 64a7ac6 Fix truncation problem in C series
pick 3a946cc Add double Ds
pick bee967d Fill out D series

We save, exit the editor, and watch rebase go to work. Unfortunately, as predicted, we run smack into a conflict right out of the gate. Annoying, but I'm sure it will make us better human beings.

error: could not apply 3ee2f5e... Bugfix to initial commits

When you have resolved this problem run "git rebase --continue".  If
you would prefer to skip this patch, instead run "git rebase --skip".
To check out the original branch and stop rebasing run "git rebase
--abort".  Could not apply 3ee2f5e... Bugfix to initial commits

You can go through the normal conflict resolution process. Call git mergetool or manually edit the files in question. Unfortunately, in this particular example, the simple resolution process of picking one alternative or the other simply isn't going to work. Instead we need to edit each file so that the appropriate letter is the one and only letter in each file. For this particular resolution process I'm going to cheat a bit and just blow in the correct values.

git status
for f in B C D; do echo $f > $f; done
git status
git diff
git add .

We can tell git to proceed with the normal git rebase --continue. It brings us into an editor to update the commit message and after exiting that, starts applying the next commit. This actually works, as does the one after that. Unfortunately, the third commit runs into another conflict.

[detached HEAD aa62f4a] Bugfix to initial commits
 3 files changed, 3 insertions(+), 3 deletions(-)
error: could not apply b9c39f0... Add double Bs

When you have resolved this problem run "git rebase --continue".  If
you would prefer to skip this patch, instead run "git rebase --skip".
To check out the original branch and stop rebasing run "git rebase
--abort".  Could not apply b9c39f0... Add double Bs

Once again picking neither option will generate the correct file. You can use whatever technique you want, but (in this case) you want to generate a file with B on the first line and BB on the second. I'll use a slightly unusual technique in this example (chosen to feature some unusual commands and because it is automated).

git checkout --theirs -- B
sed -i 's/A/B/' B
git add B

You can proceed with git rebase --continue but…if you wanted to you could actually make a commit here and then proceed. git commit -m "Add double Bs"; git rebase --continue and it works just as well. And by just as well, I mean either choice dumps us into another conflict (after successfully applying one commit without a conflict).

This is exactly the same as the previous conflict, just on another file. I'll use another resolution sequence for variety.

git checkout --ours -- C
echo CC >> C
git add C

Promenade forth with git rebase --continue and as you probably expected, we get another conflict. We've had a lot of conflicts on this example, but this is actually pretty rare in real world applications. It is just that we made two distinct commits affecting every file and even worse, essentially affecting the same line in every file, and need to reverse the order of application of those commits in the pursuit of outer beauty. Normally the changes would be distinct enough that you would not get all of these conflicts. I'm not sure how much I recommend the technique I use here to fix the conflict, but nothing else sprung to mind:

git show master:D | head -n 2 > D
git commit -am "Add double Ds"
git rebase --continue

Ah, finally we are free of the conflicts and everything is committed and git log --oneline @{u}.. reports them in the correct order (yes the order is reversed from the editing session in git rebase -i; live with it).

Team commits

The next step is to squash the related commits together. Actually, we could have done this step and the previous step in one combined INtegration step, but doing it separately might lead to more easily understood conflicts, if you were perchance to encounter conflicts. Not that this would happen to anyone here. In any case, we once again run git rebase -i @{u}

pick aa62f4a Bugfix to initial commits
pick 143d9d5 Add double As
pick f3398c8 Fill out A series
pick a64f722 Add double Bs
pick a9dcb4d Fill out B series
pick 37e842b Add double Cs
pick bacc190 Fill out C series
pick d1a9c9c Fix truncation problem in C series
pick 4cd3178 Add double Ds
pick f8d3aec Fill out D series

We tell the system to squash the related commits together, ensuring that the more recent commits (further down in the file) are the ones which get marked for squashing.

pick aa62f4a Bugfix to initial commits
pick 143d9d5 Add double As
squash f3398c8 Fill out A series
pick a64f722 Add double Bs
squash a9dcb4d Fill out B series
pick 37e842b Add double Cs
squash bacc190 Fill out C series
s d1a9c9c Fix truncation problem in C series
pick 4cd3178 Add double Ds
s f8d3aec Fill out D series

As you can see, I got tired of writing out "squash" and switched to the abbreviation of "s". Saving and exiting gives us…an editor. Any time you are squashing commits together, git makes you create a unified commit message which describes all changes. In this case, I'm just going to change the message to/select "Fill out . series". Save and exit. We will get four editing sessions for the four new combined commits which we are generating. Note that we do not see an editor for the "Bugfix to initial commits" commit since we did not squash anything into that commit.

Reviewing the current state of the commit series with git log --oneline @{u}.. shows that we are in the correct order.

15c71aa Fill out D series
f92b407 Fill out C series
2d5ffb4 Fill out B series
cbc6b7b Fill out A series
aa62f4a Bugfix to initial commits

"Everything look good! Let's push that puppy!" Not so fast cowboy. There are still two steps left in this process. Sure you could skip them, but you might regret it in the fullness of time.

Analyze commits

Penultimately, we go through each commit ensuring it compiles and is otherwise tested. If you find errors at this step, this could involve creating new commits, fixing old commits, or otherwise repeating this process.

Most likely the easiest way to do this is to…once again…run git rebase -i @{u}. This time you will want to change all of the "pick" messages to "edit".

edit aa62f4a Bugfix to initial commits
edit cbc6b7b Fill out A series
edit 2d5ffb4 Fill out B series
edit f92b407 Fill out C series
edit 15c71aa Fill out D series

You will be dumped into the state of the tree for each commit. You can do ahead and compile, run regression tests, or do whatever else is needed to validate your commits. When you are done validating a particular commit, go ahead and run git rebase --continue to proceed. Please note that you might also want to run git clean -ndx (and if that returns OK, replace the "-n" with "-f") to prevent work product from one test from contaminating a subsequent test.

Since we don't have a compilation phase for this repo or a regression test, you might be tempted to skip this step. Well, that is no excuse (or perhaps it is an excuse to add such things). For instance, take a closer look at the state of the repository when we are at the "Fill out the C series" stage. Don't those Cs in the C file look a little...weird? (Assuming you copy-pasted my instructions faithfully, of course.) They look Cish, but not quite right. Hmm. How can we tell what is going on here. Perhaps od -t x C (if you have that program).

0000000 d0 a1 0a d0 a1 d0 a1 0a d0 a1 d0 a1 d0 a1 0a d0
0000020 a1 d0 a1 d0 a1 d0 a1 0a

d0 a1 0a? Well 0a is clearly newline (\n), but what is that d0a1 about? Further investigation by someone sufficient aware of recent advances in internationalization might show that d0a1 is the UTF-8 representation of the unicode codepoint U+0421, which represents the Cyrillic capital letter ES which happens to look almost identical to the ASCII letter C (UTF-8 43). Well, in any event the point we were trying to make here is that some bugs can be subtle and looking right is no replacement for actual testing. We can go ahead and fix this right up.

sed -i 's/./C/g' C
od -t x1 C

This reveals that my sed is UTF-8 aware (quite a pleasant surprise actually) and that our files now contains the expected ASCII-C characters.

0000000 43 0a 43 43 0a 43 43 43 0a 43 43 43 43 0a

Good thing we did these final checks, neh? Now that this passes our quality inspection, we can go ahead and git commit --amend and git rebase --continue Pressing on, we do our quality inspection of the D commit (noting that the C file has properly been updated). Does that D look normal? Really? Yup. We can do our final git rebase --continue and we are complete with the quality analysis stage.

Narrate commits

Finally, we go through each commit to perfect the commit message, to ensure it says everything that needs to be said. You could go through the analyze and narrate steps at the same time in one Evaluation step, but doing it separately leads to better contrived acronyms, I mean leads to better processes since you are not trying to remember to do two things at the same time.

For the last time, run git rebase -i @{u}. This time, change the "pick"s to "reword"s.

Post-Post-Production

Final notes

This process produces a lot of invisible kruft/chaff. You can see this with the command gitk --all --date-order $(git log -g --pretty=%H) You get to see the full history of all of the intermediate states of your repository. I'm here to tell you to not worry about it. This stuff is good disaster recovery fodder and if you don't do anything to it, it will go away in the fullness of time. In any event, no-one else will see it.

Disclaimer

Information is not promised or guaranteed to be correct, current, or complete, and may be out of date and may contain technical inaccuracies or typographical errors. Any reliance on this material is at your own risk. No one assumes any responsibility (and everyone expressly disclaims responsibility) for updates to keep information current or to ensure the accuracy or completeness of any posted information. Accordingly, you should confirm the accuracy and completeness of all posted information before making any decision related to any and all matters described.

Copyright

Copyright ⓒ 2012 Seth Robertson

Creative Commons Attribution-ShareAlike 3.0 Generic (CC BY-SA 3.0) http://creativecommons.org/licenses/by-sa/3.0/

OR

GNU Free Documentation v1.3 with no Invariant, Front, or Back Cover texts. http://www.gnu.org/licenses/fdl.html

Thanks

Thanks to the experts on #git, and my co-workers, for review, feedback, and ideas.

Comments

Comments and improvements welcome.