I was tasked with containerizing our Bitbucket instance at my job. The migration went well, but what followed wasn’t quite as smooth. A sneaky, unexpected issue popped up—here’s what happened.

Bitbucket and its underlying infrastructure needed an upgrade. We wanted to get the VM to an up-to-date OS and containerize the application. This was a very exciting opportunity for me, as this was one of my first major projects. When I got the assignment, I hit the ground running. I set up a test instance, which gave me a ton of confidence going into the actual migration. The plan of the migration was as follows:

• A week before the migration, create the new Bitbucket virtual machine.
• Install Docker
• Create the Docker Compose with the necessary services and configurations
• Run rsync to get an initial copy of the Bitbucket data.
• On the night of the migration, run rsync again with the --checksum --itemize-changes

Missing Commits

The actual migration seemed to have gone smoothly. I did some validation (ensuring I could push and pull, etc.), and everything looked good. The next day, all still seemed to be going well. I was monitoring the logs and could see user activity. Around lunchtime, one of the developers messaged me to say they noticed missing commits from the day of the migration. My heart sank. "How could anything be missing?" I thought to myself. Could this be affecting other teams? A few hours later, another developer opened a ticket complaining of the same issue. I immediately informed my manager of the situation and assured him that I was actively looking into it.

So What Happened?

After a few hours of panic and googling, I stumbled upon this bitbucket article that lined up exactly with my issue. Here's the key:

While running this sequence of commands, the --delete rsync option is not used so only changes to existing files and new files are synchronized. However, files that have been copied during the earlier rsync are not deleted and this causes unexpected conflicts in the Git repositories.

This was my scenario, exactly. During the night of the migration, when running the final rsync, I never used the --delete flag. Here's what the --delete option does. From manpage:

--delete tells rsync to delete extraneous files from the receiving side (ones that aren't on the sending side), but only for the directories that are being synchronized.

Basically, anything that exists on the destination but not on the source will be deleted. The above was eye-opening, but left me confused. "Between my initial rsync and the final one, what files were being deleted?" Let me explain what I'll call "Git Housekeeping"

Git Housekeeping

In Bitbucket, each branch has a file like this :
$REPOSITORY_HOME/refs/heads/feature/my-branch

This file contains the commit hash at the tip of each branch. Git can combine many individual ref files into a single file like this: $REPOSITORY_HOME/packed-refs

At certain intervals, Bitbucket runs git pack-refs --all. We'll call this bitbucket housekeeping. This takes branch tips from refs/heads/* and writes them to packed-refs. This leaves the file containing the refs stored in $REPOSITORY_HOME/refs/heads/feature/my-branch empty.

If the first rsync has been performed before the pack-refs run and again after that, without the --delete option, both the $REPOSITORY_HOME/refs/heads/feature/my-branch and the $REPOSITORY_HOME/packed-refs file will be present in the target filesystem. When that happens, git recognizes the hash in $REPOSITORY_HOME/refs/heads/feature/my-branch as the tip of the branch, but this is now outdated and leads to what is described as missing commits.

Step 1 – First rsync (BEFORE pack-refs)

refs/heads/feature/my-branch  →  abc123
packed-refs                  →  (does not contain this branch yet)

Step 2 – Bitbucket runs pack-refs

• packed-refs now contains:

abc123 refs/heads/feature/my-branch

Final rsync (AFTER pack-refs, without –delete)

refs/heads/feature/my-branch  → OLD commit hash
packed-refs                  → NEW commit hash

Getting Things Back To Normal

The good news was that no commits were actually lost. Everything was still on disk; it just wasn’t being referenced correctly. The challenge now was figuring out how to get things back to a normal state. Around this time, a couple of interesting things happened. Interesting things, that I attribute to divine intervention 😅.

The first developer who reached out about the missing commits followed up and mentioned they had local backups they restored to recover their commits.

Shortly after that, the 2nd developer who reached out followed up and said, “Did you do something? Our commits are back.” I hadn’t changed anything. What I suspect happened is that Bitbucket ran its internal housekeeping, specifically git pack-refs --all , and cleaned up the stale references on its own, allowing Git to resolve the correct commit pointers again.

Experiences like this are never fun, but are par for the course in this industry. Things will go wrong. What matters is how you respond and the lessons you learn, so you can avoid the same mistake in the future. After a few more days of no more "My commits are missing" tickets, I was able to breathe a little easier 😄