---
title: Backup archives migration to Borg
date: 2019-11-21 18:30:00
---

Last weekend I found a number of encrypted hard-drives that were used to do periodic
backups from 2006 to 2014. At the time, the backups were made using rsync
with hard-links to save only one occurrence of the file if it was not changed
since the last backup.

I wanted to check that everything was still there and upgrade this to a
Borg repository so that I can profit from compression and deduplication
to reduce these backup size further down and store them in a more secure way.

## Check the backups
The backups were made using hard-links with one backup corresponding to
one folder as follow :

```
$ ls backups/feronia/
back-2014-06-19T19:05:10/  back-2014-10-10T07:30:00/
back-2014-12-24T14:34:44/  current@
```

To check that the backups were still readable, I listed the content of
the different folders and checked that some known configuration files were
present and matched what was expected. This worked until I processed some
backups done before I was using awesomewm, when I changed a lot of config
files to match my usage instead of using the default ones.

All in all, the backups were still good and readable, I could use these
as a basis for the transition to a more robust and space-efficient system.

I saw a number of freezes during the check I interpreted as signs of old
age for the spinning rust.

## Initialize the Borg backup
The first step is to initialize the borg repository. We will put it on
one of the known good backup drive that still has some room. To estimate
the space needed for the backups, I took the size of the most recent backup
and multiplied by two, as I know that I did not delete a lot of files and
that the deduplication will reduce the size of the old backups that contained
a lot of checked-out subversion repositories.

So, with a destination for my borg repository, I created a folder on the
disk and gave my user read-write rights on this folder.

```
$ sudo mkdir backups/borg-feronia
$ sudo chown 1000:1000 backups/borg-feronia -R
```

Then, the creation of the repository with borg :

```
$ borg init --encryption=repokey backups/borg-feronia
Enter new passphrase: 
Enter same passphrase again: 
Do you want your passphrase to be displayed for verification? [yN]: n
[...]
```

I decided to use the `repokey` encryption mode. This mode stores the key
in the repository, allowing me to only remember the passphrase and not having
to worry about backuping the key file.

## Transfer the existing backups to Borg
The borg repository has been initialized, we can now start migrating the
backups from the hard-linked folders into borg.

As borg does not care about hard-links, we can simply loop over the different
folders and create a new archive from it. It will take some time because
in each directory it will loop over the whole content, hash it, check whether
it changed, deduplicate it, compress it and then write it. Each backup
of approximately 70 GiB took one hour to migrate on my computer. It seems
that the process is limited by the single-thread performance of your CPU.

```
$ export BORG_PASSPHRASE=asdf
$ for i in back*; do \
    archivename=$(echo $i | cut -c 6-15); \
    pushd $i; \
    borg create --stats --progress ~/backups/borg-feronia::$archivename .; \
    popd; \
    done;
```

The env variable will allow us to walk away at this stage and let the computer
do its magic for some hours.

## Check the migrated backups
Once the backups have been migrated, we need to check that everything is
in order before doing anything else.

I did the same as before, using this time `borg list` and `borg extract`
to check whether the files are present and their content is correct.


## Archive these backups
Once the migrated backups have been tested, we can shred the old hard drives
that were showing signs of old age.

Since storage is so cheap nowadays, I will also transfer an archive of
the Borg backup folder to an online storage service as to be able
to retrieve it in case the local storage supports are destroyed or otherwise
unreadable in the future.

I choose to simply create a tar archive of the Borg folder and upload it
to AWS S3 since these backups will not be updated. Perhaps some day I will
add the more recent backups to this setup but for now they are a read-only
window into the laptop I had during my studies and during my first jobs.
