Category Archives: Shell scripting

Duplicity to backup safely your data

A good backup tool ? Duplicity

I’ve tried many tools to backup, from bare rsync to tools such as restic or borg. Although they’re all good solutions, my requirements were never fully met. It’s why I chose to test and use duplicity. In this post I’ll focus on full server backup.

Here are my requirements for a good backup tool:

  • compression: I don’t have much to backup but I don’t want to use much either.
  • encryption: I only have a basic FTP that I don’t own, there it’s using GnuPG.
  • incremental backup: Also to save disk space.
  • on a FTP: with restic or borg directly I had lock problems over a curlftpfs mount, any backup would fail, it’s not related to these tools but to my usage.
  • easy to use and restore, it keeps ownership and rights.

Install duplicity

It’s pretty straightforward, install the duplicity package through your package manager on your server. You may also install ncftp if you want to backup over FTP.

# apt-get install duplicity ncftp
Install duplicity and ncftp

Generate your keys

Let’s generate your GNuPG key pair so we can encrypt our backup. This time, on your own computer, and not on your server, do the following.

# gpg --gen-key
Generate your GnuPG key pair

Take a 4096 bits length key, most of the default values should be enough.

# gpg --gen-key
gpg (GnuPG) 1.4.20; Copyright (C) 2015 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

gpg: keyring `/home/floreo/.gnupg/secring.gpg' created
gpg: keyring `/home/floreo/.gnupg/pubring.gpg' created
Please select what kind of key you want:
   (1) RSA and RSA (default)
   (2) DSA and Elgamal
   (3) DSA (sign only)
   (4) RSA (sign only)
Your selection? 
RSA keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048) 4096
Requested keysize is 4096 bits
Please specify how long the key should be valid.
         0 = key does not expire
      <n>  = key expires in n days
      <n>w = key expires in n weeks
      <n>m = key expires in n months
      <n>y = key expires in n years
Key is valid for? (0) 
Key does not expire at all
Is this correct? (y/N) y

You need a user ID to identify your key; the software constructs the user ID
from the Real Name, Comment and Email Address in this form:
    "Heinrich Heine (Der Dichter) <heinrichh@duesseldorf.de>"

Real name: Floreo Backup
Email address: xxx@floreo.info
Comment: Floreo Backup
You selected this USER-ID:
    "Floreo Backup (Floreo Backup) <xxx@floreo.info>"

Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? o
You need a Passphrase to protect your secret key.

We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
........+++++
.............+++++
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
.+++++
.......+++++
gpg: /home/floreo/.gnupg/trustdb.gpg: trustdb created
gpg: key 1A6F2256 marked as ultimately trusted
public and secret key created and signed.

gpg: checking the trustdb
gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
pub   4096R/1A6F2256 2017-11-11
      Key fingerprint = CC2D 8FF5 9BB2 8D88 0E30  0056 FC84 663E 1A6F 2256
uid                  Floreo Backup (Floreo Backup) <xxx@floreo.info>
sub   4096R/874F94F5 2017-11-11
Generate your GNuPG key pair

Now export both the private and public keys, the first one you will keep it safely on a USB drive or anywhere safe, it would be used to decrypt your data, the other one will be required on your server. First find out your key UID.

# gpg --list-key
/home/floreo/.gnupg/pubring.gpg
-------------------------------
pub   4096R/1A6F2256 2017-11-11
uid                  Floreo Backup (Floreo Backup) <xxx@floreo.info>
sub   4096R/874F94F5 2017-11-11
List your GnuPG keys

There the UID is 1A6F2256, you can export the keys.

# gpg --output backup_pub.gpg --armor --export 1A6F2256
# gpg --output backup.gpg --armor --export-secret-key 1A6F2256
Export both the pub and private keys

Import your public key

SSH to your server so you can import your key only, first upload the public key through scp for example.

# scp backup_pub.gpg xxx.xxx.xxx.xxx:/home/toto/.
SCP your public key to your server

Your public key being on your server you need to import it to gpg.

# gpg --import backup_pub.gpg
Import your public key.

You would think it’s over but you have to change the trust of the key, do the following command.

# gpg --edit 1A6F2256
Change the trust of your key

One you have the GNuPG prompt, type trust, select optimal, say yes and you’re done !

First server backup

Duplicity allows you to do full backups or incremental ones. Of course the first one is a full backup. One cool thing is that you don’t have to tell it explicitly to do an incremental one once the first one is made.

I suggest you use that very simple script to do your backup, you can edit it to your needs, mostly the exclude part. It backups fully on Monday, following days of the week are incremental. On Monday it also removes old backups, it’s done this dirty way since you cannot use duplicity’s cleanup function without the private key and the passphrase which I refuse to use online. It keeps two weeks of backup which is enough for me. You need to set an autofs mount to erase old backups since I didn’t script the FTP deletion yet.

Now run it, it may take a while so I recommend that you do that in a screen or tmux.

Rather than doing it manually, don’t forget to set a cron task to backup every night your data.

0 2 * * * root bash /root/scripts/backup.sh &>/dev/null
/etc/cron.d/backup

Restore your data

Obviously if you backup anything, you need to restore it some day soon. This time this operation requires the private key so you can decrypt the data ! Do what’s best for you, but it’s safer to keep your private key on your computer, you have to import your private key in gpg first.

# gpg --import backup.gpg 
gpg: key 1A6F2256: secret key imported
gpg: key 1A6F2256: public key "Floreo Backup (Floreo Backup) <xxx@floreo.info>" imported
gpg: Total number processed: 1
gpg:               imported: 1  (RSA: 1)
gpg:       secret keys read: 1
gpg:   secret keys imported: 1
Import your private key to your local computer

Next step is really simple you have to revert the duplicity command to restore.

# export PASSPHRASE=<PASSPHRASE>
# export FTP_PASSWORD=<FTP PASSWORD>
# duplicity  ftp://<USER>@<HOST>/<REMOTE_DIRECTORY>/ <LOCAL_DIRECTORY>
# unset PASSPHRASE FTP_PASSWORD
Restore your encrypted data locally

Finally rsync your data to your server, I know it looks complicated but it makes sure that your private key is never on your server.

Do not forget to remove your private key from your computer, it’s safer too.

gpg --delete-secret-key 1A6F2256
Delete your imported private key

A word of advice if you have several backup chains, for example I backup fully on Monday and do incremental ones for other days and keep two weeks top, duplicity will show you only the last full backup even if you restore stuff before the last full backup. It’s weird, but your restoration would work, it would just display the wrong date. To be sure I check the last mail.log, you might do it as well.

Check that your backups are good quite often. For further details, you can find some more explanations following the Ubuntu documentation.

Change sed’s default delimiter to delete line(s)

The problem

When you want to delete a line or several in a file using sed you need to use that syntax:

root@machine:~# sed -i '/mymatch/d' test

This will match all lines containing mymatch and delete them from the file (smartly called here test).

But what if the string you want to match contains the character / in it? Let’s try:

root@machine:~# sed -i '/http://example/d' test
sed: -e expression #1, char 8: unknown command: `/'

Obviously it fails, so what about using an other character to delimit? One could try that:

root@machine:~# sed -i '|http://example|d' test
sed: -e expression #1, char 1: unknown command: `|'

It fails too, you can try with the # character, this time it won’t give you an error, but it will just do nothing.

A solution

One solution is very simple, it’s just to backslash the character you want as the new delimiter like that:

root@machine:~# sed -i '|http://example|d' test

P.S: again, it’s pretty obvious but don’t use a character as delimiter which is already in the match, else you are back to the begining of this post. Be careful when your match is a variable.

Bash locale problem with cron

The problem

One problem that can occur when you do bash scripting concerns the locale. If you run a script directly from the prompt or from a cron, you can have some differences.

I stumbled on that problem while writing a script to send an email containing accents (in french language) using a cron. When I was executing the script from the command line, I got the email with the accents well shaped, but once it was from the cron, it was all messed up. Problem is that they were not having the same environment. So here are two working solutions to do so.

First solution (preferred one)

Set the value of LANG inside your script, a better practice would be to put it inside a configuration file that you include. Example for the french UT8:

#!/bin/bash
export LANG=fr_FR.utf8

This solution is very practical, you get the same behavior wether you call it directly in command line or from the cron. Also the maintenance is easier, if you need to switch locale, you have your config file for this 🙂

Second solution

A more dirty and deprecated way is to write the file /etc/environment with the LANG variable too. How to do (french UTF8 example):

root@machine:~# echo "LANG=fr_FR.utf8" > /etc/environment

So as I said, this solution is deprecated and not really wise because the locale will be set for all crons ! Maybe it’s what you want, but the first solution allows you to have a different locale in different scripts.

 

P.S: “weird” behaviors can occur not only with the locale when you are using a cron, it’s possible that the PATH is different. Again you can define directly the PATH inside your script. For example, one thing you can do is an echo of the PATH in the command line, and then copy the value inside your script, this way, the cron will use the same PATH.