Debian Backups, the Command Line Way…

I’ve been wanting to write a blog on this for a long time since I’ve actually had this backup method running in my environment for years. It’s super easy to setup and, while thank god I’ve never had to recover from a backup, I have been able to go back and recover individual files from my backups. What you’ll need from an environment setup is at least one Linux box that you need backed up, and at least one NAS or other file storage server that has an SSH server. I perform all my backups to online disk storage that is based on FreeNAS. There are plenty of NAS environment, and I’m not saying FreeNAS is the best or the worst, but I like it and it works for me. It works extremely well with Linux, Windows and Mac OS X.

There are two parts to this:

  • 1. manual backups
  • 2. automated backups

  • Let’s start with the manual backups, because once we have the manual backups performed, then we can easily turn that into a script and run it in CRON.

    First, we need to specify the directories we don’t want to backup in a file that is accessible to root. Let’s list the directories in “/” first.

    steve @ steve-G75VX ~ :) ##   ll /
    total 18M
    drwxr-xr-x  25 root   root 4.0K Oct 22 14:54 ./
    drwxr-xr-x  25 root   root 4.0K Oct 22 14:54 ../
    drwxr-xr-x   2 root   root 4.0K Aug 14 02:03 bin/
    drwxr-xr-x   4 root   root 3.0K Oct  3 11:39 boot/
    drwxrwxr-x   2 root   root 4.0K May 21 11:52 cdrom/
    -rw-------   1 root   root  18M Oct  3 11:40 core
    drwxr-xr-x  24 root   root 4.8K Oct 31 12:38 dev/
    drwxr-xr-x 148 root   root  12K Oct 27 20:37 etc/
    drwxr-xr-x   3 root   root 4.0K May 21 11:53 home/
    lrwxrwxrwx   1 root   root   33 Aug 14 02:06 initrd.img -> boot/initrd.img-3.19.0-25-generic
    lrwxrwxrwx   1 root   root   33 Jul 10 08:56 initrd.img.old -> boot/initrd.img-3.19.0-22-generic
    drwxr-xr-x  26 root   root 4.0K Oct 13 13:41 lib/
    drwxr-xr-x   2 root   root 4.0K May 21 12:41 lib32/
    drwxr-xr-x   2 root   root 4.0K Apr 22  2015 lib64/
    drwx------   2 root   root  16K May 21 11:47 lost+found/
    drwxr-xr-x   3 root   root 4.0K May 21 12:01 media/
    drwxr-xr-x   2 root   root 4.0K Apr 17  2015 mnt/
    drwxr-xr-x   6 root   root 4.0K Oct 20 11:28 opt/
    dr-xr-xr-x 283 root   root    0 Oct 21 20:30 proc/
    drwx------   4 root   root 4.0K Oct 27 16:57 root/
    drwxr-xr-x  30 root   root 1.1K Oct 27 20:50 run/
    drwxr-xr-x   2 root   root  12K Aug 14 02:03 sbin/
    drwxr-xr-x   2 root   root 4.0K Apr 22  2015 srv/
    dr-xr-xr-x  13 root   root    0 Oct 22 14:55 sys/
    drwxrwxrwx   2 nobody root 4.0K Oct 22 17:55 tftp/
    drwxrwxrwt  18 root   root 4.0K Nov  1 15:17 tmp/
    drwxr-xr-x  11 root   root 4.0K May 21 12:41 usr/
    drwxr-xr-x  13 root   root 4.0K Apr 22  2015 var/
    lrwxrwxrwx   1 root   root   30 Aug 14 02:06 vmlinuz -> boot/vmlinuz-3.19.0-25-generic
    lrwxrwxrwx   1 root   root   30 Jul 10 08:56 vmlinuz.old -> boot/vmlinuz-3.19.0-22-generic

    So, based on this, we’ll exclude like this:

    steve @ steve-G75VX ~ :) ##   sudo mkdir /backups
    [sudo] password for steve:
    steve @ steve-G75VX ~ :) ##   sudo touch /backups/exclude.list
    steve @ steve-G75VX ~ :) ##   sudo nano /backups/exclude.list
    steve @ steve-G75VX ~ :) ##  


    (Ctrl+x to quit, then y to save)

    Now that we have our directory and exclude list setup, now we need to make sure RSYNC is installed on our system.

    steve @ steve-G75VX ~ :) ##   sudo apt-get update
    Fetched 1,743 kB in 21s (79.7 kB/s)
    Reading package lists... Done
    steve @ steve-G75VX ~ :) ##   sudo apt-get install rsync
    Reading package lists... Done
    Building dependency tree      
    Reading state information... Done
    rsync is already the newest version.
    0 upgraded, 0 newly installed, 0 to remove and 38 not upgraded.
    steve @ steve-G75VX ~ :) ##

    Now that we have RSYNC installed and our backup exclusions defined, lets get our backups started.

    First, edit your .bashrc file in your home directory and add this line:

    alias backupall='sudo rsync -athvz --delete / steve@

    “What does all this do?” you might ask… well, it’s quite simple really.

    First, we create an alias for your shell named, “backupall”, because we’ll be performing full system backups from here.

    Next, we call “rsync” to run as root, and ask it to run with the switches -a, -t, -h, -v and -z.

  • -a = run in archive mode, which equals -rlptgoD (no -H,-A,-X)
  • -t = makes sure to preserve modification times on your files
  • -h = ensures that output numbers in a human-readable format
  • -v = trun verbosely.
  • -z = makes sure that file data is compressed during the transfer
  • And lastly, the “–delete” means, “This tells rsync to delete extraneous files from the receiving side (ones that aren’t on the sending side), but only for the directories that are being synchronized. You must have asked rsync to send the whole directory (e.g. lqdirrq or lqdir/rq) without using a wildcard for the directory’s contents (e.g. lqdir/*rq) since the wildcard is expanded by the shell and rsync thus gets a request to transfer individual files, not the files’ parent directory. Files that are excluded from the transfer are also excluded from being deleted unless you use the –delete-excluded option or mark the rules as only matching on the sending side (see the include/exclude modifiers in the FILTER RULES section).” —

    Next is the “/”, which means we’re backing up everything in “/”, which is everything.

    Lastly, we’re specifying the destination. In this case, we’re doing RSYNC over SSH, so we’ll be specifying a location in the way that you would specify a destination in SCP.

    Now test running your backup. I’ve run mine before, so my update is pretty quick. But this is going to backup your whole system for, so expect it to take a while.

    steve @ steve-G75VX ~ :( ᛤ>   backupallnas
    steve@'s password:
    sending incremental file list

    sent 1.09M bytes  received 50.77K bytes  58.56K bytes/sec
    total size is 1.91G  speedup is 1673.17
    rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1070) [sender=3.0.9]
    steve @ steve-G75VX ~ :( ᛤ>

    Now we need to create our script, and make it executable.

    root @ steve-G75VX ~ :) ##   nano /backups/backupall
    root @ steve-G75VX ~ :) ##   chmod +x /backups/backupall
    root @ steve-G75VX ~ :) ##   ll /backups/backupall
    -rwxr-xr-x 1 root root 96 Nov  1 17:02 /backups/backupall*
    root @ steve-G75VX ~ :) ##

    I added this one line to the backup file:

    sudo rsync -athvz --delete / steve@

    This looks pretty good! Now that we have a full backup of our machine, lets get this setup in CRON.

    steve @ steve-G75VX ~ :) ##   sudo su
    root @ steve-G75VX ~ :) ##   crontab -l
    no crontab for root
    root @ steve-G75VX ~ :( ##   crontab -e
    no crontab for root - using an empty one

    Select an editor.  To change later, run 'select-editor'.
      1. /bin/ed
      2. /bin/nano        <---- easiest
      3. /usr/bin/vim.tiny

    Choose 1-3 [2]: 2
    crontab: installing new crontab
    root @ steve-G75VX ~ :) ##

    The line that I added to CRON was this:

    0 3 * * * /backups/backupall >/dev/null 2&>1

    This basically states that every day at 3am, this script will be run.

    From here we need to make sure our local system can perform password-less logon to the SSH server. To do that we’ll be working off of a prior blog I wrote on SSH Keys, here: Using SSH Keys to simplify logins to remote systems.

    You’ll want to test that your system can SSH to your remote system without entering a password. As long as that works, we’re good to go!

    That’s it! It’s that simple!

    I have run into issues on some machines where SSH keys don’t work. I haven’t had the time to troubleshoot why, so I got a different way to figure out how to make backups work, without using SSH keys. The down side is that this is MUCH less secure, and I really don’t recommend running this in a production setting. But for home or non-business use, you’re probably just fine.

    So to do this, we’re going to use “SSHPASS” package. It’s out there for Debian and Ubuntu, so I’m sure it’s out there for other Linux/Unix systems as well.

    root @ steve-G75VX ~ :) ##   sudo apt-get install sshpass
    Reading package lists... Done
    Building dependency tree      
    Reading state information... Done
    The following NEW packages will be installed:
    0 upgraded, 1 newly installed, 0 to remove and 38 not upgraded.
    Need to get 10.5 kB of archives.
    After this operation, 56.3 kB of additional disk space will be used.
    Get:1 vivid/universe sshpass amd64 1.05-1 [10.5 kB]
    Fetched 10.5 kB in 0s (65.3 kB/s)  
    Selecting previously unselected package sshpass.
    (Reading database ... 258807 files and directories currently installed.)
    Preparing to unpack .../sshpass_1.05-1_amd64.deb ...
    Unpacking sshpass (1.05-1) ...
    Processing triggers for man-db ( ...
    Setting up sshpass (1.05-1) ...
    root @ steve-G75VX ~ :) ##

    Go ahead and test logging into your NAS box, or any box really, with this. The idea is that, when you’re scripting you need to logon to remote systems without a password. If you can’t use SSH keys, then this is your next best bet. Create a file in “root’s” home dir and name it whatever you want. I named mine, “backup.dat”. It must contain only the password you use to log into your remote machine, on one line, all by itself.

    root @ steve-G75VX ~ :) ##   nano ~/backup.dat
    root @ steve-G75VX ~ :) ##   chmod 600 backup.dat

    You’ll call “sshpass”, -f for the file with the password, the location of your “ssh” program, -p and the port number (default port for ssh is 22), followed by the username you login with (make sure it’s in the format of, “user@machine-ip”).

    root @ steve-G75VX ~ :) ##   sshpass -f /root/backup.dat /usr/bin/ssh -p 22 steve@
    Last login: Sun Nov  1 17:22:08 2015 from
    FreeBSD 9.2-RELEASE (FREENAS.amd64) #0 r+2315ea3: Fri Dec 20 12:48:50 PST 2013

        FreeNAS (c) 2009-2013, The FreeNAS Development Team
        All rights reserved.
        FreeNAS is released under the modified BSD license.

        For more information, documentation, help or support, go here:
    Welcome to FreeNAS
    [steve@freenas ~]$ exit
    Connection to closed.
    root @ steve-G75VX ~ :) ##

    Okay, now that we’ve tested this and know it’s working, lets modify our script here and get this working with “sshpass”.

    root @ steve-G75VX ~ :) ##   /usr/bin/rsync -athvz --delete --rsh="/usr/bin/sshpass -f /root/backup.dat ssh -o StrictHostKeyChecking=no -l YourUserN@me" /home/steve steve@

    Now test to make sure the script is working (as soon as you see the incremental file list being sent, you know it’s working properly):

    root @ steve-G75VX ~ :) ##   /usr/bin/rsync -athvz --delete --rsh="/usr/bin/sshpass -f /root/backup.dat ssh -o StrictHostKeyChecking=no -l steve" /home/steve steve@
    sending incremental file list
    ^Crsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(632) [sender=3.1.1]
    root @ steve-G75VX ~ :) ##
    root @ steve-G75VX ~ :) ##
    root @ steve-G75VX ~ :) ##   /backups/backupall
    sending incremental file list
    ^Crsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(632) [sender=3.1.1]
    root @ steve-G75VX ~ :( ##


    VN:F [1.9.22_1171]
    Rating: 0.0/5 (0 votes cast)
    VN:F [1.9.22_1171]
    Rating: 0 (from 0 votes)
    Tagged , , , , , , . Bookmark the permalink.

    Comments are closed.