Debian Backups, the Command Line Way…

I’ve been wanting to write a blog on this for a long time since I’ve actually had this backup method running in my environment for years. It’s super easy to setup and, while thank god I’ve never had to recover from a backup, I have been able to go back and recover individual files from my backups. What you’ll need from an environment setup is at least one Linux box that you need backed up, and at least one NAS or other file storage server that has an SSH server. I perform all my backups to online disk storage that is based on FreeNAS. There are plenty of NAS environment, and I’m not saying FreeNAS is the best or the worst, but I like it and it works for me. It works extremely well with Linux, Windows and Mac OS X.

There are two parts to this:

  • 1. manual backups
  • 2. automated backups

  • Let’s start with the manual backups, because once we have the manual backups performed, then we can easily turn that into a script and run it in CRON.


    First, we need to specify the directories we don’t want to backup in a file that is accessible to root. Let’s list the directories in “/” first.

    steve @ steve-G75VX ~ :) ##   ll /
    total 18M
    drwxr-xr-x  25 root   root 4.0K Oct 22 14:54 ./
    drwxr-xr-x  25 root   root 4.0K Oct 22 14:54 ../
    drwxr-xr-x   2 root   root 4.0K Aug 14 02:03 bin/
    drwxr-xr-x   4 root   root 3.0K Oct  3 11:39 boot/
    drwxrwxr-x   2 root   root 4.0K May 21 11:52 cdrom/
    -rw-------   1 root   root  18M Oct  3 11:40 core
    drwxr-xr-x  24 root   root 4.8K Oct 31 12:38 dev/
    drwxr-xr-x 148 root   root  12K Oct 27 20:37 etc/
    drwxr-xr-x   3 root   root 4.0K May 21 11:53 home/
    lrwxrwxrwx   1 root   root   33 Aug 14 02:06 initrd.img -> boot/initrd.img-3.19.0-25-generic
    lrwxrwxrwx   1 root   root   33 Jul 10 08:56 initrd.img.old -> boot/initrd.img-3.19.0-22-generic
    drwxr-xr-x  26 root   root 4.0K Oct 13 13:41 lib/
    drwxr-xr-x   2 root   root 4.0K May 21 12:41 lib32/
    drwxr-xr-x   2 root   root 4.0K Apr 22  2015 lib64/
    drwx------   2 root   root  16K May 21 11:47 lost+found/
    drwxr-xr-x   3 root   root 4.0K May 21 12:01 media/
    drwxr-xr-x   2 root   root 4.0K Apr 17  2015 mnt/
    drwxr-xr-x   6 root   root 4.0K Oct 20 11:28 opt/
    dr-xr-xr-x 283 root   root    0 Oct 21 20:30 proc/
    drwx------   4 root   root 4.0K Oct 27 16:57 root/
    drwxr-xr-x  30 root   root 1.1K Oct 27 20:50 run/
    drwxr-xr-x   2 root   root  12K Aug 14 02:03 sbin/
    drwxr-xr-x   2 root   root 4.0K Apr 22  2015 srv/
    dr-xr-xr-x  13 root   root    0 Oct 22 14:55 sys/
    drwxrwxrwx   2 nobody root 4.0K Oct 22 17:55 tftp/
    drwxrwxrwt  18 root   root 4.0K Nov  1 15:17 tmp/
    drwxr-xr-x  11 root   root 4.0K May 21 12:41 usr/
    drwxr-xr-x  13 root   root 4.0K Apr 22  2015 var/
    lrwxrwxrwx   1 root   root   30 Aug 14 02:06 vmlinuz -> boot/vmlinuz-3.19.0-25-generic
    lrwxrwxrwx   1 root   root   30 Jul 10 08:56 vmlinuz.old -> boot/vmlinuz-3.19.0-22-generic


    So, based on this, we’ll exclude like this:

    steve @ steve-G75VX ~ :) ##   sudo mkdir /backups
    [sudo] password for steve:
    steve @ steve-G75VX ~ :) ##   sudo touch /backups/exclude.list
    steve @ steve-G75VX ~ :) ##   sudo nano /backups/exclude.list
    steve @ steve-G75VX ~ :) ##  

    /cdrom
    /dev
    /lost+found
    /proc
    /run
    /sys
    /tmp

    (Ctrl+x to quit, then y to save)


    Now that we have our directory and exclude list setup, now we need to make sure RSYNC is installed on our system.

    steve @ steve-G75VX ~ :) ##   sudo apt-get update
    ...
    ...
    Fetched 1,743 kB in 21s (79.7 kB/s)
    Reading package lists... Done
    steve @ steve-G75VX ~ :) ##   sudo apt-get install rsync
    Reading package lists... Done
    Building dependency tree      
    Reading state information... Done
    rsync is already the newest version.
    0 upgraded, 0 newly installed, 0 to remove and 38 not upgraded.
    steve @ steve-G75VX ~ :) ##


    Now that we have RSYNC installed and our backup exclusions defined, lets get our backups started.

    First, edit your .bashrc file in your home directory and add this line:

    alias backupall='sudo rsync -athvz --delete / steve@1.1.1.1:/mnt/Backups/laptop/


    “What does all this do?” you might ask… well, it’s quite simple really.

    First, we create an alias for your shell named, “backupall”, because we’ll be performing full system backups from here.

    Next, we call “rsync” to run as root, and ask it to run with the switches -a, -t, -h, -v and -z.

  • -a = run in archive mode, which equals -rlptgoD (no -H,-A,-X)
  • -t = makes sure to preserve modification times on your files
  • -h = ensures that output numbers in a human-readable format
  • -v = trun verbosely.
  • -z = makes sure that file data is compressed during the transfer
  • And lastly, the “–delete” means, “This tells rsync to delete extraneous files from the receiving side (ones that aren’t on the sending side), but only for the directories that are being synchronized. You must have asked rsync to send the whole directory (e.g. lqdirrq or lqdir/rq) without using a wildcard for the directory’s contents (e.g. lqdir/*rq) since the wildcard is expanded by the shell and rsync thus gets a request to transfer individual files, not the files’ parent directory. Files that are excluded from the transfer are also excluded from being deleted unless you use the –delete-excluded option or mark the rules as only matching on the sending side (see the include/exclude modifiers in the FILTER RULES section).” — http://linux.die.net/man/1/rsync

    Next is the “/”, which means we’re backing up everything in “/”, which is everything.

    Lastly, we’re specifying the destination. In this case, we’re doing RSYNC over SSH, so we’ll be specifying a location in the way that you would specify a destination in SCP.


    Now test running your backup. I’ve run mine before, so my update is pretty quick. But this is going to backup your whole system for, so expect it to take a while.

    steve @ steve-G75VX ~ :( ᛤ>   backupallnas
    steve@1.1.1.1's password:
    sending incremental file list
    ./
    var/lib/mysql/blog/wp_AnalyticStats.MYD
    var/lib/mysql/blog/wp_AnalyticStats.MYI
    var/lib/mysql/blog/wp_options.MYD
    var/lib/mysql/blog/wp_options.MYI
    var/lib/mysql/blog/wp_postmeta.MYD
    var/lib/mysql/blog/wp_postmeta.MYI
    var/lib/sudo/steve/0
    var/log/auth.log
    var/log/apache2/access.log
    var/log/apache2/error.log

    sent 1.09M bytes  received 50.77K bytes  58.56K bytes/sec
    total size is 1.91G  speedup is 1673.17
    rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1070) [sender=3.0.9]
    steve @ steve-G75VX ~ :( ᛤ>



    Now we need to create our script, and make it executable.

    root @ steve-G75VX ~ :) ##   nano /backups/backupall
    root @ steve-G75VX ~ :) ##   chmod +x /backups/backupall
    root @ steve-G75VX ~ :) ##   ll /backups/backupall
    -rwxr-xr-x 1 root root 96 Nov  1 17:02 /backups/backupall*
    root @ steve-G75VX ~ :) ##


    I added this one line to the backup file:

    sudo rsync -athvz --delete / steve@1.1.1.1:/mnt/Backups/laptop/



    This looks pretty good! Now that we have a full backup of our machine, lets get this setup in CRON.

    steve @ steve-G75VX ~ :) ##   sudo su
    root @ steve-G75VX ~ :) ##   crontab -l
    no crontab for root
    root @ steve-G75VX ~ :( ##   crontab -e
    no crontab for root - using an empty one

    Select an editor.  To change later, run 'select-editor'.
      1. /bin/ed
      2. /bin/nano        <---- easiest
      3. /usr/bin/vim.tiny

    Choose 1-3 [2]: 2
    crontab: installing new crontab
    root @ steve-G75VX ~ :) ##


    The line that I added to CRON was this:

    0 3 * * * /backups/backupall >/dev/null 2&>1


    This basically states that every day at 3am, this script will be run.


    From here we need to make sure our local system can perform password-less logon to the SSH server. To do that we’ll be working off of a prior blog I wrote on SSH Keys, here: Using SSH Keys to simplify logins to remote systems.

    You’ll want to test that your system can SSH to your remote system without entering a password. As long as that works, we’re good to go!

    That’s it! It’s that simple!



    I have run into issues on some machines where SSH keys don’t work. I haven’t had the time to troubleshoot why, so I got a different way to figure out how to make backups work, without using SSH keys. The down side is that this is MUCH less secure, and I really don’t recommend running this in a production setting. But for home or non-business use, you’re probably just fine.

    So to do this, we’re going to use “SSHPASS” package. It’s out there for Debian and Ubuntu, so I’m sure it’s out there for other Linux/Unix systems as well.

    root @ steve-G75VX ~ :) ##   sudo apt-get install sshpass
    Reading package lists... Done
    Building dependency tree      
    Reading state information... Done
    The following NEW packages will be installed:
      sshpass
    0 upgraded, 1 newly installed, 0 to remove and 38 not upgraded.
    Need to get 10.5 kB of archives.
    After this operation, 56.3 kB of additional disk space will be used.
    Get:1 http://us.archive.ubuntu.com/ubuntu/ vivid/universe sshpass amd64 1.05-1 [10.5 kB]
    Fetched 10.5 kB in 0s (65.3 kB/s)  
    Selecting previously unselected package sshpass.
    (Reading database ... 258807 files and directories currently installed.)
    Preparing to unpack .../sshpass_1.05-1_amd64.deb ...
    Unpacking sshpass (1.05-1) ...
    Processing triggers for man-db (2.7.0.2-5) ...
    Setting up sshpass (1.05-1) ...
    root @ steve-G75VX ~ :) ##


    Go ahead and test logging into your NAS box, or any box really, with this. The idea is that, when you’re scripting you need to logon to remote systems without a password. If you can’t use SSH keys, then this is your next best bet. Create a file in “root’s” home dir and name it whatever you want. I named mine, “backup.dat”. It must contain only the password you use to log into your remote machine, on one line, all by itself.

    root @ steve-G75VX ~ :) ##   nano ~/backup.dat
    root @ steve-G75VX ~ :) ##   chmod 600 backup.dat


    You’ll call “sshpass”, -f for the file with the password, the location of your “ssh” program, -p and the port number (default port for ssh is 22), followed by the username you login with (make sure it’s in the format of, “user@machine-ip”).

    root @ steve-G75VX ~ :) ##   sshpass -f /root/backup.dat /usr/bin/ssh -p 22 steve@1.1.1.1
    Last login: Sun Nov  1 17:22:08 2015 from 1.1.1.2
    FreeBSD 9.2-RELEASE (FREENAS.amd64) #0 r+2315ea3: Fri Dec 20 12:48:50 PST 2013

        FreeNAS (c) 2009-2013, The FreeNAS Development Team
        All rights reserved.
        FreeNAS is released under the modified BSD license.

        For more information, documentation, help or support, go here:
        http://freenas.org
    Welcome to FreeNAS
    [steve@freenas ~]$ exit
    logout
    Connection to 1.1.1.1 closed.
    root @ steve-G75VX ~ :) ##


    Okay, now that we’ve tested this and know it’s working, lets modify our script here and get this working with “sshpass”.

    root @ steve-G75VX ~ :) ##   /usr/bin/rsync -athvz --delete --rsh="/usr/bin/sshpass -f /root/backup.dat ssh -o StrictHostKeyChecking=no -l YourUserN@me" /home/steve steve@1.1.1.1:/mnt/Backups/laptop/


    Now test to make sure the script is working (as soon as you see the incremental file list being sent, you know it’s working properly):

    root @ steve-G75VX ~ :) ##   /usr/bin/rsync -athvz --delete --rsh="/usr/bin/sshpass -f /root/backup.dat ssh -o StrictHostKeyChecking=no -l steve" /home/steve steve@1.1.1.1:/mnt/Backups/laptop
    sending incremental file list
    ^Crsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(632) [sender=3.1.1]
    root @ steve-G75VX ~ :) ##
    root @ steve-G75VX ~ :) ##
    root @ steve-G75VX ~ :) ##   /backups/backupall
    sending incremental file list
    steve/.cache/google-chrome/Default/Cache/
    ^Crsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(632) [sender=3.1.1]
    root @ steve-G75VX ~ :( ##

    Success!







    http://linux.die.net/man/1/rsync
    https://www.debian-administration.org/article/56/Command_scheduling_with_cron

    VN:F [1.9.22_1171]
    Rating: 0.0/5 (0 votes cast)
    VN:F [1.9.22_1171]
    Rating: 0 (from 0 votes)

    Connecting FreeNAS 9.2 iSCSI to ESXi 5.5 Hypervisor and performing VM Guest Backups

    In this blog we’re going to work through connecting your FreeNAS box to your ESXi server for easy backups. ESXi is a hypervisor from VMware that is arguably the best ever made. I like it the most because of many reasons, including the fact that it’s free, it’s a non-Microsoft product, and if you need additional features, the licensing is reasonably priced.

    Now, I’ve already gone through and setup my whole server, so what I’ll do is just retrace my steps here and show how easy this process really is.
    Lets get started with FreeNAS. I have an old computer that I used for setting up FreeNAS. It’s an ASUS motherboard with an Intel Core 2 Quad (Q6600), 12GB of DDR2 memory, and 6x Western Digital 3TB Disks. I did a basic install on the system, and I’m running the OS from a SanDisk 8GB USB thumb drive. Microcenter has them cheap, so I grabbed a bunch for redundancy.

    After you have FreeNAS installed and running with a ZFS volume of your choice, that’s the point that I’ll be starting with here. My need for an iSCSI target was brought on by the need for backups. While I am running a RAID 10 on my ESX server (there are 2 striped RAID arrays mirrored, 1+0), the redundancy still wasn’t enough for my peace of mind. SAS drives can fail at any time, and since my environment is pretty high availability, I didn’t want to take a chance losing 2 drives in my ESX server and losing all my data.

    So I have a dilemma, spend a lot of money on backup software, or connect my ESXi server to my FreeNAS server. Seems pretty cut and dry to me, so I’ll backup my VM guests from the ESXi server and save money at the same time.

     

     

    Start of at your FreeNAS web interface. Go to the Services section and click on “Control Services”. You should see a screen like this. If you never setup iSCSI before, then you’ll need to turn on the “iSCSI” service. After you start the service, go ahead and click on the wrench icon next to the on/off switch.

     

     

    In the “Target Global Configuration” section, fill out the info that you need to so that it pertains to your environment. The biggest item here is to make sure that the “Enable LUC” option is ENABLED. If it isn’t your iSCSI target won’t ever show up in ESXi.

     

     

    From there, now go to the “Portals” tab. And click on “Add Portal”. If you have multiple NICs in your FreeNAS box, you may want a direct link to your FreeNAS box with an Ethernet Cable. If so, select the adapter you expect it to be on, otherwise you can leave it as “0.0.0.0”. You’ll want to keep the port number set as 3260.

     

     

    Now go to the Initiators tab. Click on “Add Initiator”. You can leave it set to “ALL” for both, but I would recommend at least setting it up for the network or host that you expect to connect from.

     

     

    Now go to the “Targets” tab. Make the target name and alias “esxi”, leave the serial number as-is, Target Flags should be “Read-Write”, your dropdown menus for Portal and Initiator should be available from the ones you setup previously, and then click “OK”.

     

     

    Now head on over to the Extents tab. Name the Extent “esxi”. What I did is create a folder on my existing ZFS volume named “iSCSI”, which is located in the “/mnt/primary/iSCSI” path. Then just type the file name that you want to use. In my path it looked like this: “/mnt/primary/iSCSI/esxi.extent”. I allocated 550GB of space because that’s approximately how much space I have on the ESXi server. Then click OK.

     

     

    Now click on the “Associated Targets” tab. Click on “Add Target / Extent”. Your options should be available in the two dropdown menus. Select those and then click OK.

     

     

    Looking great so far. Now log in on your ESXi server with the vSphere software. After you log in, go to the “Inventory” view, then click on the “Configuration” tab.

     

     

    Click on the “Add Networking…” button. Select the option for creating a new “VMkernal” and click “Next >”

     

     

    From here you can see what network adapters you can choose to assign to your new network. We’re going to use VMNIC3, which is actually Port 4 on our server. Click Next to continue.

     

     

    Here is where you can assign a name to your network. I like to name things so they can be easily identified. We’re going to name our’s “iscsi”. We’re not using VLANs here, so leave that as NONE, and click “Next” to continue.

     

     

    Since this is a direct cable connection, we’re going to lock down the network with a subnet where there are only two hosts. We aren’t using the “10.254.254.X” network anywhere so that works too. We will make the secondary adapter in the FreeNAS box “10.254.254.1” and this can be “.2”. Notice the subnet mask ends in 252. That states that there are only 4 IPs in the network, with “.0” being the network, and “.4” as the broadcast. There doesn’t need to be a gateway, so don’t worry about changing that; it’ll never be used. Click “Next” to continue.

     

     

    This is just a summary page, just click on finish.

     

     

    Now you’re back to the ” Configuration / Networking ” section. From here you can see your newly added “iscsi” switch that was just added, and see that it is tied to the “vmnic3”.

     

     

    Now head on over to the “Storage Adapters” link in the left column. We’re going to reuse the Broadcom iSCSI Adapter “vmhba35”. Right click on that adapter and then click “Properties”.

     

     

    You’ll see this screen come up. From there, click on the “Network Configuration” tab.

     

     

    We need to bind the adapter to this iSCSI initiator so that our ESX box knows where to send iSCSI traffic. Click on the “Add…” button.

     

     

    You should see the “iscsi” switch that we just created listed here. Click on that, then click “OK”.

     

     

    You should now see your “iscsi” virtual switch listed in the “VMkernel Port Bindings” section. Now click on the “Dynamic Discovery” tab.

     

     

    Click the “Add…” button near the bottom of the window. In the window that appears, type in the IP address of your FreeNAS server. It should be 10.254.254.1 if you set yours up exactly like mine. Otherwise, change it accordingly. Leave the port number default at 3260. Then click “OK”.

     

     

    After clicking “OK” on the last window, you should see your FreeNAS box listed in the “iSCSI Server Location”. Click the “Close” button on that window.

     

     

    When you click close, you’ll see a window appear that asks you to rescan the location. Click “OK” on that, and wait for the rescan process. After the rescan, you should see your storage pool show up in the Details pane, as you can see in this screenshot.

     

     

    From here, all you need to do is Click on the “Storage” link in the left hand column, and then click on the “Add Storage…” link in the upper right hand corner. That will bring you to this screen. From here, just click “Next”.

     

     

    After you follow the prompts you should see your new Data Store listed.

     

     

     

     

    While there’s a ton of individual steps involved here, it’s not that difficult to complete this, nor does it really take that long.

    VN:F [1.9.22_1171]
    Rating: 4.5/5 (2 votes cast)
    VN:D [1.9.22_1171]
    Rating: +3 (from 3 votes)