Backups with AMANDA

Restore Files From Amanda

Background

We have two main file-servers (musashi, and yamato), which each perform their own backups Monday through Friday (typically). Musashi has a SCSI-attached HP LTO-1 Ultrium 230 (Aug. 2003) tape-drive; yamato has a replacement Ultrium 232 (Nov. 2006) tape-drive.
Yamato only backs up it's SCSI-connected RAID drives. Musashi doesn't back up one of it's own SCSI-connected RAID drives (too much non-essential data), but musashi does "reach out" and backup other machines' files. What gets backed up is listed/controlled in the /etc/amanda/Dailyxxx/disklist. Both yamato and musashi have amanda configured as a server and client, while the remote machines captured by musashi have amanda configured as a client-only.

Amanda Server Configuration

First, you've got to install amanda, and it's dependencies; Gentoo shown:

hostname / # emerge -pv amanda

These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild  N    ] sys-block/mtx-1.2.18  164 kB 
[ebuild  N    ] sci-visualization/gnuplot-4.0-r1  USE="emacs gd png readline -X -doc -ggi -pdf -plotutils (-svga) -xemacs" 2,111 kB 
[ebuild  N    ] app-arch/mt-st-0.7-r1  35 kB 
[ebuild  N    ] sys-apps/xinetd-2.3.14  USE="perl tcpd" 295 kB 
[ebuild  N    ] app-arch/dump-0.4.41  USE="readline -ermt -static" 277 kB 
[ebuild  N    ] app-backup/amanda-2.4.5  USE="-berkdb -debug -gdbm -samba -xfs" 1,506 kB 

Total: 6 packages (6 new), Size of downloads: 4,386 kB

NOTE: for on-campus backup-servers, unless you know differently, we will require the xfs USE-flag enabled (because that's what our RAID-arrays use).
Determine the tapetype entry, in /etc/amanda/Dailyxxx/amanda.conf; here are the results from two different drives, on two separate systems for our LTO-1 tapes:

Ultrium 230 (LTO1)

define tapetype Ultrium230 {
    comment "HP Ultrium 230, produced by amtapetype"
    length 101376 mbytes
    filemark 0 kbytes
    speed 13871 kps
}

Ultrium 232 (LTO1)

amanda@hostname ~ $ amtapetype -f /dev/nst0 -e 100G -o
Writing 1024 Mbyte   compresseable data:  37 sec
Writing 1024 Mbyte uncompresseable data:  72 sec
WARNING: Tape drive has hardware compression enabled
Estimated time to write 2 * 102400 Mbyte: 14400 sec = 4 h 0 min
wrote 3080192 32Kb blocks in 94 files in 7025 seconds (short write)
wrote 3096576 32Kb blocks in 189 files in 7511 seconds (short write)
define tapetype unknown-tapetype {
    comment "just produced by tapetype prog (hardware compression on)"
    length 96512 mbytes
    filemark 0 kbytes
    speed 13611 kps
}

Ultrium 960 (LTO3)

amanda@hostname ~ $ amtapetype -f /dev/nst0 -e 400G -o
Writing 4096 Mbyte   compresseable data:  45 sec
Writing 4096 Mbyte uncompresseable data:  77 sec
WARNING: Tape drive has hardware compression enabled
Estimated time to write 2 * 409600 Mbyte: 15400 sec = 4 h 16 min
wrote 12320768 32Kb blocks in 94 files in 7558 seconds (short write)
wrote 12386304 32Kb blocks in 189 files in 8121 seconds (short write)
define tapetype unknown-tapetype {
    comment "just produced by tapetype prog (hardware compression on)"
    length 386048 mbytes
    filemark 0 kbytes
    speed 50486 kps
}

Tape Drive Compression

Compression can be turned on or off using the mt tool noted below, then initiating the tapetype check; kill the tapetype check with ctrl-C when you've got enough info (takes quite a few minutes)

hostname ~ # mt -f /dev/nst0 defcompression 1   turns internal-hardware compression on
hostname ~ # su - amanda
amanda@hostname ~ $ amtapetype -f /dev/nst0 -e 100G -o
Writing 1024 Mbyte   compresseable data:  42 sec
Writing 1024 Mbyte uncompresseable data:  72 sec
WARNING: Tape drive has hardware compression enabled
Estimated time to write 2 * 102400 Mbyte: 14400 sec = 4 h 0 min

amanda@hostname ~ $ logout
hostname ~ # mt -f /dev/nst0 defcompression 0   turns internal-hardware compression off
hostname ~ #  su - amanda
amanda@hostname ~ $ amtapetype -f /dev/nst0 -e 100G -o
Writing 512 Mbyte   compresseable data:  38 sec
Writing 512 Mbyte uncompresseable data:  39 sec
Estimated time to write 2 * 102400 Mbyte: 15600 sec = 4 h 20 min

NOTE: We've tried enabling hardware-compression within the tape-drive, and also using software-compression on the server. While both approaches have their strengths and weaknesses, we've found that software-compression on the server consumes too much CPU :-( This is particularly a problem if amanda's size-estimation is used: an already-slow compression process doubles in time :-( Generally, enabling the tape-drive's (auto-)compression and omitting software-compression seems reasonable.

Amanda Client Configuration

Unfortunately, with the Gentoo packaging, you get more than you need for just a client (as seen above).
NOTE the currently-unstable amanda-2.5.1_p3-r4 contains the minimal USE flag. This definitely helps cut down on extraneous stuff, when all you need is amanda-client.
Once you've got amanda or amanda-minimal installed, there are a couple of separate steps required for amanda-client to work - configuring the xinetd meta-server, and configuring amanda. Most aggravations seem to stem from permissions, whether it's the server-access through xinetd, or the amanda-permitted-user in .amandahosts.

inferno / # emacs -nw /etc/xinetd.d/amanda

service amanda
{
        socket_type     = dgram
        protocol        = udp
        wait            = yes
        user            = amanda
        group           = amanda
        groups          = yes
        server          = /usr/libexec/amandad
        # You need to ensure this points to your Amanda server!
        # Don't just remove it!
        only_from       = 192.168.0.192 (clearly, this should point to your server)
        disable         = no
}

If this is a brand-new installation of xinetd, don't forget to start it! And, add it to your startup-scripts (Gentoo shown):

inferno / # /etc/init.d/xinetd start
 * Caching service dependencies ...                                         [ ok ]
 * Starting xinetd ...                                                      [ ok ]

And:

inferno / # rc-update add xinetd default
 * xinetd added to runlevel default

inferno / # emacs -nw /etc/amanda/amandahosts

# Machine-name for those hosts which are permitted to back up (fully-qualified
# is best).  An entry in /etc/hosts adds fail-safe against DNS death :-)
# |
# |                                     permitted user-name
# |                                     |

trainwreck.shaw.ca                      amanda      (again, use your server)

That's really all the client needs, unless you're actively using tcpwrappers, in which case /etc/hosts.allow and /etc/hosts.deny may require changes. Similarly, if iptables is running, you'll need to add firewall rules to allow amanda through.
To now bring this client into the backup routine, some entries are needed on the backup-server, in the disklist file (example only):

trainwreck ~ # emacs -nw /etc/amanda/Daily/disklist

inferno         /etc                    nocomp-user     2       le0
inferno         /srv/www                nocomp-user     2       le0

Now, run a check from your backup-server, to ensure things look good:

amanda@trainwreck ~ $ amcheck Daily
Amanda Tape Server Host Check
-----------------------------
Holding disk /holding: 16814560 kB disk space available, that's plenty
NOTE: skipping tape-writable test
Tape Daily07 label ok
NOTE: info dir /usr/local/var/amanda/Daily/curinfo/inferno: does not exist
NOTE: it will be created on the next run.
Server check took 0.245 seconds

Amanda Backup Client Hosts Check
--------------------------------
Client check: 2 hosts checked in 10.069 seconds, 0 problems found

(brought to you by Amanda 2.4.5)

In Case of Difficulty

• for backing up /var/lib/mysql, you may have to add amanda to the mysql group
• for other "Permission Denied" issues, you may have to add amanda to the disk group

Client-side logs (version 2.5):

/var/spool/amanda/tmp/client/

• dump doesn't get along with reiserfs. In fact, dump should probably be avoided, preferring GNU-tar instead.

Values to Change in Client Configurations

vi /etc/xinetd.d/amanda:

yamato.iat.sfu.ca musashi.iat.sfu.ca

vi /etc/services:

amanda 10080/tcp # amanda backup services

amanda 10080/udp

kamanda 10081/tcp # amanda backup services (Kerberos)

kamanda 10081/udp

amandaidx 10082/tcp # amanda backup services

amidxtape 10083/tcp

vi /etc/amanda/amandahosts:

sr-musashi.iat.sfu.ca amanda

musashi.iat.sfu.ca amanda

209.87.56.239 amanda

sr-yamato.iat.sfu.ca amanda

yamato.iat.sfu.ca amanda

209.87.56.240 amanda

localhost amanda

vi /etc/amanda/inetd.amanda:

amanda dgram udp wait amanda /usr/libexec/amanda/amandad amandad

amandaidx stream tcp nowait amanda /usr/libexec/amanda/amindexd amindexd

amidxtape stream tcp nowait amanda /usr/libexec/amanda/amidxtaped amidxtaped

vi /var/spool/amanda/.amandahosts:

sr-musashi.iat.sfu.ca amanda

musashi.iat.sfu.ca amanda

209.87.56.239 amanda

sr-yamato.iat.sfu.ca amanda

yamato.iat.sfu.ca amanda

209.87.56.240 amanda

localhost amanda

vi /etc/hosts:

209.87.56.240 yamato.iat.sfu.ca yamato

209.87.56.239 musashi.iat.sfu.ca musashi

Experimental Value (troubleshooting)

usermod -s /bin/false amanda

Amanda Tape Usage Overflow

There are times when Amanda has too much data to dump and one tape is not enough. In this case, the Amanda backup report will look similar to this:

Hostname: ServerName
Org     : OrgName
Config  : SomeConfigName
Date    : June 01, 2099

These dumps were to tape ServerName07.
There are 260210700k of dumps left in the holding disk.
Run amflush to flush them to tape.

The next tape Amanda expects to use is: ServerName08.

To correctly deal with this, please follow these steps:

1. The current tape (ServerName07, in the above example) is full, so eject it from the tape drive and store it
2. Put the next tape (ServerName08, in the above example) into the tape drive, as Amanda instructed
3. Log on to the server (ServerName, in the above example) with your userID and password
4. While logged in as userID@ServerName, switch to the ROOT user
5. While logged in as ROOT@ServerName, switch to the Amanda user (by typing su - amanda)
6. While logged in as Amanda@ServerName, run amflush SomeConfigName
7. Confirm the process. Amanda will now start dumping the remaining data to the new tape (ServerName08, in the above example)
8. Log off the server
9. Wait until Amanda finishes. When done, Amanda will send out a new backup report
10. Eject the current tape (ServerName08, in the above example) and put the next tape (ServerName09, in the above example) into the tape drive, as Amanda instructed

Replacing a Defective Tape

In the event that a tape cartridge needs to be replaced (eg: tape got eaten, tape backup unit keeps rejecting the tape, etc.), the problematic tape must be removed from Amanda's database. The new, replacement tape must be labelled with the same name as the defective tape that it is replacing.

For example, let's say that hurricane05 was defective. We'd put a brand new tape into the tape backup unit and then run the following commands:

amrmtape hurricane hurricane05
amlabel hurricane hurricane05

After that, the new tape completely replaces the old hurricane05 tape.

Enabling / Disabling a Backup

To disable or re-enable a backup, you have to edit the cron file.

1. Log onto the server that is associated with the amanda backup using your personal credentials. Be aware, though, that Spitfire's backup is handled by Hercules.
2. Switch to root user. (In the case of Hercules, skip the first step and log on as root right away.)
3. Type crontab -l -u amanda to list the contents of the cron file.
4. Type crontab -e -u amanda to edit the contents of the cron file.
5. To disable the line that contains the amanda backup job, place a pound character at the start of the line.
6. If you're using vi, type i to go into insert edit mode. After you've inserted or deleted the pound sign, press esc to exit editing mode. Then type :x to save changes and exit. (If you want to cancel your changes, type :q!) If you're using nano, type ctl-o and ctl-x to save changes and exit.
7. Confirm that the changes were retained by viewing the cron file again: crontab -l -u amanda