Backups with AMANDA
Background
We have two main file-servers (musashi, and yamato), which each perform their own backups Monday through Friday (typically). Musashi has a SCSI-attached HP LTO-1 Ultrium 230 (Aug. 2003) tape-drive; yamato has a replacement Ultrium 232 (Nov. 2006) tape-drive.
Yamato only backs up it's SCSI-connected RAID drives. Musashi doesn't back up one of it's own SCSI-connected RAID drives (too much non-essential data), but musashi does "reach out" and backup other machines' files. What gets backed up is listed/controlled in the /etc/amanda/Dailyxxx/disklist. Both yamato and musashi have amanda configured as a server and client, while the remote machines captured by musashi have amanda configured as a client-only.
Amanda Server Configuration
First, you've got to install amanda, and it's dependencies; Gentoo shown:
hostname / # emerge -pv amanda These are the packages that would be merged, in order: Calculating dependencies... done! [ebuild N ] sys-block/mtx-1.2.18 164 kB [ebuild N ] sci-visualization/gnuplot-4.0-r1 USE="emacs gd png readline -X -doc -ggi -pdf -plotutils (-svga) -xemacs" 2,111 kB [ebuild N ] app-arch/mt-st-0.7-r1 35 kB [ebuild N ] sys-apps/xinetd-2.3.14 USE="perl tcpd" 295 kB [ebuild N ] app-arch/dump-0.4.41 USE="readline -ermt -static" 277 kB [ebuild N ] app-backup/amanda-2.4.5 USE="-berkdb -debug -gdbm -samba -xfs" 1,506 kB Total: 6 packages (6 new), Size of downloads: 4,386 kB
NOTE: for on-campus backup-servers, unless you know differently, we will require the xfs USE-flag enabled (because that's what our RAID-arrays use).
Determine the tapetype entry, in /etc/amanda/Dailyxxx/amanda.conf; here are the results from two different drives, on two separate systems for our LTO-1 tapes:
Ultrium 230 (LTO1)
define tapetype Ultrium230 {
comment "HP Ultrium 230, produced by amtapetype"
length 101376 mbytes
filemark 0 kbytes
speed 13871 kps
}
Ultrium 232 (LTO1)
amanda@hostname ~ $ amtapetype -f /dev/nst0 -e 100G -o
Writing 1024 Mbyte compresseable data: 37 sec
Writing 1024 Mbyte uncompresseable data: 72 sec
WARNING: Tape drive has hardware compression enabled
Estimated time to write 2 * 102400 Mbyte: 14400 sec = 4 h 0 min
wrote 3080192 32Kb blocks in 94 files in 7025 seconds (short write)
wrote 3096576 32Kb blocks in 189 files in 7511 seconds (short write)
define tapetype unknown-tapetype {
comment "just produced by tapetype prog (hardware compression on)"
length 96512 mbytes
filemark 0 kbytes
speed 13611 kps
}
Ultrium 960 (LTO3)
amanda@hostname ~ $ amtapetype -f /dev/nst0 -e 400G -o
Writing 4096 Mbyte compresseable data: 45 sec
Writing 4096 Mbyte uncompresseable data: 77 sec
WARNING: Tape drive has hardware compression enabled
Estimated time to write 2 * 409600 Mbyte: 15400 sec = 4 h 16 min
wrote 12320768 32Kb blocks in 94 files in 7558 seconds (short write)
wrote 12386304 32Kb blocks in 189 files in 8121 seconds (short write)
define tapetype unknown-tapetype {
comment "just produced by tapetype prog (hardware compression on)"
length 386048 mbytes
filemark 0 kbytes
speed 50486 kps
}
Tape Drive Compression
Compression can be turned on or off using the mt tool noted below, then initiating the tapetype check; kill the tapetype check with ctrl-C when you've got enough info (takes quite a few minutes)
hostname ~ # mt -f /dev/nst0 defcompression 1 turns internal-hardware compression on hostname ~ # su - amanda amanda@hostname ~ $ amtapetype -f /dev/nst0 -e 100G -o Writing 1024 Mbyte compresseable data: 42 sec Writing 1024 Mbyte uncompresseable data: 72 sec WARNING: Tape drive has hardware compression enabled Estimated time to write 2 * 102400 Mbyte: 14400 sec = 4 h 0 min amanda@hostname ~ $ logout hostname ~ # mt -f /dev/nst0 defcompression 0 turns internal-hardware compression off hostname ~ # su - amanda amanda@hostname ~ $ amtapetype -f /dev/nst0 -e 100G -o Writing 512 Mbyte compresseable data: 38 sec Writing 512 Mbyte uncompresseable data: 39 sec Estimated time to write 2 * 102400 Mbyte: 15600 sec = 4 h 20 min
NOTE: We've tried enabling hardware-compression within the tape-drive, and also using software-compression on the server. While both approaches have their strengths and weaknesses, we've found that software-compression on the server consumes too much CPU :-( This is particularly a problem if amanda's size-estimation is used: an already-slow compression process doubles in time :-( Generally, enabling the tape-drive's (auto-)compression and omitting software-compression seems reasonable.
Amanda Client Configuration
Unfortunately, with the Gentoo packaging, you get more than you need for just a client (as seen above).
NOTE the currently-unstable amanda-2.5.1_p3-r4 contains the minimal USE flag. This definitely helps cut down on extraneous stuff, when all you need is amanda-client.
Once you've got amanda or amanda-minimal installed, there are a couple of separate steps required for amanda-client to work - configuring the xinetd meta-server, and configuring amanda. Most aggravations seem to stem from permissions, whether it's the server-access through xinetd, or the amanda-permitted-user in .amandahosts.
inferno / # emacs -nw /etc/xinetd.d/amanda
service amanda
{
socket_type = dgram
protocol = udp
wait = yes
user = amanda
group = amanda
groups = yes
server = /usr/libexec/amandad
# You need to ensure this points to your Amanda server!
# Don't just remove it!
only_from = 192.168.0.192 (clearly, this should point to your server)
disable = no
}
If this is a brand-new installation of xinetd, don't forget to start it! And, add it to your startup-scripts (Gentoo shown):
inferno / # /etc/init.d/xinetd start * Caching service dependencies ... [ ok ] * Starting xinetd ... [ ok ]
And:
inferno / # rc-update add xinetd default * xinetd added to runlevel default
inferno / # emacs -nw /etc/amanda/amandahosts # Machine-name for those hosts which are permitted to back up (fully-qualified # is best). An entry in /etc/hosts adds fail-safe against DNS death :-) # | # | permitted user-name # | | trainwreck.shaw.ca amanda (again, use your server)
That's really all the client needs, unless you're actively using tcpwrappers, in which case /etc/hosts.allow and /etc/hosts.deny may require changes. Similarly, if iptables is running, you'll need to add firewall rules to allow amanda through.
To now bring this client into the backup routine, some entries are needed on the backup-server, in the disklist file (example only):
trainwreck ~ # emacs -nw /etc/amanda/Daily/disklist inferno /etc nocomp-user 2 le0 inferno /srv/www nocomp-user 2 le0
Now, run a check from your backup-server, to ensure things look good:
amanda@trainwreck ~ $ amcheck Daily Amanda Tape Server Host Check ----------------------------- Holding disk /holding: 16814560 kB disk space available, that's plenty NOTE: skipping tape-writable test Tape Daily07 label ok NOTE: info dir /usr/local/var/amanda/Daily/curinfo/inferno: does not exist NOTE: it will be created on the next run. Server check took 0.245 seconds Amanda Backup Client Hosts Check -------------------------------- Client check: 2 hosts checked in 10.069 seconds, 0 problems found (brought to you by Amanda 2.4.5)
In Case of Difficulty
- for backing up /var/lib/mysql, you may have to add amanda to the mysql group
- for other "Permission Denied" issues, you may have to add amanda to the disk group
Client-side logs (version 2.5):
/var/spool/amanda/tmp/client/
- dump doesn't get along with reiserfs. In fact, dump should probably be avoided, preferring GNU-tar instead.
Values to Change in Client Configurations
vi /etc/xinetd.d/amanda:
yamato.iat.sfu.ca musashi.iat.sfu.ca
vi /etc/services:
amanda 10080/tcp # amanda backup services
amanda 10080/udp
kamanda 10081/tcp # amanda backup services (Kerberos)
kamanda 10081/udp
amandaidx 10082/tcp # amanda backup services
amidxtape 10083/tcp
vi /etc/amanda/amandahosts:
sr-musashi.iat.sfu.ca amanda
musashi.iat.sfu.ca amanda
209.87.56.239 amanda
sr-yamato.iat.sfu.ca amanda
yamato.iat.sfu.ca amanda
209.87.56.240 amanda
localhost amanda
vi /etc/amanda/inetd.amanda:
amanda dgram udp wait amanda /usr/libexec/amanda/amandad amandad
amandaidx stream tcp nowait amanda /usr/libexec/amanda/amindexd amindexd
amidxtape stream tcp nowait amanda /usr/libexec/amanda/amidxtaped amidxtaped
vi /var/spool/amanda/.amandahosts:
sr-musashi.iat.sfu.ca amanda
musashi.iat.sfu.ca amanda
209.87.56.239 amanda
sr-yamato.iat.sfu.ca amanda
yamato.iat.sfu.ca amanda
209.87.56.240 amanda
localhost amanda
vi /etc/hosts:
209.87.56.240 yamato.iat.sfu.ca yamato
209.87.56.239 musashi.iat.sfu.ca musashi
Experimental Value (troubleshooting)
usermod -s /bin/false amanda
Amanda Tape Usage Overflow
There are times when Amanda has too much data to dump and one tape is not enough. In this case, the Amanda backup report will look similar to this:
Hostname: ServerName Org : OrgName Config : SomeConfigName Date : June 01, 2099 These dumps were to tape ServerName07. There are 260210700k of dumps left in the holding disk. Run amflush to flush them to tape. The next tape Amanda expects to use is: ServerName08.
To correctly deal with this, please follow these steps:
- The current tape (ServerName07, in the above example) is full, so eject it from the tape drive and store it
- Put the next tape (ServerName08, in the above example) into the tape drive, as Amanda instructed
- Log on to the server (ServerName, in the above example) with your userID and password
- While logged in as userID@ServerName, switch to the ROOT user
- While logged in as ROOT@ServerName, switch to the Amanda user (by typing su - amanda)
- While logged in as Amanda@ServerName, run amflush SomeConfigName
- Confirm the process. Amanda will now start dumping the remaining data to the new tape (ServerName08, in the above example)
- Log off the server
- Wait until Amanda finishes. When done, Amanda will send out a new backup report
- Eject the current tape (ServerName08, in the above example) and put the next tape (ServerName09, in the above example) into the tape drive, as Amanda instructed
Replacing a Defective Tape
In the event that a tape cartridge needs to be replaced (eg: tape got eaten, tape backup unit keeps rejecting the tape, etc.), the problematic tape must be removed from Amanda's database. The new, replacement tape must be labelled with the same name as the defective tape that it is replacing.
For example, let's say that hurricane05 was defective. We'd put a brand new tape into the tape backup unit and then run the following commands:
amrmtape hurricane hurricane05 amlabel hurricane hurricane05
After that, the new tape completely replaces the old hurricane05 tape.
Enabling / Disabling a Backup
To disable or re-enable a backup, you have to edit the cron file.
- Log onto the server that is associated with the amanda backup using your personal credentials. Be aware, though, that Spitfire's backup is handled by Hercules.
- Switch to root user. (In the case of Hercules, skip the first step and log on as root right away.)
- Type crontab -l -u amanda to list the contents of the cron file.
- Type crontab -e -u amanda to edit the contents of the cron file.
- To disable the line that contains the amanda backup job, place a pound character at the start of the line.
- If you're using vi, type i to go into insert edit mode. After you've inserted or deleted the pound sign, press esc to exit editing mode. Then type :x to save changes and exit. (If you want to cancel your changes, type :q!) If you're using nano, type ctl-o and ctl-x to save changes and exit.
- Confirm that the changes were retained by viewing the cron file again: crontab -l -u amanda