You are here: Foswiki>AGLT2 Web>AmandaBackup (18 Feb 2019, WenjingWu)Edit Attach

AFS Tape Backups with Amanda

AFS Tape Backups with Amanda

Amanda Commands

For operations with amanda, you should be the amanda user on bambi: "su - amanda". The exception is "amrecover".

Here are some useful amanda commands. For more info they all take "-h" for help. Nearly all require "umatlas" as an argument to indicate which configuration to use.

amrecover	Browse and restore NON-AFS backups. Won't properly restore afs images. Use /pool1/restore to temporarily hold recovered items on bambi. Usage: "amrecover -C umatlas". You can also run this from a remote host - please see further details in the section about this command
amtape	Example: "amtape umatlas eject 0" - eject drive 0 and return tape to library slot.
amadmin	Find dumps, many other admin commands. Example: amadmin umatlas find linat08
amlabel	Example: "amlabel umatlas L09045 slot 3" - label tape in slot 3 for Amanda usage with label L09045. Give "-f" flag to overwrite existing Amanda label. There is a script to label an entire magazine of tapes with barcodes as scanned by the library in /etc/amanda/label-changer-barcode.
amrmtape	Example: "amrmtape umatlas L09045" - remove tape L09045 and all backups on it from backup indexes.
amflush	Example: "amflush umatlas" - flush dumps from holding disk to tape. Use if you have a backlog of holding disk backups due to tape being offline for some period of time.
mtx	Not part of amanda. For manipulating tape changer, but always use amtape for changing tapes around so amanda doesn't get out of sync and need to re-inventory. Example: mtx -f /dev/scsi/changer/c11t500104F000B87A68d0 status
mt	Not part of amanda. For manipulating tape drive. Example: mt -f /dev/rmt/2 rewind

Every Amanda command begins with "am" and there are many more if you need to do something not listed here.

Schedules, volumes, rotation

See /etc/amanda/amanda.conf.common, /etc/amanda/umatlas/amanda.conf, and /etc/amanda/umatlas/disklist on bambi.

Summary: We use 43 tapes and keep 1 year of history. Amanda decides when it flushes from holding disk to tape based on keeping about 3 tapes worth online (on-disk) before flushing anything to a tape. This should result in the last couple days being on disk and faster to restore.

Restoring a file with amrecover (not afs)

You can use amrecover directly on bambi or from a remote host. Our amanda RPMS were build with the necessary config information as default (the next section will detail command line arguments to connect to the backup server if necessary). As root navigate to some temporary place to keep the restored files. Make sure it's ok if everything inside current working dir is deleted before you start - amrecover will add/delete such that your working directory will be an exact image of what was backed up.

 amrecover -C umatlas

You'll be brought into a command line interface which allows you to use conventional 'ls' and 'cd' commands to explore the tape index at any available point in time. The online help should get you going (type "?"), but here's a quick tutorial.

First, type "listhost" to get a list of available hosts. Set a host using (for example) "sethost linat06". This shows all hosts available. Which host you want depends on where the particular AFS volume was hosted.
Once you have set a host, you can type "listdisk" to see volumes backed up from that host. Set a volume by typing something like "setdisk afs:home".
At any point before or after choosing disk and host, you can choose which date to browse with a command like "setdate 2007-08-01". If you've chosen or when you do choose a disk, amrecover will inform you if there are no backups to browse from your chosen date.
Browse the listings using "ls" and "cd" as usual. When you find the file/directory you want, add it to the restore list by typing "add filename/dir". There is also an "addx" command to make use of regex/wildcards.
You can add as many things as you wish to the extract list. When you're ready to extract to the current directory, type "extract".

Here is an example I did when recently a certain config file got over-written in a package update:

amrecover> listhost
200- List hosts for config umatlas
201- localhost.localdomain
 ... etc ... etc ...

amrecover> sethost localhost.localdomain
200 Dump host set to localhost.localdomain.

amrecover> listdisk
200- List of disk for host localhost.localdomain
201- /pool1/mysql-backup
201- /etc/mysql-zrm
201- /etc/amanda
201- /
201- /pool1/lustre-backup

amrecover> setdisk /etc/mysql-zrm

amrecover> history
200- Dump history for config "umatlas" host "localhost.localdomain" disk /etc/mysql-zrm
201- 2013-08-17-00-00-00 1 L09006:33
201- 2013-08-16-00-00-00 0 L09007:21
201- 2013-08-15-00-00-01 1 L09019:42
... etc.... etc.

amrecover> setdate 2013-08-07

amrecover> ls

...etc...
2013-08-07-00-00-01 mysql-zrm.conf
2013-08-07-00-00-01 mysql-zrm-reporter.conf
...etc.

amrecover> add mysql-zrm.conf
Added file /mysql-zrm.conf
amrecover> extract

Extracting files using tape drive sl500 on host localhost.
The following tapes are needed: L09010

Extracting files using tape drive sl500 on host localhost.
Load tape L09010 now
Continue [?/Y/n/s/d]? Y
Restoring files into directory /pool1/restore
All existing files in /pool1/restore can be deleted
Continue [?/Y/n]? Y

./mysql-zrm.conf
amrecover> quit

That's it. Your file is in the working directory.

Remote recovery with amrecover

It is also possible to connect to amrecover remotely from another host with amanda installed. As noted before, our RPMS will have this configuration built in. If necessary here is the command line to point at our backup server.

 amrecover -C umatlas -s bambi.local -t bambi.local

You have to enable the access on bambi.local. Wildcards do not work here to enable access for our whole network. In file /etc/amanda/.amandahosts add a line like this:

linat02.local root amindexd amidxtaped

Also it will not work from an AFS server because they have modified installations that point to non-standard GNU tar that amrecover doesn't like.

Restoring a file (and preserving AFS ACLs)

Tested 08/01/2012.

Finding the dump you need

First, figure out which server hosts the RW volume you want to restore (vos examine on any host with AFS). Then we'll use use "amadmin" to find the available dumps. You can be as specific as you want...if you simply specified "linat08" and none of the rest it would show you all dumps for linat08. If you add a date it will limit to that date. Since we are backing up with a volset wildcard we cannot search by a specific volume in many cases (such as home.user volumes).

amadmin umatlas find linat08 "afs:linat08.grid.umich.edu/vicepg/home..*"
...
2012-11-14 02:00:01 linat08.local "afs:linat08.grid.umich.edu/vicepg/home..*"  1 L09036          5  1/1 OK 
2012-11-15 02:00:01 linat08.local "afs:linat08.grid.umich.edu/vicepg/home..*"  1 L09036         12  1/1 OK 
2012-11-16 02:00:00 linat08.local "afs:linat08.grid.umich.edu/vicepg/home..*"  1 L09019         35  1/1 OK 
2012-11-19 02:00:00 linat08.local "afs:linat08.grid.umich.edu/vicepg/home..*"  1 L09009          3  1/1 OK 
2012-11-20 02:00:00 linat08.local "afs:linat08.grid.umich.edu/vicepg/home..*"  1 L09006         26  1/1 OK 
2012-11-21 02:00:00 linat08.local "afs:linat08.grid.umich.edu/vicepg/home..*"  1 L09011         26  1/1 OK 
2012-11-22 02:00:00 linat08.local "afs:linat08.grid.umich.edu/vicepg/home..*"  1 L09026         29  1/1 OK 
2012-11-23 02:00:00 linat08.local "afs:linat08.grid.umich.edu/vicepg/home..*"  0 L09032         41  1/1 OK

Once we know which dumps we want to restore, amfetchdump can find and restore them. It will load tapes or use holding disk as needed. You will need both a level 0 dump and a level 1 dump from any date after that.

 
amfetchdump umatlas -p linat06.local "afs:linat06.grid.umich.edu/vicepe/home..*" 20150407000000 | volset restore - home.user home.user.dumpfile.0

(now it will go through volumes in the volset and restore the one specified)

Done with home.otheruser
Done with home.otheruser
Restoring: home.user
Done with home.user
... etc ...

The datestamp is exactly the same as the ones listed by 'amadmin find'. Just take out the dashes and colons. Repeat for the incremental you would like to restore (if desired, not required if the full dump from whatever date is new enough for you).

Next transport the dumps someplace with afs access and restore the dumps using vos restore. I often copy to one of the interactives since I can use a password (ci-1-12.local or 13,14). Note that they are reverse named...1-12 is umt3int03. You can give the restored volume any name that makes sense. In many cases you might want to restore it without overwriting the old volume and then mount it someplace so the user can recover files.

/usr/sbin/vos restore -server linat08.grid.umich.edu -partition /vicepg -name home.username.restored -file home.user.dumpfile.0

If also have an incremental dump to include:

/usr/sbin/vos restore -server linat08.grid.umich.edu -partition /vicepg -name home.username.restored -overwrite incremental -file home.user.dumpfile.1

...and so on through all incrementals you may have.

OR...if your backup server has access to AFS (bambi does not) restore directly into afs. Get the AFS tokens allowing you to do 'vos restore' (admin). As noted above, if this is one of the incremental dumpfiles use an incremental vos restore.

amfetchdump umatlas -p linat06.local "afs:linat06.grid.umich.edu/vicepe/home..*" 20150407000000 | volset restore home.user - | vos restore -server linat08.grid.umich.edu -partition /vicepg -name home.user.restored

If this is a restore of a home that was wiped out entirely, don't forget to make a replica again (do +1 to RW server host number):

/usr/sbin/vos addsite -server linat06.grid.umich.edu -partition /vicepe -id home.user
/usr/sbin/vos release home.user

Finally, you'll need to mount the volume someplace:

/usr/bin/fs mkmount /afs/.atlas.umich.edu/home/user home.user
/usr/sbin/vos release root.cell

restore single volumes out of a single volume backup into AFS space

These are the volumes specified by afs:volumename in our backup disklist.

Get the AFS tokens allowing you to do 'vos restore', also make sure you can access the tape device. Load the correct tape as determined from amrecover.

amfetchdump -p  linat06.local afs:hardware 20120629020000 > restored.hardware.dumpfile

(if server has AFS access, can pipe to  | vos restore -server dest-server -partition dest-part -name dest-name)

NOTE: You may need to use the 'cut_garbage' perl script on Bambi in /pool1/amholding/restore to get a dumpfile that is suitable for 'vos restore'. Just use it like:

amfetchdump -p linat06.local afs:hardware 20120629020000 | /pool1/amholding/restore/cut_garbage > restored.hardware.dumpfile

If restoring from two tapes (one full, one incremental) you need to use the -incremental flag to vos restore for the incremental dumpfile(s).

Entire volumesets onto a partition (like doing vos restore on each volume in the volume set)

Get the AFS tokens allowing you to do 'vos restore', also make sure you can access the tape device. Insert and rewind the correct tape. Also, you MUST be root to use this, as 'vos restore' is spawned with the '-localauth' flag.

# fetchdump -p hostname 'afs:src-serv/src-part/src-pattrn' |   volset restore - '*' afs:dest-server/dest-part

The restored volumes will have the names as follows 'src.volume' in the volumeset will become 'src.volume.r' on the dest-part partition.

Troubleshooting

Taper log says: taper: critical (fatal): Can't locate auto/Amanda/Device/Device/swig volume.al in @INC

This is most likely due to the tape Amanda was trying to use being unlabeled although Amanda somehow had the idea it was/should be labeled. Label the tape with amlabel:

 amlabel umatlas L09010 slot 30

If there is an error but you are sure this tape is causing the issue use "-f" to force it. Label tapes with barcode. You can list the changer inventory with mtx. In general, the tape in the first tape drive will be the tape causing the problems with reading the label (it loaded the tape, tried to read the label, and stopped on the error). This error possibly is indicating a failing tape? Watch for it again on L09010. Typically in our case I believe it indicates a tape labeled with previous version of amanda.

 mtx -f /dev/scsi/changer/c11t500104F000B87A68d0 status

http://wiki.zmanda.com/index.php/Taper_log_says:_taper:_critical_(fatal):_Can't_locate_auto/Amanda/Device/Device/swig_volume.al_in_@INC

Installation notes

Client

You will need to install the amanda and amanda-afs RPMS on backup clients (not required on server). Packages are available in our umatlas repository. The SRPMS already have included all the appropriate patches, updated amanda-afs source code, and appropriate flags to Amanda compile. At some point the amanda-afs code had to have a couple small changes made to work anymore with newer AFS versions and with SL6. The info below about patching and where to find the sources is left for reference. All necessary code can be found on our website now: https://www.aglt2.org/amanda.php

The amanda-afs source tree is tracked under the "packages" svn repo. Perhaps we could be considered the current maintainers of this code though I am not sure anyone uses it. Once I had an inquiry from somebody else that downloaded it from us who was having problems. svn+ssh://ndt.aglt2.org/repos/packages/amanda-afs

Server

Amanda is compiled from source on our backup server (bambi). You have to use 'gmake' to compile. In each source tree I download I have placed a shell script with the contents as below (named 'aglt2-config-command'). One could just copy this from a previous version to run configure on a new version. Or, here is the full config used:

#!/usr/bin/bash

./configure  --with-bsdtcp-security  --with-bsdudp-security  --with-rsh-security  --with-ssh-security  --prefix=/usr  --libexecdir=/usr  --sysconfdir=/etc  --with-group=sys  --localstatedir=/var  --with-debugging=/var/log/amanda  --with-fqdn  --with-config=umatlas  --with-smbclient=/usr/bin/smbclient  --with-gnuplot=/opt/csw/bin/gnuplot  --with-star=/opt/csw/bin/star  --with-gnutar=/opt/csw/bin/gtar

Note that the programs in /opt/csw have to come from the sunfreeware.com archive. They are installed using a util called "pkg-get".

Early Setup Notes

Just for reference. Build amanda-afs packages from code hosted in packages repo which has this patch applied already. https://ndt.aglt2.org/viewvc/packages/amanda-afs/

Notes about amanda-afs: The tools needed to perform a "proper" backup of AFS (as in, preserves AFS acls) are available here:
ftp://ftp.ccmr.cornell.edu/pub/amanda-afs/amanda-afs.tar.gz

I also needed to apply this patch to the amanda-afs sources to compile successfully:
https://secure.linuxbox.com/tiki/tiki-download_file.php?fileId=75

Expand the amanda-afs.tar.gz package somewhere and then run this command without cd'ing into the dir:

patch -p0 < amanda-afs-cornell-dflushvol.patch

(patch and amanda-afs.tar.gz are attached to this document)

The "amanda" package from our yum repository has been compiled with the right options for AFS backups (according to directions from amanda-afs distribution). Regardless of whether you are doing an AFS backup or not, you also need the "amanda-afs" package to get /usr/sbin/gtar-wrapper.pl installed.

Short summary is you compile using the --with-gnutar=/path/to/gtar-wrapper.pl.

Various operations

how to setup Amanda backup on a client

On the client

[root@c-16-40 umatlas]# yum --enablerepo=umatlas install amanda-backup_client
[root@c-16-40 umatlas]#vi /etc/xinetd.d/amandaclient (modify the user)

user = amanda

[root@c-16-40 umatlas]# systemctl restart xinetd
[root@c-16-40 ~]#vi /var/lib/amanda/.amandahosts (adding the following line)
bambi.local amanda amdump

Verify if it is running on client:

[root@c-16-40 ~]# netstat -anp|grep 10080

tcp 0 0 0.0.0.0:10080 0.0.0.0:* LISTEN 909590/xinetd

On the server (bambi.local)

Addling the following lines:

root@bambi /var/log/amanda/amandad# grep dc48-16-40 /etc/amanda/umatlas/disklist
dc48-16-40.local /opt/bro medium-tar

Adding the following line to allow the client to run amrecover from the localhost

root@bambi# grep dc48-16-40 /etc/amanda/.amandahosts dc48-16-40.local root amindexd amidxtaped

Verify if the server can connect to the client:

amanda@bambi ~$ amcheck umatlas -c dc48-16-40

Amanda Backup Client Hosts Check -------------------------------- Client check: 1 host checked in 1.124 seconds. 0 problems found.

From the client, to recover from a recent copy

[root@c-16-40 umatlas]# amrecover -C umatlas -s bambi.local -t bambi.local
amrecover> sethost umfs02
501 Host umfs02 is not in your disklist.
Trying host umfs02.local ...
200 Dump host set to umfs02.local.
amrecover> listdisk</p>
200- List of disk for host umfs02.local
201- /data08/backup
201- /data08/manage
amrecover> setdisk /data08/backup
200 Disk set to /data08/backup.
amrecover> ls
2018-12-07-00-00-00 umdist

How to force amanda server to run backup

su - amanda
amanda@bambi ~$ amdump umatlas --from-client umfs02.local

How to force amanda to flush the holding disk to tape

su - amanda
amanda@bambi ~$ amflush umatlas

Or to flush a specific host and disk

amanda@bambi ~$ amflush umatlas umfs02 /data08/backup

How to check which disk/tape are holding the backup file from a specific host

amanda@bambi ~$ amadmin umatlas find umfs02

Issues with bambi(Amanda) after rebooting

Log file is /etc/amanda/umatlas/logs

after power back on to bambi, I had to do a few things to make it work again:

1) start the services (now it is enabled to auto start at boot)

svcadm enable amanda/tcp
svcadm enable amanda/udp
svcadm enable amandaidx/tcp
root@bambi /etc/amanda/umatlas/logs# svcs |grep amanda
online Feb_11 svc:/network/amanda/tcp:default 
online Feb_11 svc:/network/amanda/udp:default
online Feb_11 svc:/network/amandaidx/tcp:default

2) the tape was not working , due to a stuck tape in one of the driver. amtape won't recoginize the volume.

I used the mtx command to unload the volume from the driver (amtape eject 0 would not work, because it did not see a volume loaded in the driver)

mtx -f /dev/scsi/changer/c7t500104F000B87A69d0 unload 31 0

afterwards,

amtape umatlas inventory

show the correct information.

In the meanwhile, also run

amtape umatlas show

to scan all the slots in the changer.

3) verify the connections to server and client

amcheck umaltas -s amcheck umatlas -c [host]

4) force to run a dump on all the clients (since the last backup was about 2 months ago due to power off)

amdump umaltas

This will collect all the dumped files from the clients, and write to the /pool1/amholding directory.

Amanda will decide when to flush (amflush) the contents in the holding disk to the tape library.

Note. a full dump(backup) of all clients has about 1.7TB, so it takes a few hours..by default, amanda starts the amdump at middle night, but it won't run if there is already a amdump process running.

-- BenMeekhof - 10 Jul 2007

Topic attachments
I	Attachment	Action	Size	Date	Who	Comment
patch	amanda-afs-cornell-dflushvol.patch	manage	4 K	14 Aug 2007 - 19:16	BenMeekhof	patch for amanda-afs (needed to compile)
gz	amanda-afs.tar.gz	manage	42 K	14 Aug 2007 - 19:15	BenMeekhof	AFS tools for Amanda

Topic revision: r30 - 18 Feb 2019, WenjingWu

AGLT2

Copyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback