TSM 6.4 and VMware

Some TSM and VMware basics

TSM 6.4 introduced some new features for backing up VMware Virtual Machines. TSM will back up any VMware guests that are supported by VMware. There's lots of new names involved with VM, especially if you've not worked with VMware very much.

VMware Vsphere is the name given to latest incarnation of the VMware virtualisation platform.
A VM is a Virtual Machine, or simply a virtual Windows or Linux server. VMs are hosted on physical boxes called Hypervisors, or ESX devices and ESX devices are grouped together into clusters.
A VMware vSphere Datacenter contains the physical components needed to host a Vmware system, such as x86 virtualization servers, storage networks and arrays, IP networks, a management server, and desktop clients. Don't confuse this with the popular definition of a datacenter, which is a building that houses IT equipment, possibly including several VMware datacenters.

A VMware vCenter server is used to centrally manage a Vsphere infrastructure from a single console. TSM comes in at this point by providing a plug-in to the vCenter server, so backup and recovery services can be managed from the same central point.
If you run several vCenter servers, then you need to install a plug-in on each of them and each plug-in can just manage its own vCenter. You cannot restore VMs between Vcenter servers.
However, TSM continues to provide a standard command line interface to VM, which is much faster and easier to use than the plug-in.

To complete the physical picture, a vStorage Backup Server is a dedicated Windows machine, physical or virtual, that interfaces between the VMware vSphere infrastructure and the Tivoli Storage Manager server. You might need more than one Backup server in each DataCenter, depending on your workload. It hosts various TSM clients.

  • A standard TSM client is used to backup the vStorage server
  • One or more Data Mover clients are required to actually copy the data from vSphere to the TSM server. Each Data Mover will be allocated a set of ESX devices to manage. TSM will automatically backup all powered on VMs that are allocated to the ESX hypervisors that are defined to a Data Mover, so no configuration changes are needed as VMS are added, deleted or moved. However as VMs can be moved between ESX hypervisors and so potentially between vStorage Backup servers, all the VMs in a DataCenter should be backed up to a single proxy node. This means that restores can happen from any Datamover on a vStorage Backup server.
  • A Data center client is used for data center TDP.
  • A Vcenter client is required for command line prccessing.

TSM supports two different backup and restore types
A full VM backup takes a VMware guest disk snapshot then copies the VM configuration information and a block level copy of VM disks to the TSM Server. The FULL-VM backup operation does not require a Data Protection for VMware license. Full VM backups are excellent de-dup candidates as they will produce loads of duplicate operating system files
An INCREMENTAL VM backup will backs up only those VM blocks that have changed since the last backup completed, provided that VMware's Change Block Tracking (CBT) is enabled. If CBT is not enabled, a full VM backup is taken and a warning message issued. This backup requires a Data Protection for VMware licence. If CBT is enabled, then TSM will not backup empty areas of VM disks either.
TSM uses the VMsnap function, which can be used to take a consistent backup of SQL databases hosted on a VM, as it quiesces database activity as part of the snap.

Changes required at the Server

You will probably need a new policy domain and associated objects for your VM backups. Exactly how you set these up will depend on your site standards. One recommendation is to define two management classes, one to keep the VMware metadata on disk, and one to place the actual data on a storage pool that migrates to tape. Of course you may decide to write direct to tape and even to use LAN free. IBM advises that you store CTL files on disk to speed up restores, and it certainly does help. In general terms, the size of the metadata storage pool needs to be about 1% of the total VM backup capacity.

To make an example a bit more useful, I'm going to propose a single DataCenter called VCSDC001 with 16 ESX Hypervisors called esx001 - esx016. These will be hosted on two vStorage Backup Servers called TSMBACK01 and TSMBACK02, with each server managing 8 ESX hypervisors. The whole lot will be proxied by a single node called VCSDC001_PX.

Client Node Definitions

You need several client nodes as follows.
2 'normal' TSM clients just used to backup the data on the vStorage servers.
4 Data Movers, called TSMBACK01_DM, TSMREST01_DM, TSMBACK02_DM, TSMREST02_DM. The reason you need 4 Data Movers clients is because a Data Mover can only be used for one operation at a time, so if it is being used for backups, it cannot be used for restores. So here we have 2 Data Movers dedicated for backups and 2 for restores.

The Data Mover nodes need to be defined with a high MAXSESSIONS parameter. The exact value to use depends on how many sessions you will allow to run in parallel and is given by the formula
Total number of sessions = (2 + 2 * VMMAXParallel)
The reason you need this many is because you need 1 session for main thread, 2 sessions per vm operation - one for the consumer and one for the API, and 1 more session for a consumer that runs at the start of the backup, starts, connects to the server, checks that there's nothing to do and closes the session.
So for our example, if we backup 8 VMs in parallel, we would need to set MAXSESSIONS to at least 2+2*8=18. If you set MAXSESSIONS lower than 18 you will probably get a 'ANS1351E Session rejected: All server sessions are currently in use' error.

1 proxy node, called VCSDC001_PX, which is used to collate all the backups from the vStorage servers.
You also need 2 more clients defined on the TSM server, A 'Virtual Center' client that is used by the TDP for VMware on this DataCenter, which will be called VCSDC001_DC and another client for the Vcenter command line, which will be called VCSDC001_VCLI

You need to set up a fairly complicated set of TSM client proxy relationships, so each client will store backups in the correct place, and can access all VMs for restores. This is illustrated in the picture below, and the proxy commands that you need to run on your TSM server to set these relations up are:

grant proxynode target=VCSDC001_PX agent=VCSDC001_DC,VCSDC001_VCLI
grant proxynode target=VCSDC001_DC agent=TSMBACK01_DM, TSMREST01_DM, TSMBACK02_DM, TSMREST02_DM,VCSDC001_VCLI
grant proxynode target=TSMBACK01_DC agent=VCSDC001_VCLI

TSM VMware proxy node relationships

Client Schedules

A typical VMware schedule definitions looks like this - note that all the ESX hypervisors that will be scanned for VMs are explicitly defined in the schedule, and that the schedule uses the -asnodename option so the backups are stored under the VCSDC001_PX proxy node. The Data Mover client is assocated with this schedule.

DEFINE SCHEDULE VMWARE VMWARE_DAILY_INCR Type=Client DESCription="Daily if incremental backup of VM servers" ACTion=Backup SUBACTion=VM Options='-vmfulltype=vstor -vmbackuptype=fullvm -asnodename=VCSDC001_PX -domain.vmfull="VMHOST=esx001.machine.group, esx002.machine.group,esx003.machine.group, esx004.machine.group,esx005.machine.group, esx006.machine.group,esx007.machine.group, esx008.machine.group" -MODE=IFIncremental' STARTDate=12/07/2013 STARTTime=21:45:00 SCHEDStyle=Enhanced DAYofweek=ANY

DEFINE ASSOCIATION VMWARE VMWARE_DAILY_INCR TSMBACK01_DM

VM, Disk and File Exclusions

To exclude individual VMs from automatic backup, you can include them in the schedule OBJECTS statement with a '-' in front:

UPD SCHEDULE VMWARE VMWARE_DAILY_INCR objects='-vmfulltype=vstor -vmbackuptype=fullvm -domain.vmfull="VMHOST=esx003.machine.group;-VM=VMname1,VMname1"'

This will override the settings in the client option DOMAIN.VMFULL statement. You can also exclude entire VMs in the DOMAIN VMFULL statement like this

domain.vmfull "all-vm;-vm=VM004,New Test Server"

You can exclude multiple VMs separated by commas as shown. Note that as the second VM name includes spaces, quotes are required, but you quote the ENTIRE domain statement, NOT just the individual VM name. If you just put quotes round the VMname the exclude will be ignored.
You can exclude individual disks within a VM by adding exclude statements to your datamover dsm.opt file like this - see below for datamover option file details. However you cannot use this option to bind individual disks to a different management class, the management class applies to the whole VM. To pick up a management class you need an include statement

EXCLUDE.VMDISK vmname "Hard Disk 3"

If you are backing up through a proxy node and you want to exclude a file from a specific VM, then you need to include the VM name in the exclude list. For example to exclude the temporary internet folder

exclude \\vmname\C:\Users\*\AppData\Local\Microsoft\Windows\INetCache\*

To exclude this folder from every VM, use

exclude \\*\C:\Users\*\AppData\Local\Microsoft\Windows\INetCache\*

To exclude all files and directories on the C: drive for all virtual machines but still backup the root directory entry use:

exclude \\*\c$\*
exclude.dir \\*\c$\*

You can use equivalent statements to bind individual files to a specfic management class

include \\*\c$\...\*.mp3 mclass3

Definition at the Windows Clients

Each vStorage server will need a standard dsm.opt file for normal client processing, and also the following clients.
The DataCenter client, one on each vStorage server and identical on both, which we will call dsm.VCSDC001.opt. A values of a lot of the parameters in these files will depend on your site names and standards, but I give you these as examples. In particular, the DOMAIN.VMFULL will depend on your domain name standards and the VMCUSER will be the userid that you use to connect to vSphere.

* DATACENTER TSM OPT FILE

commmethod   TCPIP
TCPSERVERADDRESS   TSM_SERVER_DNS
TCPPORT   1500
PASSWORDACCESS   GENERATE

MANAGEDSERVICES   schedule webclient
DATEFORMAT   2

NODENAME   VCSDC001_VC
DOMAIN.VMFULL   "VMHOST=TSMBACK01.DOMAIN.NAME.COM"

VMCHOST   TSMBACK01.DOMAIN.NAME.COM
VMCUSER   VSphere_userid
VMFULLTYPE   VSTOR
VMBACKUPTYPE   FULLVM
VMCPW ****

** Management class control options
*VMMC   MC_VM_TAPE
VMCTLMC   MC_VM_CTL

** Log files
ERRORLOGNAME   "D:\SYSTEM\IBM\TSM\baclient\dsmerror.TSMBACK01.log"
ERRORLOGRETENTION   30 d
SCHEDLOGNAME   "D:\SYSTEM\IBM\TSM\baclient\dsmsched.TSMBACK01.log"
SCHEDLOGRETENTION   30 d

The Data Mover clients, two for each VStorage server. Only one example is given here. The file names will be dsm.tsmback01_DM.opt, dsm.tsmrest01_DM.opt on TSMBACK01, with equivalent names on TSMBACK02.

* DATAMOVER TSM OPT FILE

commmethod   TCPIP
TCPSERVERADDRESS   TSM_SERVER_DNS
TCPPORT   1500
PASSWORDACCESS   GENERATE

MANAGEDSERVICES   schedule webclient
DATEFORMAT   2

NODENAME   TSMBACK01_DM
DOMAIN.VMFULL    "VMHOST=esx001.machine.group,esx002.machine.group,
esx003.machine.group,esx004.machine.group,
esx005.machine.group,esx006.machine.group,
esx007.machine.group,esx008.machine.group "

VMCHOST   TSMBACK01.DOMAIN.NAME.COM
VMCUSER   vSphere_userid
VMFULLTYPE   VSTOR
VMBACKUPTYPE   FULLVM
VMCPW    ****

*VMMC   MC_VM_TAPE
VMCTLMC   MC_VM_CTL

ERRORLOGNAME   D:\SYSTEM\IBM\TSM\baclient\dsmerror.TSMBACK01_DM.log"
ERRORLOGRETENTION   30 d
SCHEDLOGNAME   "D:\SYSTEM\IBM\TSM\baclient\dsmsched.TSMBACK01_DM.log"
SCHEDLOGRETENTION   30 d

The VCENTER option file, used by the vCenter for TDP plugin. We will call this one dsm.VCSDC001_VC.opt, and it will be required and identical on both TSMBACK servers

* VCENTER TSM OPTIONS FOR TDP Plugin

commmethod   TCPIP
TCPSERVERADDRESS   TSM_SERVER_DNS
TCPPORT   1500
NODENAME   VCSDC001_VC
SCHEDMODE   PROMPTED
DATEFORMAT   2
PASSWORDACCESS   GENERATE
SUBDIR   YES

** VMware connection options
VMCHOST   TSMBACK01.DOMAIN.NAME.COM
VMCUSER   VSPHERE_USERID
VMFULLTYPE   VSTOR
VMBACKUPTYPE   FULLVM
VMCPW ****

** Log files
ERRORLOGNAME   "D:\SYSTEM\IBM\TSM\baclient\dsmerror.TSMBACK01_VC.log"
ERRORLOGRETENTION   30 d
SCHEDLOGNAME   "D:\SYSTEM\IBM\TSM\baclient\dsmsched.TSMBACK01_VC.log"
SCHEDLOGRETENTION   30 d

The vSphere password in the above dsm.opt files is commented out as we do not want it to appear in plain text, so it must be created for each option file by command instead.
Run the following commands as admin in D:\SYSTEM\IBM\TSM\baclient, but note that the dsm.opt names are the ones I used above, and may not correspond to your site standards. VSphere_password is your password of course.

dsmc set password -type=vm TSMBACK01.DOMAIN.NAME.COM vSphere_userid vSphere_password -optfile=dsm.VCSDC001.opt
dsmc set password -type=vm TSMBACK01.DOMAIN.NAME.COM vSphere_userid vSphere_password -optfile=dsm.VCSDC001_VC.opt

dsmc set password -type=vm TSMBACK01.DOMAIN.NAME.COM vSphere_userid vSphere_password -optfile=dsm.TSMBACK01_DM.opt
dsmc set password -type=vm TSMBACK01.DOMAIN.NAME.COM vSphere_userid vSphere_password -optfile=dsm.TSMREST01_DM.opt

Several Windows services need to be configured on the vStorage servers. As well as standard schedule services for the local backups, the following scheduler processes for each datamover node are required. These services are best defined using the dsmcutil command as you have more control over the parameters that you would get using the TSM GUI wizard.

dsmcutil install scheduler /name:"TSM Scheduler tsmback01 Data Mover" /optfile:D:\SYSTEM\IBM\TSM\baclient\dsm.tsmback01_DM.opt /autostart:no /password:node_password /errorlog:D:\SYSTEM\IBM\TSM\baclient\dsmerror.tsmback01_DM.log /schedlog:D:\SYSTEM\IBM\TSM\baclient\dsmsched.tsmback01_DM.log /startnow:no /node:TSMBACK01_DM

dsmcutil install cad /name:"TSM CAD tsmback01 Data Mover" /cadschedname:"TSM Scheduler tsmback01 Data Mover " /optfile:D:\SYSTEM\IBM\TSM\baclient\dsm.tsmback01_DM.opt /autostart:yes /password:node_password /startnow:no /node:TSMBACK01_DM

dsmcutil install remote /name:"TSM Agent tsmback01 Data Mover" /optfile:D:\SYSTEM\IBM\TSM\baclient\dsm.tsmback01_DM.opt /partnername:"TSM CAD tsmback01 Data Mover " /password:node_password /startnow:no /node:TSMBACK01_DM

dsmcutil install RemoteAgent /name:"TSM RemoteAgent tsmback01 Data Mover" /optfile:D:\SYSTEM\IBM\TSM\baclient\dsm.tsmback01_DM.opt /partnername:"TSM CAD BAK2X001 Data Mover" /password:node_password /startnow:no /node:TSMBACK01_DM

Updating the TDP option file

This file is usually held in C:\Program Files (x86)\Common Files\Tivoli\TDPVMware\VMwarePlugin\scripts\vmcliprofile. Most of the parameters can be left at the supplied default values.
The VE_DATACENTER_NAME profile option is case sensitive it must exactly match the datacenter name as used by VMware. You can manage multiple Data Centers from one vCenter TDP, and then they would be specified with multiple VE_DATACENTER_NAME entries. These must be unique and have separate TSM datacenter nodenames. The syntax is
VE_DATACENTER_NAME DataCenter::TSM_Node
Most of the options in this file can be left as the defaults, the ones you need to change are below, but obviously use your own values for userids and port numbers.

VE_TSM_SERVER_NAME TSM_SERVER_DNS
VE_TSM_SERVER_PORT 1500
VE_TSMCLI_NODE_NAME VCSDC001_CLI
VE_VCENTER_NODE_NAME VCSDC001_PX
VE_DATACENTER_NAME DC_001::VCSDC001_DC
..... loads of default options

Once you change this profile, you need to restart the windows Data Protection Services

Now you need to set the TDP password that is used to connect to vSphere. Open a command line as Administrator then run commands like these, with your own values substituted. The password you need here is the one used by your CLI TSM client.
cd C:\Program Files (x86)\Common Files\Tivoli\TDPVMware\VMwarePlugin\scripts
echo CLI_password>pwd.txt
vmcli -f set_password -I pwd.txt
The vmcli command will delete that temporary password file. You can test that this works by running vmcli commands, for example

C:\Program Files (x86)\Common Files\Tivoli\TDPVMware\VMwarePlugin\scripts>
Vmcli -f inquire_config -t TSM

The final task is to configure the vCenter plugin on the vCenter GUI.
Start the VMware vCenter client GUI up, navigate to the Home directory from the address bar, then right at the bottom of the Home screen you will see a Solutions and Applications panel, and in there you should see a Data Protection for VMware vCenter plug-in icon. click on that icon, then edit the Tivoli Storage Manager server settings by clicking Configuration tab > Edit.Configuration. The stuff you need to input includes:

Administrator
Userid for Proxynode
Password
password for proxy node
Address
TSM server address
Port
TSM server port

If you have problems with this, go to Windows Services and check that 'IBM WebSphere Application Server V7.0 - TSMVEplugin' and 'Data Protection for VMware command-line interface' are both started.

Some tips for running backups and restores

Open up a command line and navigate to d:\system\ibm\tsm\baclient, then start s dsmc session, but you need to point to the data mover dsm.sys and log in as the proxy node like this.

dsmc -optfile=dsm.tsmback01_DM.opt -asnode=VCSDC001_PX

To see what VMs are configured, try the command

show VM all

To see if any backups exist for a VM, use the command

q VM vmname -inact

you can backup a single VM with the following commands.

backup vm vmname -vmbackuptype=fullvm -mode=ifincremental
backup vm vmname -vmbackuptype=fullvm -mode=iffull

If you are backing up as type HYPERVFULL, you can backup a multiple VMs using the VMLIST option.

backup vm vmlist="vm1,vm2,vm3" -vmbackuptype=HYPERVFULL

You can exclude individual hard disks from a vm backup like this

backup vm "vmname:-vmdk=Hard Disk 3" -vmbackuptype=fullvm -mode=iffull

you can't restore a single file from the command line, and it is impossible to restore over the top of an existing VM, you need to restore files to a new name

restore vm vmname -vmname="restvmname"

If you want to recover VMs using the GUI then this is quite straightforward, but you will not see error messages or progress messages as you do with the command line. One thing to be aware of is that when you want to select the VM to restore, click on the virtual machine name, not on the check box to the left of the virtual machine. If you do click on the select box you will see an error message like 'This node cannot be selected'.

Some VMware backup issues

A known restore issue with TSM 6.2 is that a backup of virtual machine fails with ANS1370E and ANS4148E RC 114 when backing up to a proxy node.
This is because the administrator user of the proxy node is not an owner of the agent node and does not have authority to run a backup. Error messages on the TSM server would show something like
ANR0473W Session 7285 for administrator admin_id ( ) refused - administrator name does not have the correct level of authority over client node. (SESSION: 7285)

To fix the problem, use the grant auth command to give the proxy node permission to backup the client.

grant authority proxy_admin_name class=node authority=owner node=vm_node_name

 

 

If you see an error 'ANS4151E Failure mounting Virtual Machine 'MY_VM_MACHINE'. RC=115' then check the client dsmerror.log and you should also see messages like

This can be caused by a shortage of memory on the VM guest. Check the memory size on the VM Guest OS and it is is too small then increase it.

File level Restores

It is possible to recover an individual from from a snapshot, to do this you mount the backup tape as a 'drive' on your TSM backup server, then you can copy the data over. The process in outline is

  • Open up the 'Data Protection for VMware Recovery Agent' GUI
  • Select the 'new Tivoli Storage Manager Server' window and fill in the details for your server
  • Set the 'Data Access - Storage Type' window as appropriate, usually to 'tape'
  • Pick out the VM that you need to restore the file from in the 'Virtual Machine' window, select the date that you want the snapshot from, and select the disk name
  • Click on the 'Mount' button, select the drive you want to restore from and the drive that you want to mount to on the backup server. The default mount drive is e:
  • Click OK and wait for your tape to be mounted
  • If you see a message that the TDP Mount Virtual Volume Driver is not installed, just click 'OK' to get it installed and the tape should mount OK
  • Now open Explorer and navigate to the file you want recovered. If you are reading a tape, this could take a little while. Drag and drop the file onto the VM that you are restoring to.
  • Finally, go back to the TDP GUI, highlight the drive you were working with and dismount it with the dismount button to free up the drive