TSM and VMware

The evolution of TSM VMware support

TSM 6.4 was the first release to provide comprehensive VMware support, but the configuration was manual and complicated, the vCenter plugin was clunky, and we had a few problems with backups and restores.
TSM 7.1 introduced a GUI that was intended to simplify the initial configuration, and that GUI meant you could access the vCenter Server without needing a plugin. You could recover individual MSSQL databases from a VM backup and 'instantly' restore a full VM from a snapshot. Also you could backup and recover a VM that was hosting a Microsoft Active Directory domain controller.
SP 8.1 continues to build on the VMware backup experience, with a new VMware plugin design, VMware tagging which simplifies including or excluding VMs and disks and provides easier backup monitoring and history via the vSphere interface. VMDKs up to 8TB are now supported, with increased maximum parallelism for both backups and restores to speed them up.

Some TSM VMware basics

VMware Vsphere is the VMware virtualisation platform.
A VM is a Virtual Machine, or simply a virtual Windows or Linux server. VMs are hosted on physical boxes called Hypervisors, or ESX devices and ESX devices are grouped together into clusters.
A VMware vSphere Datacenter contains the physical components needed to host a Vmware system, such as x86 virtualization servers, storage networks and arrays, IP networks, a management server, and desktop clients. Don't confuse this with the popular definition of a datacenter, which is a building that houses IT equipment, an IT datacenter could possibly contain several VMware datacenters.

A VMware vCenter server can be used to centrally manage a Vsphere infrastructure from a single console. TSM comes in at this point by providing an interface to the vCenter server, so backup and recovery services can be managed from the same central point.
However, TSM continues to provide a standard command line interface to VM, which is much faster and easier to use than the plug-in.

To complete the physical picture, a vStorage Backup Server is a dedicated Windows machine, physical or virtual, that interfaces between the VMware vSphere infrastructure and the Tivoli Storage Manager server.

back to top


Initial Configuration using the 7.1 vSphere GUI

IBM recommends that you do most of your configuration work using the vSphere GUI, as it is used to run manual and scheduled backups, to recover VMs and to run reports. You can either access this GUI as a plugin to the vCenter Server, or as a stand-alone web browser GUI. You normally install this GUI on the vStorage Backup Server, but you can install it anywhere that has network connectivity to the vStorage Backup Server, the TSM server and the vCenter Server. However the GUI performance suffers if you have a large environment with lots of VMs.

First, check your installation documentation for the current list of pre-requisite actions that you need to take before starting the configuration, and complete those tasks. They are mainly to make sure you have a network connection between the component servers, and to set some Windows parameters. Once you have completed this, the install tasks are:

  • Log into the VMware vSphere GUI using the vCenter user name and password.
  • From the 'getting Started' window go to the Configuration window and click Run Configuration Wizard.
  • Follow the instructions on the wizard, checking settings and adding data as necessary and go through to completion.

You have now installed Data Protection for VMware on your server. This is obviously much easier than the manual method described below, but the downside is that you will not have developed the same level of expertise and knowledge as you would have done if you did the install manually.
You should end up with the following:

For a vSphere install, the vCenter node, VMCLI node, datacenter nodes, and data mover nodes are registered on the TSM server.
For a vCloud install, the vCloud Director node, VMCLI node, Provider VDC nodes, Organization VDC nodes, and data mover nodes are registered on the TSM server.
The proxy relationships are all defined for these nodes, the vmcliprofile is updated and set and the local VMCLI password is set.
You should see four user interfaces, the vSphere GUI, the Recovery Agent, the vCloud GUI and my old favorite, the command-line interface. The full and offical names forHowever

The Recovery Agent GUI lets you take snapshots from the TSM server and then mount them on the Data Mover Node. If you want to recover a few files, the snapshot content can be viewed in read only mode and the required files copied over to the original. If you want to restore an entire VM, you can use these snapshots to instantly restore a VM, so you access the data from the snapshot while it is being copied in the background.

The vCloud GUI is used to backup and recover vApps and organization vDCs in a vCloud Director environment. This GUI is accessed from a URL, there is more detail about VMware vCloud management below

The vSphere command-line interface is still there. While Tivoli states tha the vSphere GUI is the primary way to manage backups and restores I always feel that you get much better control and messages with a command line. The command line works with both vSphere and vCloud environments. However the fastest way to manage the environment is just to use the Windows command line and dsmc commands.

The TSM VMware Nodes

OK, so we've mentioned lots of nodes above. What do they all do? A Node is a TSM client, physical or virtual, which runs some of the TSM software necessary to backup VMs to a TSM server. Every node is registered on the TSM server and has a unique name to identify it. The names will be specific to your site. A typical vSphere environment will have:

At least one data mover node. This node represents a specific Tivoli Storage Manager backup-archive client that "moves data" from one system to another. If you are running a small environment where the VMs are backed up by a single client, the VM data is stored directly under the data mover node. That is, if you run a 'query filespace nodename *' command on the TSM server with the name of the data mover node substituted for nodename, you will see all the VMs backed up to that node listed as file spaces.

A Proxy, or Datacenter node. In a larger environment, several data movers are used to back up a complete virtual environment, such as a VMware datacenter. Although the backup work is distributed among multiple data movers, the VM data is stored in a single shared node and this shared node is called the datacenter node or proxy node. The reason for this is that if you backup files to a shared node, then you can recover any VM from any datamover node.

An optional VMCLI node. If you use the vSphere GUI for management, then in a large vSphere virtual environment with several data movers and maybe several datacenters, you need a third node to communicate among the nodes and TSM server and this node is the VMCLI node. It connects the command-line interface to the TSM server and the TSM data mover node. As this node is just used for communication, it does not need a TSM client acceptor or scheduler service.

back to top


Configuring a VMware backup server manually

Changes needed at the TSM server

You will probably need a new policy domain and associated objects on your TSM server, for your VM backups. Exactly how you set these up will depend on your site standards. You will need two management classes, one to keep the VMware metadata on disk, and one to place the actual data on a storage pool that migrates to tape. Of course you may decide to write direct to tape and even to use LAN free. IBM advises that you store CTL files on disk to speed up restores, and it certainly does help. In general terms, the size of the metadata storage pool needs to be about 1% of the total VM backup capacity.

To make the example configuration below a bit easier to follow, I'm going to propose a single DataCenter called VCSDC001 with 16 ESX Hypervisors called esx001 - esx016. These will be hosted on two vStorage Backup Servers called TSMBACK01 and TSMBACK02, with each server managing 8 ESX hypervisors. The whole lot will be proxied by a single node called VCSDC001_PX.

Client Node Definitions

You need several client nodes as follows.
2 'normal' TSM clients just used to backup the data on the vStorage servers.
4 Data Movers, called TSMBACK01_DM, TSMREST01_DM, TSMBACK02_DM, TSMREST02_DM. The reason you need 4 Data Movers clients is because a Data Mover can only be used for one operation at a time, so if it is being used for backups, it cannot be used for restores. So here we have 2 Data Movers dedicated for backups and 2 for restores.

The Data Mover nodes need to be defined with a high MAXNUMMPOINTS parameter. The exact value to use depends on how many sessions you will allow to run in parallel and is given by the formula
Total number of sessions = (2 + 2 * VMMAXParallel)
The reason you need this many is because you need 1 session for main thread, 2 sessions per vm operation - one for the consumer and one for the API, and 1 more session for a consumer that runs at the start of the backup, starts, connects to the server, checks that there's nothing to do and closes the session.
So for our example, if we backup 8 VMs in parallel, we would need to set MAXNUMMPOINTS to at least 2+2*8=18.

1 proxy node, called VCSDC001_PX, which is used to collate all the backups from the vStorage servers.

You might need to set up TSM client proxy relationships, depending on how many nodes you configure. Each client will store backups in the correct place, and can access all VMs for restores. This is illustrated in the picture below, and the proxy commands that you need to run on your TSM server to set these relations up are:

grant proxynode target=VCSDC001_DC agent=TSMBACK01_DM, TSMREST01_DM, TSMBACK02_DM, TSMREST02_DM

Client Schedules

A typical VMware schedule definitions looks like this - note that all the ESX hypervisors that will be scanned for VMs are explicitly defined in the schedule, and that the schedule uses the -asnodename option so the backups are stored under the VCSDC001_PX proxy node. The Data Mover client is assocated with this schedule.

DEFINE SCHEDULE VMWARE VMWARE_DAILY_INCR Type=Client DESCription="Daily if incremental backup of VM servers" ACTion=Backup SUBACTion=VM Options='-asnodename=VCSDC001_PX -MODE=IFIncremental' STARTDate=today STARTTime=21:45:00 SCHEDStyle=Enhanced DAYofweek=ANY

DEFINE ASSOCIATION VMWARE VMWARE_DAILY_INCR TSMBACK01_DM

VM, Disk and File Exclusions

To exclude individual VMs from automatic backup, you can include them in the schedule OBJECTS statement with a '-' in front:

UPD SCHEDULE VMWARE VMWARE_DAILY_INCR objects='-VM=VMname1,VMname1'

This will override the settings in the client option DOMAIN.VMFULL statement.

You can exclude multiple VMs separated by commas as shown.

You can use equivalent statements to bind individual files to a specfic management class

include \\*\c$\...\*.mp3 mclass3

Definition at the Windows Clients

Each vStorage server will need a standard dsm.opt file for normal client processing, and also the following clients.

The Data Mover clients, two for each VStorage server. Only one example is given here. The file names will be dsm.tsmback01_DM.opt, dsm.tsmrest01_DM.opt on TSMBACK01, with equivalent names on TSMBACK02.

* DATAMOVER TSM OPT FILE

commmethod   TCPIP
TCPSERVERADDRESS   YOUR_TSM_SERVER_DNS
TCPPORT   YOUR_TSM_SERVER_PORT
PASSWORDACCESS   GENERATE

MANAGEDSERVICES   schedule
DATEFORMAT   2

NODENAME   
DOMAIN.VMFULL    "VMHOST=esx001.machine.group,esx002.machine.group,
esx003.machine.group,esx004.machine.group,
esx005.machine.group,esx006.machine.group,
esx007.machine.group,esx008.machine.group "

VMCHOST   VcenterServerDomainName
VMCUSER   vSphere_userid
VMFULLTYPE   VSTOR
VMBACKUPTYPE   FULLVM
VMCPW    ****

*VMMC   MC_VM_TAPE
VMCTLMC   MC_VM_CTL

ERRORLOGNAME   D:\SYSTEM\IBM\TSM\baclient\dsmerror.TSMBACK01_DM.log"
ERRORLOGRETENTION   30 d
SCHEDLOGNAME   "D:\SYSTEM\IBM\TSM\baclient\dsmsched.TSMBACK01_DM.log"
SCHEDLOGRETENTION   30 d

If you chose to use the vSphere interface then you will need opt files for those nodes too, but they are not mentioned here to save a bit of space.

The vSphere password in the above dsm.opt file is commented out as we do not want it to appear in plain text, so it must be created for each option file by command instead.
Run the following commands as admin in D:\SYSTEM\IBM\TSM\baclient, but note that the dsm.opt names are the ones I used above, and may not correspond to your site standards. VSphere_password is your password of course.

dsmc set password -type=vm TSMBACK01.DOMAIN.NAME.COM vSphere_userid vSphere_password -optfile=dsm.TSMBACK01_DM.opt
dsmc set password -type=vm TSMBACK01.DOMAIN.NAME.COM vSphere_userid vSphere_password -optfile=dsm.TSMREST01_DM.opt

Several Windows services need to be configured on the vStorage servers. As well as standard schedule services for the local backups, the following scheduler processes for each datamover node are required. These services are best defined using the dsmcutil command as you have more control over the parameters that you would get using the TSM GUI wizard.

dsmcutil install scheduler /name:"TSM Scheduler tsmback01 Data Mover" /optfile:D:\SYSTEM\IBM\TSM\baclient\dsm.tsmback01_DM.opt /autostart:no /password:node_password /errorlog:D:\SYSTEM\IBM\TSM\baclient\dsmerror.tsmback01_DM.log /schedlog:D:\SYSTEM\IBM\TSM\baclient\dsmsched.tsmback01_DM.log /startnow:no /node:TSMBACK01_DM

dsmcutil install cad /name:"TSM CAD tsmback01 Data Mover" /cadschedname:"TSM Scheduler tsmback01 Data Mover " /optfile:D:\SYSTEM\IBM\TSM\baclient\dsm.tsmback01_DM.opt /autostart:yes /password:node_password /startnow:no /node:TSMBACK01_DM

Updating the TDP option file

You would only need to do this if you use the vSphere GUI interface. This file is usually held in C:\Program Files (x86)\Common Files\Tivoli\TDPVMware\VMwarePlugin\scripts\vmcliprofile. Most of the parameters can be left at the supplied default values.
The VE_DATACENTER_NAME profile option is case sensitive it must exactly match the datacenter name as used by VMware. You can manage multiple Data Centers from one vCenter TDP, and then they would be specified with multiple VE_DATACENTER_NAME entries. These must be unique and have separate TSM datacenter nodenames. The syntax is
VE_DATACENTER_NAME DataCenter::TSM_Node
Most of the options in this file can be left as the defaults, the ones you need to change are below, but obviously use your own values for userids and port numbers.

VE_TSM_SERVER_NAME TSM_SERVER_DNS
VE_TSM_SERVER_PORT 1500
VE_TSMCLI_NODE_NAME VCSDC001_CLI
VE_VCENTER_NODE_NAME VCSDC001_PX
VE_DATACENTER_NAME DC_001::VCSDC001_DC
..... loads of default options

Once you change this profile, you need to restart the windows Data Protection Services

Now you need to set the TDP password that is used to connect to vSphere. Open a command line as Administrator then run commands like these, with your own values substituted. The password you need here is the one used by your CLI TSM client.
cd C:\Program Files (x86)\Common Files\Tivoli\TDPVMware\VMwarePlugin\scripts
echo CLI_password>pwd.txt
vmcli -f set_password -I pwd.txt
The vmcli command will delete that temporary password file. You can test that this works by running vmcli commands, for example

C:\Program Files (x86)\Common Files\Tivoli\TDPVMware\VMwarePlugin\scripts>
Vmcli -f inquire_config -t TSM

The final task is to configure the vCenter plugin on the vCenter GUI.
Start the VMware vCenter client GUI up, navigate to the Home directory from the address bar, then right at the bottom of the Home screen you will see a Solutions and Applications panel, and in there you should see a Data Protection for VMware vCenter plug-in icon. click on that icon, then edit the Tivoli Storage Manager server settings by clicking Configuration tab > Edit.Configuration. The stuff you need to input includes:

Administrator
Userid for Proxynode
Password
password for proxy node
Address
TSM server address
Port
TSM server port

If you have problems with this, go to Windows Services and check that 'IBM WebSphere Application Server V7.0 - TSMVEplugin' and 'Data Protection for VMware command-line interface' are both started.

back to top


Some tips for running backups and restores

Backup types

TSM supports two different backup and restore types
A full VM backup takes a VMware guest disk snapshot then copies the VM configuration information and a block level copy of VM disks to the TSM Server. The FULL-VM backup operation does not require a Data Protection for VMware license. Full VM backups are excellent de-dup candidates as they will produce loads of duplicate operating system files
An INCREMENTAL VM backup will backs up only those VM blocks that have changed since the last backup completed, provided that VMware's Change Block Tracking (CBT) is enabled. If CBT is not enabled, a full VM backup is taken and a warning message issued. This backup requires a Data Protection for VMware licence. If CBT is enabled, then TSM will not backup empty areas of VM disks either.
TSM uses the VMsnap function, which can be used to take a consistent backup of SQL databases hosted on a VM, as it quiesces database activity as part of the snap.

Open up a command line and navigate to d:\system\ibm\tsm\baclient, then start s dsmc session, but you need to point to the data mover dsm.sys and log in as the proxy node like this.

dsmc -optfile=dsm.tsmback01_DM.opt -asnode=VCSDC001_PX

To see what VMs are configured, try the command

show VM all

To see if any backups exist for a VM, use the command

q VM vmname -inact

you can backup a single VM with the following commands.

backup vm vmname -vmbackuptype=fullvm -mode=ifincremental
backup vm vmname -vmbackuptype=fullvm -mode=iffull

You can exclude individual hard disks from a vm backup like this

backup vm "vmname:-vmdk=Hard Disk 3" -vmbackuptype=fullvm -mode=iffull

TSM 7.1 backup enhancements

TSM 7.1 can backup and restore VMCloud templates and vApps.
A VM template is a master image of a VM. The template can include an installed guest operating system and a set of applications.
A vApp is a logical entity that consists of one or more VMs that make up an application. You probably want to consistently backup the whole application together, and if a restore is required, you may want to restore the whole application, that is, all the VMs in the Vapp, to the same point in time. Managing by vApp makes this easier.

Active Directory Backups

TSM uses VMware Tools Snapshots for VM backups, and by default it takes the snapshots in quiesced mode. If the VM contains Active Directory components, then TSM can have problems trying to take a quiesced snaoshot. Microsoft suggests that the best solution is to backup Active Directory VMs from non-quiesced snapshots. To do this, you need to add a parameter

INCLUDE.VMSNAPSHOTATTEMPTS vmname n1 n2

Where vmname is the name of the VM that contains AD components, n1 is the number of attempts to take a snapshot in quiesced mode and n2 is the number of attempts to take a snapshot in non-quiesced mode. The default values are n1=2, n2=0.
You can use wildcards on the VMname, so if all your Active Directory VMs start AD, you could code the following

INCLUDE.VMSNAPSHOTATTEMPTS AD* 0 2

Common Backup Errors

If you run a full VM backup on a 32bit Windows OS, it may fail with an ANS9365E error and API return code 60 as shown in this error log extract -

ANS9365E VMware vStorage API error.
...
API return code : 60
...
ANS5250E An unexpected error was encountered.
TSM function name : vmVddkStartOffloadMount
TSM function : snapMoRefP is null
TSM return code : 115
TSM file : vmbackvddk.cpp (1957)
ANS4151E Failure mounting Virtual Machine 'PASVM002A'. RC=115
ANS4150E Incremental backup of Virtual Machine 'PASVM002A' failed with RC 115

One reason for this error is that there was a timeout, due to lack of memory in the VM guest. Check out how much memory is allocated to the VM, and if it is quite small, say about 2GB, then get it increased.

Restores

It is possible to recover files from snapshots by mounting a snapshot volume. The TSM 6.4 section has details on how this is done.

This command will run an instant restore of a vm called prd047. The temporary datastore is used to store the vm configuration data while the restore is in progress and the name must be unique.

dsmc restore vm prd047 -vmrest=INSTANTRestore -vmtempdatastore=temp_datastore_02

The command to restore a vm called dev25 is shown below. This command will restore from the most recent backup. If you want an earlier backup you can add the -pick option so see a list of available backups and then you can select the one you want.

dsmc restore vm dev25

The vCloud restore command to restore a vApp called vApp_Admin is shown below. You obviously need to substitute your own values for the org name, the orgvdc name and the vapp name.

restore vApp org=org_HR,orgvdc=prod,vapp=vApp_Admin

You cannot restore the Windows system state from a VMware snapshot. The reason is the the system state consists of more than just the files that are backed up by a snapshot, it also consists of the registry.

There are some restrictions on vCloud restores. It is not possible to restore a single VM template as TSM marks the vCloud template object as a single file.
It is possible to restore individual VMs that are contained in a vApp, but if you select a vApp for restore then you restore all the components within that vApp. If you restore vms within a vApp and that vApp still exists, then the restored vms are restored to the vApp. If the vApp does not exist then the VM is restored to the top-level default location on the target ESX host.

you can't restore a single file from the command line, and it is impossible to restore over the top of an existing VM, you need to restore files to a new name

restore vm vmname -vmname="restvmname"

If you want to recover VMs using the GUI then this is quite straightforward, but you will not see error messages or progress messages as you do with the command line. One thing to be aware of is that when you want to select the VM to restore, click on the virtual machine name, not on the check box to the left of the virtual machine. If you do click on the select box you will see an error message like 'This node cannot be selected'.

File level Restores

It is possible to recover an individual from from a snapshot, to do this you mount the backup tape as a 'drive' on your TSM backup server, then you can copy the data over. The process in outline is

  • Open up the 'Data Protection for VMware Recovery Agent' GUI
  • Select the 'new Tivoli Storage Manager Server' window and fill in the details for your server
  • Set the 'Data Access - Storage Type' window as appropriate, usually to 'tape'
  • Pick out the VM that you need to restore the file from in the 'Virtual Machine' window, select the date that you want the snapshot from, and select the disk name
  • Click on the 'Mount' button, select the drive you want to restore from and the drive that you want to mount to on the backup server. The default mount drive is e:
  • Click OK and wait for your tape to be mounted
  • If you see a message that the TDP Mount Virtual Volume Driver is not installed, just click 'OK' to get it installed and the tape should mount OK
  • Now open Explorer and navigate to the file you want recovered. If you are reading a tape, this could take a little while. Drag and drop the file onto the VM that you are restoring to.
  • Finally, go back to the TDP GUI, highlight the drive you were working with and dismount it with the dismount button to free up the drive

back to top


Manuals

If you want to know more about TSM 7.1 and VMware then search for the Installation and User guides, called.
IBM Tivoli Storage Manager for Virtual Environments Version 7.1: Data Protection for VMware Installation Guide
IBM Tivoli Storage Manager for Virtual Environments Version 7.1: Data Protection for VMware User's Guide

Monitoring VMware activity

Normally, if you want to find out what backups and restores have been running against a TSM client, you would use a command like this, where the ENTITY filter will pick up the clients that you want -

select activity,,NUMBER,entity,SCHEDULE_NAME from summary where ENTITY is like 'VE%' and start_time>(current_timestamp-1 days)

However, if you run this command against VMware clients you will not get the correct data. You need to use the SUMMARY_EXTENDED table instead as this has been specifically introduced for VMware clients. So the correct command to use is

select activity,,NUMBER,entity,SCHEDULE_NAME from SUMMARY_EXTENDED where ENTITY is like 'VE%' and start_time>(current_timestamp-1 days)

back to top