TSM 7.1 and VMware Backup and Recovery

What's new with VMware backups and TSM 7.1?

  • Probably the biggest change with VMware backups is that with TSM 7.1 the initial configuration exercise has been simplified. Configuring a new 6.4 system was complicated and difficult to understand, the 7.1 configuration is a simple GUI exercise.
  • TSM 7.1 lets you backup vCloud Director objects with access through a stand-alone web GUI.
  • To access the TSM 6.4 VMware Vcenter server you had to use a plugin to the VMware GUI and to use this you needed VMware access. TSM 7.1 also offers a stand alone web based GUI to access the vCenter Server.
  • You can now recover individual MSSQL databases from a VM backup.
  • You can 'instantly' restore a full VM from a snapshot.
  • You can backup and recover a VM that is hosting a Microsoft Active Directory domain controller, in both stand-alone and clustered environments.
  • If the primary TSM server fails then DP for VMware can automatically fail over to a secondary TSM server.

INITIAL CONFIGURATION

First, check your installation documentation for the current list of pre-requisite actions that you need to take before starting the configuration, and complete those tasks. They are mainly to make sure you have a network connection between the component servers, and to set some Windows parameters. Once you have completed this, the install tasks are:

  • Log into the VMware vSphere GUI using the vCenter user name and password.
  • From the 'getting Started' window go to the Configuration window and click Run Configuration Wizard.
  • Follow the instructions on the wizard, checking settings and adding data as necessary and go through to completion.

You have now installed Data Protection for VMware on your server. This is obviously much easier than the manual method that was needed for TSM 6.4, but the downside is that you will not have developed the same level of expertise and knowledge as you would have done if you did the install manually.
You should end up with the following:

For a vSphere install, the vCenter node, VMCLI node, datacenter nodes, and data mover nodes are registered on the TSM server.
For a vCloud install, the vCloud Director node, VMCLI node, Provider VDC nodes, Organization VDC nodes, and data mover nodes are registered on the TSM server.
The proxy relationships are all defined for these nodes, the vmcliprofile is updated and set and the local VMCLI password is set.
You should see four user interfaces, the vSphere GUI, the Recovery Agent, the vCloud GUI and my old favorite, the command-line interface. The full and offical names for these interfaces is the Data Protection for VMware vSphere GUI etc. but I find these names too big a mouthfull and so they will all appear abbreviated as above on this page.

If you opted to install the backup-archive client data mover on the same system and you selected the 'Create Services' option, then the following Windows services are all set up and running. They are the Client Acceptor Service, the Backup-Archive Scheduler Service, and the Remote Client Agent Service

DP for VMware Components

The User Interfaces

You will probably do most of your GUI work using the vSphere GUI, as it is used to run manual and scheduled backups, to recover VMs and to run reports. You can either access this GUI as a plugin to the vCenter Server, or as a stand-alone web browser GUI. You normally install this GUI on the vStorage Backup Server, but you can install it anywhere that has network connectivity to the vStorage Backup Server, the TSM server and the vCenter Server.
If you have several VM datacenters installed in your environment, you should consider managing each one from a separate vSphere GUI. The reasons for this are that:
Queries and processes will be faster as each GUI has fewer VMs to manage.
Each vSphere GUI will be given a unique VMCLI name and will have their own data mover nodes, so simplifying the node management.

The Recovery Agent GUI lets you take snapshots from the TSM server and then mount them on the Data Mover Node. If you want to recover a few files, the snapshot content can be viewed in read only mode and the required files copied over to the original. If you want to restore an entire VM, you can use these snapshots to instantly restore a VM, so you access the data from the snapshot while it is being copied in the background.

The vCloud GUI is used to backup and recover vApps and organization vDCs in a vCloud Director environment. This GUI is accessed from a URL, there is more detail about VMware vCloud management below

I'm pleased to see that the command-line interface is still there. While Tivoli states tha the vSphere GUI is the primary way to manage backups and restores I always feel that you get much better control and messages with a command line. The command line works with both vSphere and vCloud environments.

The TSM VMware Nodes

OK, so we've mentioned lots of nodes above. What do they all do? A Node is a TSM client, physical or virtual, which runs some of the TSM software necessary to backup VMs to a TSM server. Every node is registered on the TSM server and has a unique name to identify it. The names will be specific to your site. A typical vSphere environment will have:

At least one data mover node. This node represents a specific Tivoli Storage Manager backup-archive client that "moves data" from one system to another. If you are running a small environment where the VMs are backed up by a single client, the VM data is stored directly under the data mover node. That is, if you run a 'query filespace nodename *' command on the TSM server with the name of the data mover node substituted for nodename, you will see all the VMs backed up to that node listed as file spaces.

A Datacenter node. In a larger environment, several data movers are used to back up a complete virtual environment, such as a VMware datacenter. Although the backup work is distributed among multiple data movers, the VM data is stored in a single shared node and this shared node is called the datacenter node. The reason for this is that if you backup files to a shared node, then you can recover any VM from any datamover node.

A VMCLI node. In this large vSphere virtual environment with several data movers and maybe several datacenters, you need a third node to communicate among the nodes and TSM server and this node is the VMCLI node. It connects the command-line interface to the TSM server and the TSM data mover node. As this node is just used for communication, it does not need a TSM client acceptor or scheduler service.

VMCloud Objects

TSM can backup and restore VMCloud templates and vApps.
A VM template is a master image of a VM. The template can include an installed guest operating system and a set of applications.
A vApp is a logical entity that consists of one or more VMs that make up an application. You probably want to consistently backup the whole application together, and if a restore is required, you may want to restore the whole application, that is, all the VMs in the Vapp, to the same point in time. Managing by vApp makes this easier.

After you run the initial configuration wizard in a vCloud environment you should see

  • A vCloud Director Node which is the top-level node.
  • A Provider VDC Node which represents a provider vDC for vCloud environment.
  • An Organization VDC Node which represents an organization vDC for vCloud environment.
  • A Data Mover Node which represents a backup-archive client and is associated with a single provider vDC and all of its organization vDCs. It is authorized to do operations for all organization vDCs that belong to the provider vDC.
  • A VMCLI Node which connects the Data Protection for VMware vCloud GUI to the Tivoli Storage Manager server and the other nodes.

Backup and Restores

Backups

Before trying a backup from the command line it is always a good idea to run the following command first.

vmcli -f inquire config

This command will list out the current configuration, including the name of the TSM server you are attached to and the names of the various DP for VMware nodes described above. You can check to see if these are the ones you expect to use. You can either chose to backup VMs directly from the command line, or you can store the VM detail in a file. The vSphere backup command using a file is:

vmcli -f backup -t backup_type vm_list_file

where 'backup type' is one of
TSM_FULL or TSM_INCR. These are vSphere options only and create a full image backup or an incremental.
TSM_IFFULL and TSM_IFINCR, which are valid for both vSphere and vCloud. IFFUL will create an incremental forever full backup of the specified backup objects, and IFINCR will create an incremental forever incremental backup of the specified backup object.

The vm_list_file parameter is the name of a file that contains the list of objects to back up. In vSphere mode, the file should contain lines like this, where vm1 and vm2 are the names of your VMs.
vmname:vm1
vmname:vm2
You are restricted on what characters can appear in a VM name. Neither Commas or colons are allowed, and in general, names must contain standard English characters and all the VM names must be unique.

In vCloud mode, you could place the VM details in a file, but the following command can be used

backup vapp Org=organization,OrgvDC=organization_VDC,vApp=vApp -ASNODENAME=provider_vdc_node

'Org' is the name of the organization from which the vApps are backed up and 'orgvdc' is the organization vDC from which the vApps are backed up. The 'vapp' parameter is optional. If you miss it out then all the vApps in this organisation will be backed up. If you want to select several Vapps, or more than one org or orgvDC, then run multiple commands.

When backing up a VM template, and a full backup does not exist for this VM template, the following occurs:

  • If the selected backup type is incremental-forever-incremental, and the VM template contains changes, the backup type changes to incremental-forever-full.
  • If the selected backup type is incremental, and the VM template contains changes, the backup type changes to full.
  • If the selected backup type is full, and the VM template contains changes, the VM template is backed up.
  • If the selected backup type is incremental-forever-full, this type ensures that the VM template is backed up regardless of whether it contains changes.

Restores

It is possible to recover files from snapshots by mounting a snapshot volume. The TSM 6.4 section has details on how this is done.

This command will run an instant restore of a vm called prd047. The temporary datastore is used to store the vm configuration data while the restore is in progress and the name must be unique.

dsmc restore vm prd047 -vmrest=INSTANTRestore -vmtempdatastore=temp_datastore_02

The command to restore a vm called dev25 is shown below. This command will restore from the most recent backup. If you want an earlier backup you can add the -pick option so see a list of available backups and then you can select the one you want.

dsmc restore vm dev25

The vCloud restore command to restore a vApp called vApp_Admin is shown below. You obviously need to substitute your own values for the org name, the orgvdc name and the vapp name.

restore vApp org=org_HR,orgvdc=prod,vapp=vApp_Admin

You cannot restore the Windows system state from a VMware snapshot. The reason is the the system state consists of more than just the files that are backed up by a snapshot, it also consists of the registry.

There are some restrictions on vCloud restores. It is not possible to restore a single VM template as TSM marks the vCloud template object as a single file.
It is possible to restore individual VMs that are contained in a vApp, but if you select a vApp for restore then you restore all the components within that vApp. If you restore vms within a vApp and that vApp still exists, then the restored vms are restored to the vApp. If the vApp does not exist then the VM is restored to the top-level default location on the target ESX host.

Manuals

If you want to know more about TSM 7.1 and VMware then search for the Installation and User guides, called.
IBM Tivoli Storage Manager for Virtual Environments Version 7.1: Data Protection for VMware Installation Guide
IBM Tivoli Storage Manager for Virtual Environments Version 7.1: Data Protection for VMware User's Guide

Monitoring VMware activity

Normally, if you want to find out what backups and restores have been running against a TSM client, you would use a command like this, where the ENTITY filter will pick up the clients that you want -

select activity,,NUMBER,entity,SCHEDULE_NAME from summary where ENTITY is like 'VE%' and start_time>(current_timestamp-1 days)

However, if you run this command against VMware clients you will not get the correct data. You need to use the SUMMARY_EXTENDED table instead as this has been specifically introduced for VMware clients. So the correct command to use is

select activity,,NUMBER,entity,SCHEDULE_NAME from SUMMARY_EXTENDED where ENTITY is like 'VE%' and start_time>(current_timestamp-1 days)

TSM backup issues with VMware

You may see an error message like 'ANS2718W The virtual machine 'machine_name' requires snapshot consolidation' This basically means that the VMware snapshots were not deleted after the previous VM backup. VMware snapshots are managed by VMware, not TSM, so this is a VMware issue. To confirm this, take a look at the hostd.log and the vmware.log for errors like:

Consolidate Disks message: The virtual machine has exceeded the maximum downtime of 12 seconds for disk consolidation.

SnapshotVMXConsolidateHelperProgress: Stunned for 20 secs (max = 12 secs). Aborting consolidate.

As mentioned, this is a VMware issue that is described in this VMware fix note. You should refer your this note to get the full details of the issue, but the basics of the fix are
"This message is reported in ESXi 5.5.x/6.0.x if the virtual machine is powered on and the asynchronous consolidation fails after 10 iterations. An additional iteration is performed if the estimated stun time is over 12 seconds. This occurs when the virtual machine generates data faster than the consolidated rate.
To resolve this issue, turn off the snapshots consolidation enhancement in ESXi 5.5/6.0.x, by setting the snapshot.asyncConsolidate.forceSync to TRUE:"