Managing the IBM Spectrum Protect Server

Administration

Installation

Scheduling

Administration

Using the IBM SPcommand line

UNIX Server Command Line

To start a IBM SP server command line in AIX, you use the dsmadmc command with an optional servername. The command line is useful if you want to do bulk upgrades, as you can then do them with a script. For example, if you want to register 10 new nodes, rather than go through the tedium of adding 10 nodes through a panel, it is much easier to set this up as 10 commands within a UNIX script, then just execute the script. A typical script could look like

dsmadmc -se=servername -id=youruserid -pa=yourpassword register node nodename nodepassword passe=9999 con=\"This server contact details\" do=policy-set clo=cloptset userid=none comp=no archdel=no backdel=no etc.

you obviously need to substitute your own values for the parameters, and note how special characters like quotes are escaped out with a forward slash.

Other dsmadmc options

The command can also be very useful if you want to start up a session to a TSM server in different modes. for example, if you want to run some queries that you want to feed into an excel spreadsheet or maybe programs, and you do not want the screen titles displayed you would use the following. -comma means comma delimited output, the dataonly switch suppresses the headers.

dsmadmc -se=servername -dataonly=yes -comma

an alternate option is -tab for tab delimited. These commands can be usefully argumented with the -out parameter, which writes the console output into an external file. For example if you start a server session as below, then the output of any commands you issue will be placed in sqlout.txt

dsmadmc -se=servername -dataonly=yes -comma -out="c:\program files\tivoli\tsm\sqlout.txt"

Three other useful options are

dsmadmc -con
dsmadmc -mount
dsmadmc -noconfirm

The first one starts a server session in console mode, so all the console messages automatically scroll past your terminal. This is useful if you are waiting for a process to complete, as it saves you from keep typing q actlog or q pr commands.
The second option is similar, but instead of displaying all the console messages, it just displays tape mount messages. This one was intended for a media librarian that had to mount tapes manually
The Third option suppresses the 'do you really want to do this' type messages, for example if you run a query that will produce a lot of output, you will see a message like, 'Do you wish to proceed? (Yes (Y)/No (N))'. -noconfirm will suppress these messages, but obviously you need to use it with care as the messages can be very useful. For example, if you start to delete objects, the confirmation message gives you a second chance to check that you are not deleting items by accident.

Logging into a remote IBM SP server

With IBM SP servers running on Windows, you can log into any remote IBM SP server, provided you have the dsmadmc command code loaded with your IBM SP client, and provided your firewall rules permit remote access. If you have a remote server on a Windows box with IP address 99.245.24.123 and with a DNS name of WSTSM01, you can start a session by navigating to \program files\tivoli\tsm\baclient and running either

dsmadmc -tcps=99.245.24.123
or
dsmadmc -tcps=WSTSM01

If you get a message like "dsmadmc not found" then you've got a basic client install and you need to install the server management component.
This works on the IP address of the Windows box that is hosting IBM SP. What if you are running more than one IBM SP server on that box? By default you will get IBM SP Server1. To reach the other servers you need to add a -tcpp switch that specifies the port of the other server

dsmadmc -tcps=WSTSM01 -tcpp=1502

back to top


Canceling sessions and processes

To cancel server processes like migration you need to know the process number, which you get with the Q PROCESS command. However note that if you cancel a migration session then a new one will start unless you also reset the pool thresholds. You generally need system or opertor priviledge to run cancel commands.

CANCEL PROCESS process-number

To cancel tape mount requests you first need to know the request number, which you find with the QUERY REQUEST command, then run CANCEL REQUEST with the corret process. You can cancel all tape mounts with the ALL parameter.

QUERY REQUEST
CANCEL REQUEST request-number
CANCEL REQUEST ALL

To cancel active sessions you either need to know the session numbers, which you get with the Q SESSION command, or you can cancel all active sessions

CANCEL SESSION request-number

To prevent new sessions or processes from starting use

DISABLE SESSION client
DISABLE SESSION Server
DISABLE SESSION Admin

These commands just prevent new sessions from starting. All active session will run to completion unless you cancel them with the cancel command. The disable session server command will stop new server to server sessions from starting, it will not stop expiration or migration.

To do an emergency cancel of all sessions use the command

DISABLE SESSION ALL
CANCEL SESSION ALL

If you want to cancel a node replication, then the command below will cancel all node replication processes.

CANCEL REPL

back to top


How to determine when IBM SP will expire a backup

The key to understanding TSM backup retention is to understand the difference between 'active' and 'inactive' backups. The 'active' backup is the most recent backup while all the older backups are 'inactive backups'. However, once a file is deleted from the client, it becomes inactive after the next incremental backup. The active backup of a file is never deleted, as it is needed to recreate a disk
IBM SP uses 4 parameters to retain backups.

The version parameters and the retain extra parameter can take a value of 1-9999 or NOLIMIT, while the retain only parameter can take these values and also take a value of 0.
There is a fundamental difference between the versions parameters and the retain parameters. The versions parameters are controlled by the backup function, so changes to versions will not take effect until the next backup runs. The retain parameters are controlled by expiration, so changes to retention parameters take effect immediately.
The pecking order goes like
If the file is deleted, then the most recent backup is kept for the number of days specified in 'retain only version', while older backups are retained by whichever of 'retain extra versions' and 'versions data deleted' is met first
If the file is not deleted, then the most recent or 'active' backup is kept forever, while older backups are retained by whichever of 'retain extra versions' and 'versions data deleted' is met first

For example, you have RETEXTRA=31 and VEREXISTS=31. If you create 31 versions in the same day, and then (still on the same day) you create version 32, version 1 will expire, regardless of RETEXTRA, because the VEREXISTS criterion has been exceeded. Likewise, if you create version 1 today, then create version 2 a week later, then never create another version after that, then version 1 will expire 31 days after the creation of version 2, since the RETEXTRA criterion has been exceeded.

If you need to GUARANTEE data retention for 31 days you would need to code the parameters as below, but be aware that you could end up keeping a lot of backups.
Versions Data Exists = NOLIMIT
Versions Data Deleted = NOLIMIT
Retain Extra Versions = 31
Retain Only Version = 31

If you wish to keep sets of data for an extended time period, then use Retention Sets.

back to top


How to list out backups that are marked for expiration

How can you find out which backups have been marked for deletion?
Backups and Archives are marked for deletion when their expiry date or version limit is reached, or if they are deleted manually, but they are not purged from the IBM SP database until the next Expire Inventory is run.

IBM SP flags files that are eligible by giving them a special deactivation date, which is 1900-01-01. Then when an Expire Inventory process runs, any objects which have deactivate_date of 1900-01-01 are removed from the database.
It is possible to generate a list of these files with SQL queries, but be aware that the file listing might be very large and generating it might degrade your server performance. A suitable SQL queries is:

select node_name,state,deactivate_date,backup_date,hl_name,ll_name from backups where deactivate_date='1956-09-05-00.00.00.000000' and type='FILE'

The query output will still return a deactivation_date of '1900-01-01 00:00:00.000000' even though you specified '1956-09-05-00.00.00.000000' on the SQL query. That's just the way IBM SP stores the date.

back to top


MAXSESSIONS and MAXSCHEDSESSIONS

TSM limits the maximum number of sessions it will run at any time to try to prevent performance issues by runnng too many tasks at once. This is controlled by the MAXSESSIONS server option, but the default value is quite low at 25 client sessions. If you have it set too low you will see errors like

ANR0429W Session xxx refused maximum server sessions exceeded.

You can update this server option without stopping and restarting the server by using a SETOPT command like this

MAXSessions 100

The value you would use depends on how many sessions you are trying to run and how much resources you have available. There is no definite rule here, it very much depends on your environmeent. IBM states that 'the maximum value is limited only by available virtual storage size or communication resources'. So how do you know if you have increased it too far and you are getting performance issues? Check that your database cache hit is at least 98% by running the q db f=d command.

There are two types of session, scheduled and unscheduled. Unscheduled includes things like restores and retrieves. You can limit the number of sessions available to scheduled operations, to ensure that unscheduled operations can run, using the set maxschedsessions command. The command to do this is set maxschedsessions followed by a percentage. It may be a better option to increase maxschedsessions rather than maxsessions if maxschedsessions is set too low.

You may often see the number of active client sessions exceeding the maxschedsessions value. A scheduled session is a type 5 session, a generated session is a type 4 session. When a node starts a scheduled backup to the storage manager server, only the initial session is opened as a type 5 session. All other sessions opened as part of the node's scheduled operation are of type 4. Only type 5 sessions count when calculating the number of scheduled sessions.
To find out what MAXSCHEDSESSIONS is set to, run the 'SHOW CSVARS' command. Output looks like

ANS8000I Server command: 'show csvars'
Max %% Scheduled Sessions : 50
Max Scheduled Sessions : 100

The command output shows that there are 100 type 5 sessions available for schedule backups. The 'SHOW SESSIONS' command will then tell you what type of sessions are active. Output typically looks like

ANS8000I Server command: 'show sessions'
Session 73568: Type=Node, Id=TSM_PRW51120_ORA
Platform=AIX, NodeId=537, Owner=
SessType=5, Index=0, TermReason=0

Session 73867: Type=Node, Id=TSM_PRW51120_ORA
Platform=AIX, NodeId=537, Owner=
SessType=4, Index=6, TermReason=0

Now count all the 'SessType=5' sessions in the SHOW SESSIONS output and see how close you are to the maxschedsession limit. If you are still getting ANR0429W errors and you cannot increase maxsessions further, consider increasing the maxschedsession limit, and if you cannot do that your only other option is to spread your backup schedules out to reduce the load at any one time.

back to top


Running processes in parallel

It is mentioned in the Administrator's Guide manual that running multiple processes simultaneously might lead to process termination and it is recommended to avoid the situation if possible. If you run data movement or delete operations, like pool migration, simultaneously with data access operations like storage pool backup, they can contend and one process may terminate early but report completion without errors.
You need to run such processes serially to avoid this happening. Examples to avoid include running 'storage pool backup' alongside 'reclaim storage pool' and running 'copy activedata' alongside 'reclaim stgpool'

back to top


Which IP ports does IBM SP need open on a firewall?

This is a list of listener ports that various IBM SP products expect other hosts to connect to, and may be useful in firewall setup. This applies to IBM SP version 8.1 and higher

Each server instance requires a unique base port, so if you run several TSM instances in your organisation, each will use a different number. The default port is 1500 and is bi-directional.
You can define an SSL-only port (SSLTCPPORT) if you want to restrict communication on the port to SSL-enabled sessions only. A server can support both SSL and non-SSL communication by using the TCPPORT or TCPADMINPORT options. There is no default port number for this, and it is bi-directional.
configuration wizards use ports to communicate by using native protocols with multiple hosts. The SMB port is 45 and the SSH port is 22 and both are bi-directional. SMTP port 25 is used to send email alerts from the server and this is Outbound only.
Replication ports are set by the DEFINE SERVER command and have no default values. They are bi-directional.
You may also need ports opened for LAN free backups, and for VMware backups.

From the Client side, a client uses a schedule port with a default value of 1501 which is Outbound only. The client passes that port number to the server, then the server contacts the client if server prompted scheduling is used. You can specify the port number in the client options file.
The Operations Center web browser default port is 11090, and is Inbound. You can specify an alternative port number.
If you use Client Management Services, then this service port must be accessible from the Operations Center. The port number is 9028 and it is Inbound.

back to top


Server to Server Communications

If you have more than one IBM SP server, then you will probably want them to be able to communicate with eachother. There are several reasons why you would do this, some of which are for enterprise configuration and event logging, virtual volumes, LAN-free data movement, node replication and server-to-server export.
Cross server communication uses SSL, since version 8.1.2 and SSL should be configured automatically. If you use LAN free storage agents, then they too should be configured automatically, from version 8.1.3 onwards. If you chose to configure them manually, then run the following command on each server, to import the other server's self signed certificate file.

gsk8capicmd_64 -cert -add -label Other_servers_IP_address -db cert.kdb -stashed
gsk8capicmd_64 -cert -list -db cert.kdb -stashed

The second commad can be used to check that the first one worked correctly. Once this is done, restart both the servers
Now run the DEFINE SERVER command, assuming the servers are called ServerS (source) and ServerT (target)

On ServerS
Define server ServerT servert_password HLAaddress=I.P Addres_of_ServerT LLAaddress=Port_Number_of_ServerT commmethod=TCPIP
On ServerT
Define server ServerS servers_password HLAaddress=I.P Address_of_ServerS LLAaddress=Port_Number_of_ServerS commmethod=TCPIP
Set crossdefine yes

You might sometimes get an error with cross server communications, with an error message

ANR0454E Session rejected by server ServerT, reason: 200 - Verification Failure.

If you get this, then the first thing to check is that the high-level and low-level address for the defined server is correct. If either or both are wrong then update them. Check this from both sides, on both ServerS and ServerT
If they are correct, then another possibility is that the verification tokens between the servers are out of synch. Try running this command on the source server

update server ServerT FORCESYNC=YES

and on the target server:

update server ServerS FORCESYNC=YES

back to top


Node Replication

The TSM team introduced a new function in version 6.3, the ability to replicate backups of IBM SP client nodes between IBM SP servers. This means that you can have two active copies of your backups in two different locations, giving you an instant ability to recover servers in a disaster. You can also recover data from the replicated site to the live site.

The easiest way to manage this is to combine similar nodes into node groups, so you work with groups of nodes rather than hundreds of individual nodes.
Assuming you have 2 servers called PROD_SERVER and DR_SERVER with IP addresses 200.100.100.0 and 200.100.100.50 then you would set replication us like this:

Define the production IBM SP server to DR IBM SP server using the process in the section above

define server PROD_SERVER SERVERPA=password hla=200.100.100. lla=1500
set replserver PROD_SERVER

.. and the DR IBM SP server to the Production IBM SP Server

define server DR_SERVER SERVERPA=password hla=200.100.100.50 lla=1500
set replserver DR_SERVER

Create a IBM SP node group on both IBM SP servers and add some nodes to it

define nodegroup repl_group desc="Node Group for Replication"
define nodegroupmember repl_group node1
define nodegroupmember repl_group node2
define nodegroupmember repl_group node2

On the production IBM SP server, enable replication for each node:

update node node1 replstate=enabled
update node node2 replstate=enabled
update node node3 replstate=enabled

On the production IBM SP server, add an admin schedule to run the replication:

define schedule replicate_nodes type=admin cmd="replicate node repl_group wait=yes" desc="3 Hourly Node Replication Schedule" dayofweek=any period=3 perunits=hours active=yes starttime=00:00

Now all 3 nodes will replicate their backup data to the DR server every three hours. You can check the progress of replication with the Query Process command.

Note that if the nodes are deduplicated then the 'Amount Transferred' represents the deduplicated bytes sent. Also, if you keep issuing the Query Process command you will notice that the replication statistics keep changing. This is normal. When older releases of TSM started to run processes like replication they would sit for a long time while they worked out what to do. This could be a bit worrying as it looked like the process had hung. Now, TSM will batch the work up and start processing as soon as it determines the first batch. Filespaces can be in 4 states as shown in the output above.

Identifying: means the server is still looking at the objects in the file space and have not yet dispatched any batches for replication.
Identifying and Replicating: means the server is still looking at the objects (Identifying) but is also moving objects (Replicating).
Replicating: means the server is done looking at the objects (in the file space) and is just replicating.
Complete: means the server finished the file space.

If you try to use the QUERY OCCUPANCY command to check that your replication is working, by looking to see if your target node has the same amount of data as your source node, then you will get confusing results. This is because the QUERY OCCUPANCY command includes data residing in copy storage pools and active-date storage pools and this data is not replicated. You would need to use a SELECT command that just looked at the file count and data in the primary pools, something like this;

select sum(a.NUM_FILES) as "# of files", sum(a.LOGICAL_MB) as "Logical(MB)" from occupancy a, stgpools b where a.stgpool_name=b.stgpool_name and b.pooltype='PRIMARY'

By default the REPLICATE NODE command uses the "DATATYPE=ALL" parameter. If you use a different value for this parameter, then you will need to adjust your SELECT command by using the 'type' parameter in your 'where' clause for the occupancy column. If you need to recover node data back to the production server, you would need to have the REPLRECOVERDAMAGED server parameter set to 'ON', then you can use the RECOVERDAMAGED command.

back to top


Installation

Synchronising your IBM SP server with the OS

If your IBM SP server is out of step with your hosting server, possibly due to a daytime savings change, you can easily change TSM to match the OS server by entering the following command at the admin command line. It is possible that your server will not start normal processing if the date do not match.

ACCEPT DATE

back to top


Setting up an Enterprise Server

Controlling and managing several TSM servers with hundreds of clients can be a headache, if only when working out which client is managed by which server. TSM offers the Enterprise Configuration facility to make this easier. This lets you create and distribute settings and configurations from one configuration TSM server to several managed TSM servers.

You start with the TSM server that is going to be the configuration server.

From the Object view, select Server - Server Groups, then 'Define New Server Group' from the drop down menu. Give this group a meaningful name and description, add it, then select 'Server Group Members' and add the TSM servers that you want to be associated with this group.

On that server GUI, open Operational View; Configuration Manager Operations option and select 'Establish this server as a configuration manager', set the configuration manager drop down window to 'ON' and press the Finish button (or use console command SET CONFIGMANAGER ON)

Next, you need to select the 'managed server operations' menu The configuration manager works on profiles, which are used to collect the objects that are to be managed together, so they can be managed with a single command.

You select the 'define a new Configuration Profile' option from the drop down menu, then you will be presented with a list of objects that can be managed by this profile. The available objects are:

You can then define which of these objects will be managed centrally with the DEFINE PROFASSOCIATION command. However it is not as simple as this, as, for example, if you decide to centrally manage all your command schedules, then they need to be identical on every server in the group. Installing an Enterprise Configuration is a good opportunity to clean up and standardise your TSM estate.
The DEFINE PROFASSOCIATION command has a number of parameters corresponding to the available objects above. If you define an object then change your mind you can remove it with the DELETE PROFASSOCIATION command.

Once you get the configuration manager setup, you then need to log onto each of your managed servers, and subscribe each one to the profile that you just created. Suppose you called it WINDOWS-PROD and your Configuration Manager server was called CONF-MAN, then you would issued the command

define subscription WINDOWS-PROD server=CONF-MAN

What does this all mean?

Suppose that you defined a server group called WINDOWS that contains all your Windows TSM servers. You can now send commands to all your Windows TSM servers by prefixing your command with WINDOWS:

WINDOWS: Q DB
WINDOWS: upd stg cartpool reclaim=40

Suppose you defined your WINDOWS-PROD profile to control administrators, client schedules and admin schedules. They are all centrally managed by the CONF-MAN server now and you can only make changes to your managed servers by applying the updates to CONF-MAN then distributing the changes out to the managed servers by issuing the command

NOTIFY SUBSCRIBERS PROFILE=*

From the CONF-MAN server. This both simplifies management and maintains consistency across all TSM servers.

back to top


How to Configure a TSM Server on a Windows 2015 Cluster

Define a Cluster Resource Group

Install the IBM SP servers on every node

Complete the following steps for every node in your cluster:

Initialize the Server on the Primary Node

Once you have installed an IBM SP server on every node in the cluster, you have to initialise the IBM SP server on the primary node.

Procedure

Set up the Services on the Secondary Nodes

To set up the services on the secondary nodes, run the following command on each secondary node:

C:\windows\system\sc.exe create "IBM Spectrum Protect instance_name" binPath= "server_installation_directory\dsmsvc.exe -k instance_name" start = demand obj=instance_username password = instance_user's password
Copy

For example, issue the following command:

C:\Windows\system32\sc.exe create "IBM SP SERVER1" binPath="C:\TSM\Server\dsmsvc.exe -k SERVER1" start=demand obj=tsm13\Administrator password="PassW0rd"
Copy

Check the configuration

To verify that the IBM SP server instance in a Microsoft Failover Cluster is created and configured correctly, complete the following steps:

Test Failover

After you complete the cluster configuration, run a failover test to ensure that the nodes are working properly. Procedure

back to top


Scheduling

Defining an IBM SP schedule is a 2 step process. First you define a schedule to run at a particular time, then you add, or associate the nodes that you want to run at that time, with the schedule. The commands to look up are:
DEFINE SCHEDULE
DEFINE ASSOCIATION

Scheduling - controlling the start time

Ever been in the situation where you have a tight backup window (and if you haven't, how do you get a job like that?) You schedule 8 backups to start at midnight, but some of them don't start until 01:00? Frustrating or what?

The problem is that by default, TSM tries to spread backups out in a schedule, unless you tell it not to. Issue the command QUERY STATUS on your TSM server, and look for the parameter Schedule Randomisation Percentage. By default, this will be set to 25, which means IBM SP will spread the start times of all the backups in a schedule, over the first 25% of the schedule window.

If you want all your backups to fire in right at the start of the schedule, then change this parameter to 0. However be aware that starting a lot of backups simultaneously could cause performance problems.
If you have one especial node that you want to start on time, but are not worried about the others, define a special schedule for that node with a higher priority than the others.

If you get a problem with the schedule, the error will be either 'failing' or being 'missed'. 'Failed' means that the scheduled event began, but encountered an error condition that prevented it from completing successfully. 'Missed' means that the scheduled event never began. For example, if the scheduler is not running on the client, then of course it can not execute its scheduled events; nor can it report that it did not run.

On the client machines that missed their schedule, verify that the scheduler service is started. Check the dsmsched.log and dsmerror.log to verify that the scheduler was able to successfully contact the server and query its next scheduled event.

Scheduling - choosing specific weekdays

If you define your client schedules using the GUI interface, you can chose to run a backup every day, weekdays only, or on a specific day of the week. You can only chose one of these options. If you use a command line, you can be more selective. Say you want to define a schedule to run every Monday, Wednesday and Friday. Use a command like the one below. Note the SCHEDS=E parameter, which means use enhanced schedule style. You need enhanced style to pick out individual weekdays.

DEF SCH domain_name schedule_name T=C ACT=I STARTT=22:00:00 DUR=1 SCHEDS=E DAY=Monday,Wednesday,Friday

Another possible option is to schedule a backup to run on the last Sunday of the month. An appropriate command is

DEF SCH domain_name schedule_name T=C ACT=I STARTT=22:00:00 DUR=1 SCHEDS=E DAY=Sunday WEEK=LAST

back to top


Scheduling a one-off backup

If you want to run an immediate, one-off scheduled backup of a client, the easiest way to do this is to use the DEFINE CLIENTACTION command. The basic syntax is

define clientaction 'node_name' domain='domain_name' action=incremental

where you substitute your own node and domain names. IBM SP will then define a one-off schedule and associate the client node with that schedule. IBM SP will generate the schedule name, and return that name to the server console with messages ANR2500I and ANR2510I. You can use most of the parameters that are available to the define schedule command, like the 'OPTIONS' and 'OBJECTS' parameters, if you need to expand on this command.

The term 'immediate' should not be taked too literally, as what happens next depends on how your client scheduling mode is defined. If the schedule mode is PROMPTED, then the schedule is added to the list of schedules waiting to execute, and will normally run within 5 minutes or so.

If the client scheduling mode is POLLING, then the schedule will run next time the client scheduler polls the server to see if a schedule is waiting and that could take some time. The best action, is to recycle the DSMCAD or the scheduler daemon/service at the client end once you run the define clientaction command at the server, and that should immediately trigger the schedule.

back to top


Incremental backup of multiple directories using the objects field

To perform an incremental backup for several directories, code them in the objects field in the schedule like this.

OBJECTS='/path1/ /path2/ /etc/'

The terminating slash at the end of the directory name backs up the files within the directories, and they need to be separated by spaces. If you wish to back up subdirectories within these directories, then you also need

OPTIONS=-SUBDIR=YES

back to top


Which nodes are associated with each schedule

The command syntax is 'query association domain-name schedule-name'. You can query all associations by using

q assoc * *

If you want to query a specific schedule, but do not know the domain, use

q assoc * sched-name

back to top


Spectrum Protect pages

Lascon latest major updates

Welcome to Lascon Storage. This site provides hints and tips on how to manage your data, strategic advice and news items.