- VSAM structures
- VSAM commands
- Performance tuning
- JCL Buffers
- LSR Buffers
- System Buffers
- VSAM parameters
- IAM, a VSAM alternative
- VSAM Recovery
- VSAM RLS, DFSMStvs
As the ICF catalog system is so critical to mainframe operations, there are several products which have been developed to simplify catalog management. The following page lists the facilities offered by some of them. The first product, Integrated Catalog Forward Recovery Utility (ICFRU) from IBM, is a simple catalog recovery utility. The other products are Catalog Solution and Catalog Recovery Plus from Rocket Software, Catalog Information Manager from Host Systems and T-REX from Dino Software Corporation. These products are more sophisticated and have an ISPF interface to assist in generating JCL. Each product is summarised below.
The Integrated Catalog Forward Recovery Utility ICFRU is an IBM tool for basic catalog recovery that comes with z/OS 1.7 and above. It is not a full catalog management tool like the rest of the utilities on this page, but is useful to consider as a baseline for comparison purposes. It can handle all types of BCS catalog, including VSAM, non-VSAM, and GDG. ICFRU can recover catalogs allocated on a single LPAR, or catalogs shared between multiple LPARS. It can also recover a master catalog, provided it is not in use as a master catalog.
ICFRU does not have any diagnostics capability that detects catalog errors before they cause problems. For this you need to use the IDCAMS utility. If you are actually running a recovery, then ICFRU does have extensive diagnostic facilities to explain how the recovery is working and if it finds any issues with it.
ICFRU does not have a native backup facility; it relies on an IDCAMS EXPORT backup. If your catalog recovery plan will rely on ICFRU then you must schedule frequent EXPORTS of all your catalogs. If you don't use Export, but backup your catalogs with another utility, then you need to restore the catalog with that utility, then Export the restored catalog before you can run ICFRU.
To recover a catalog with ICFRU you must have a valid IDCAMS EXPORT of the catalog from before it was damaged, and a full set of type 61, 65, and 66 SMF records (see the SMF page for details) from every LPAR that is connected to your faulty catalog.
The recovery process in outline then goes like this
The recovery parameters for CRURRSV and CRURRAP are entered on a PARM statement, and include Catalog name, Start date and time, End date and time and two numeric parameters. The first parameter is the maximum acceptable gap between SMF records, in minutes and is used to indicate potential missing SMF data. This is the time between switching SMF datasets.
The final parameter is used if recovering a catalog that is shared between LPARS and is the maximum clock difference in seconds. This lets ICFRU adjust the timing of records by a short period, to allow for different clock times between systems. A value of 2 seconds is typical. .
You can also do non-disruptive testing of catalog recovery procedures. The process basically involves taking two Exports with a reasonable time gap in between them, ensuring all the SMF data is available between those times, then running all three recovery steps ( CRURRSV, SORT and CRURRAP) with the start end end times corresponding to the times of the two Exports. The output will be an Export dataset, which you can compare with the second Export. They should be identical.
ICFRU does not have any catalog reorganisation capability. You would need to use IDCAMS.
Catscrub is part of the CR+ product below but is also part of IBM's ABARS. The CATSCRUB command is intended to help when doing a disaster recovery test. It will sychronise the data in your catalogs with the actual restored data on disk. You can use the ABARS Manager Submit CATSCRUB panel to help you build a CATSCRUB command.
To do this, Go into ABARS manager, enter 4 in the command line to get the features panel, then enter 1 to get the Submit CATSCRUB panel. There is a good online help that describes the panel's fields.
What CATSCRUB does is compare each record in a catalog against the actual data sets on the disk storage volumes. If the records do not match reality you can delete the catalog record, depending on what options you specify.
The basic syntax is
CATSCRUB BCS(bcsnames) EXEC/SIM FAT-CAT-ERR(end/cont)
You can supply one BCS, a list of BCS names or a pattern mask. The Master Catalog must be named explicitly, it cannot be processed with a mask.
EXECUTE or EXEC means that you want to update the catalogs, SIM or SIMULATE means just test the command and see what will happen. It's obviously a good idea to run Simulate before you run Execute.
FATAL-CATALOG-ERR is optional and defaults to END. If you don't want the command to stop if it hits a fatal catalog error, use CONTinue.
Other parameters exist, such as Message-Text and Match-Volser.
An important set of keywords are the Anomaly Action Keywords, as these keywords are used to decide what to do if Catscrub finds a mismatch between catalog and storage. The options for these keywords include DELETE (the catalog entry) , KEEP (the catalog entry) set a Return code if this anomaly is found and decide whether or not to continue with the command.
The default values for all the Amomaly Action keywords below are (KEEP,RC=4,Continue)
CATSCRUB can treat three different entities in different ways, GDGs, VSAM files and non-VSAM files.
The GDG-MIGRATED-TAPE is uses to determine what happens if an individual GDS entry is migrated to tape. The probability is that you have not taken a copy of your ML2 tapes to your DR site, so rather than get continuous recall failed errors, you may just want to delete catalog entries for migrated GDG files.
The GDG-TAPE keyword applies where the physical files were just created on tape and are not at the DR ste.
NONVSAM-MIGRATED-TAPE can be used in exactly the same way for non-GDG files.
The GDG-NOT-FOUND and the NONVSAM-NOT-FOUND keywords are used to decide what action to take where these files either do not exist at all, probably because the volumes they were stored on were not recovered. You can also say that you want to keep the GDG base, but delete all the GDS entries.
Finally, what about multi-volume datasets, where maybe part of the dataset has been restored but part is missing? The NONVSAM-MULTI-VOL-ERR keyword can be used to decide what to do with those ones.
The VSAM-SPHERE keyword is the equivalent parameter used for VSAM component processing and the NON-VSAM keyword should be self explanatory.
CR+ is developed and supplied by Rocket Software. It is also an IBM recommended product and is resold by them as Tivoli Advanced Catalog Management for z/OS. The difference between CR+ and TACM is that TACM is Tivoli Enterprise Portal enabled.
CR+ has 4 sub commands within its DIAGNOSE command;
DIAGNOSE ALIAS is used to compare two master catalogs and check the connections and alias relationships with their user catalogs
DIAGNOSE BCS-VVDS can be used both ways, to check for 'orphaned' BCS records or for 'orphaned' VVDS records.
DIAGNOSE VOL-BCS will check for uncatalogued datasets
DIAGNOSE VVDS-VTOC will check that VVDS entries are synchronised with the VTOCs
All functions will automatically generate commands to fix any problems that they find.
The backup command has two options; BACKUP BCS and BACKUP VVDS. You have the option to select catalogs for backup individually, by name masks, or every connected catalog with BACKUP BCS(**). A VVDS is selected by VOLSER, not the full VVDS name. BACKUP VVDS(**) will backup the VVDS on every online disk.
A PARALLEL keyword can be used to multi-task the backup and make it run faster.
Each Catalog is opened as a dataset rather than through Catalog Services. This means the processing bypasses the index so you can be sure to read all the data portion. Each BCS backup will copy the aliases definitions in the master catalog, though you can optionally turn that off. You also have the option to check the consistency of the catalogs at backup time by using keywords like DIAGNOSEBCS and DIAGNOSEVVDS.
The RECOVER command will only process one catalog at a time. The selected catalog will be locked by default. As the backup just reads the data portion of the catalog it must rebuild the index on restore, so guaranteeing index integrity with the data. If you kept the ALIAS keyword active on the backup, then the default action on restore is to rebuild all aliases in the master catalog.
The RECOVER BCS FORWARD option will read SMF dump files to recover a catalog to a point in time. It is also possible to forward recover a VVDS file from SMF records. The RECOVER command also has a simulate option for testing.
The RECOVER command also has a newname option that lets you recover a catalog to a different name.
A couple of useful options for DR is EMPTY-GDG, which will just recover GDG bases and the CATSCRUB command which will synchronise a catalog with the data on a specified set of volumes. CATSCRUB is explained in detail above as a separate product.
CR+ can do a catalog reorg or a catalog mergecat while the catalog is open. This process also lets you change catalog attributes on the fly.
A single step MERGECAT function will move records between catalogs for alias balancing.
The CAS interface lets you run CR+ commands against open and in-use catalogs, without compromising the integrity of the catalog or VVDS.
CR+ can be used to resize a VVDS, rename a VOLSER with a SUPERCLIP command and properly recatalog all data sets without unloading and reloading them. You must ensure that the volume is offline on all systems that share access to it.
CR+ can rename a catalog with the RECOVER - NEWNAME option.
CR+ provides a fast way to split and merge catalogs. This can be required if your catalogs are too large and need to be split into smaller entities.
CR+ has a EXPLORE command that allows you to put together search arguments using over 100 dataset attributes, so you can find all datasets that might be about to cause issues. EXPLORE now supports HFS and zFS files and file systems.
CR+ allows you to zap BCS, VVDS and VTOC records, using an onscreen formatting system that allows you to see the fields, cells, and sub-records that you might need to change. Not a recommended way to alter dataset metadata, but sometimes a zap is the only way to fix an issue.
Catalog Information Manager from Host Systems GMBH is an ISPF driven system that generates batch jobs. I find it useful to be able to generate a batch job the first time I need that function using CIM, save the JCL and so build up a library of suitable jobs for running in future.
A quote from a CIM user; 'My last day before I was taking time off to get married and go on a week-long honeymoon, I inadvertently deleted a catalog index for our TSO catalog, no less! With the help of CIM, I was able to restore the most recent backup of the catalog and then forward recover, using SMF records. The whole process took less than 30 minutes, and I was still able to get to the church on time.' Some male cynics might say that was a good reason to stick with IDCAMS.
CIM has five principle diagnostics options from its main ISPF panel
VVDS/VTOC to check the physical, logical and SMS relationships between datasets on disk and in the VVDS
Usercatalog performs various checks on the user catalogs, including consistency with tape management catalogs
Mastercatalogs checks user catalog connections, aliases and VVDS relationships in master catalogs
Sams/Tms/Hsm/Rmm checks various tape management catalogs and the DFHSM MCDS and BCDS.
IDCAMS will generate standard IDCAMS diagnostic commands
All these commands can automatically generate fixes if problems are found.
The Catalog Monitor can continuously check the status of your catalogs and report problems by e-mail or TSO Notify.
CIM uses separate commands to backup catalogs and VVDS files. CAT=** will backup all connected catalogs. VVDS can be selected by SMS Storage Group.
You can optionally request that CIM does an index and data test at backup time.
CIM will backup catalog aliases from the master catalog, and also delete, define or restore them.
CIM will recover from a broken index if the backup used dump without index. A recovery will automatically rebuild aliases in the master catalog. CIM will lock a catalog before recovery starts and it also has commands to deallocate catalogs from VLF if VLF is used.
Forward recovery is automatic with the RECOVER command which reads SMF dump files. It is possible to forward recover several catalogs with a single command, recover a catalog to a new name and update an active catalog with SMF records.
CIM has comprehensive VVDS recovery utilities and can rebuild a VSAM KSDS without using the index.
CIM has the ability to reorganise a catalog while it is open, without affecting other running processes.
CIM functions include
VVR / NVR functions
VTOC / VVDS / Dataset functions
When EMC bought out Softworks, the original Catalog Solution developers left the company and formed DINO-Software. One of DINO-Software's products is T-Rex, a catalog management utility. The philosophy behind T-REX is to be able to run diagnostics then automatically fix errors from a single command. You also have to option to generate control cards that can be checked and run at a later time.
Enhancements in T-REX V7.4
z/OS 2.4 Support
T-REX Product Enhancements
T-REX can check that multiple master catalogs are synchronised, that the VVDS/BCS and BCS/VVDS relationships are correct, and that HSM CDS files match the BCS. It will also verify various tape management catalogs against the ICF system. T-REX can automatically fix problems, or generate control statements to run at a later time.
The alias validation feature can automatically fix missing aliases, or remove unused aliases. T-REX can monitor multi-volume VSAM clusters and alert if any are approaching the 59 volume limit.
The T-REX EXPORT command can backup all BCS and VVDS files with a single command to multiple output files. As T-REX uses the IDCAMS EXPORT command for backup, it does not use the catalog index and so avoids the problem where a backup using the index appears to work fine, but is useless as the index was corrupt.
The DUMP command can also be used for VSAM clusters. T-REX is efficient at multitasking. It does its own performance analysis to determine the optimum number of tasks to run.
T-REX can also generate a point in time backup by combining a previous backup with SMF extracts. The backup can be either a T-REX backup or an IDCAMS EXPORT.
T-REX will always recover from broken index as dump without index is the default. By default it will lock a catalog before recovery and will automatically rebuild the aliases in a master catalog.
The ICFRU command provides automatic forward recovery from SMS dumps. There is also a VVDS recovery feature.
The IMPORT command will restore VSAM files, as well as Catalogs and VVDS files.
The DRIMPORT command is designed for disaster recovery test scenarios, where the full complement of disks have not been provisioned. This command can be used to just restore catalog entries for disks that exist, and so avoid issues with catalog entries that point to missing volumes. The SCRUB command is also intended to help here. It will synchronise BCS entries with online volumes, deleting entries that point to disks that do not exist.
T-REX can reorganise a catalog while it is open, and can remove imbed/replicate parameters while the catalog is open. You can also use T-REX to move a catalog that is open and allocated to CAS. This would be a very useful function where a catalog is on a small disk that is running out of space.
T-REX includes the following support for DFHSM: The AUDIT command has been enhanced to provide two levels of validation for Migration Control Data Set (MCDS) entries, complete structural verification and RECALL verification. RECALL functionality allows you to determine if a migrated entry can be successfully recalled. Relational inconsistencies do not always prohibit the recall of user's data.
The LISTHSM command will select BCDS, MCDS and OCDS records based of descriptive keywords and record relationships. With one control statement, you can select any record type and all of its related records, determine used entries that exist on an OCDS TTOC volume, and select entries that are related across different control data sets.
T-REX will maintain VVDS integrity if the VVDS is held in the coupling facility.
It can recovery GDG base catalog entries only, leaving GDG data files to be recovered and catalogued by a file restore utility like FDRABR or DFDSS. T-REX supports extended GDGs.
T-REX supports all 4 types of VSAM, as well as BCS and VVDS.
Re-cataloging of multi-volume VSAM is fully supported, and T-REX can also detect hardware changes and dynamically change performance parameters to suit.
T-REX will relabel DASD volumes containing catalogs, checking VSAM and nonVSAM data sets as part of the process to ensure consistency by updating any catalog entries pointing to data sets on the volume. To speed the process up, multiple catalogs are processed concurrently as separate subtasks.
T-REX can resize a VVDS
T-REX will identify any encrypted datasets
ZFS clusters are supported
T-REX will recatalog multi-volume datasets
T-REX can work with most Tape Management systems, including CA1, TLMS, Control-T, RMM and Zara. It will identify any discrepancies between the TMC and catalogued tape datasets.
When you run an IDCAMS DIAGNOSE ICFCATALOG, you need to compare the catalog with all online volumes to seek out discrepancies. Each volume is identified with a COMPAREDD statement. T-REX has the ability to call and provide unlimited COMPAREDD support and as been tested with over 4000 volumes.
T-REX can rebuild and synchronize your BCS with your TMC entries. This is accomplished through the AUDIT command. This is particularly useful in a DR setting to get the restored BCS completely in synch with the DR backup tapes shipped to the DR site.