Day 38

Different types of data movement in TSM Infrastructure

Types of data Movement in TSM Infrastructure

There are 2 main data movement types which are generally used for taking backup and archive from clients to TSM server.

Backup through Local Area Networks (LAN) & Wide Area Networks (WAN)

In a traditional LAN and WAN environment, the Tivoli Storage Manager  backup-archive client reads data from locally attached disks and sends it over  the LAN to the Tivoli Storage Manager server. The server receives the data, then writes it out to its storage pool — tape, disk, or optical media — based on predefined policies and server configuration. Data is read and written by both the Tivoli Storage Manager client and server machines. In  addition, control information is also sent over the LAN to the Tivoli Storage Manager server.

Restore operations also follow the same path in the opposite direction. Data is read by the Tivoli Storage Manager server, sent via the LAN to the client system, and  is written there on local attached storage devices.

Backup through Storage Area Networks (SAN)

SAN technology provides an alternative path for data movement between the Tivoli Storage Manager client and the server. Shared storage resources (disk, tape) are accessible to both the client and the server through the SAN. Data movement is off-loaded from the LAN and from the server processor and allows for greater scalability. LAN-free backups decrease the load on the LAN by introducing a Storage Agent. The Storage Agent can be thought of as a small Tivoli Storage Manager server (without a database or recovery log) that is installed and run on the Tivoli Storage Manager client. The Storage Agenthandles the communication with the Tivoli Storage Manager server over the LAN but sends the data directly to SAN attached tape devices, relieving the Tivoli Storage Manager server from the actual I/O transfer.

From a technical point of view, a SAN topology also offers higher bandwidth and  therefore faster speed availability for data movement. In reality this depends heavily on attached storage device technology and the nature of the transferred data. Usually, large files such as databases are transferred faster via SAN than LAN. When transferring small files, SAN performance may radically decrease.Moreover, LAN-free backups do add some complexity to a Tivoli Storage Manager implementation. You need to be careful of how the SAN-attached backup storage devices are utilized — insufficient resource planning can create 

a storage device overload. 

Tape library sharing between multiple Tivoli Storage Manager servers also requires proper planning. Instead of only one Tivoli Storage Manager server and its LAN-free clients using the attached library, multiple servers and their clients are sharing the storage device. Storage area networks are beneficial when moving large files, such as database backups, and application data, such as Exchange, Domino, SQL, or Oracle. Local area networks are appropriate for file server backups of small files.

Backup and Restore Overview

A backup creates an additional copy of a data object to use for recovery. A data object might be a file, directory, or a user-defined data object, such as a database table. The backup version of this data object is stored separately in the Tivoli Storage Manager server storage repository. When you restore an object, you copy it back from the server to the client.

You can restore a file, a directory, or even a whole machine. You need to know two things:

• Items to restore: file name, directory

• The time in the past of the saved items, optional

By default, TSM always takes Progressive incremental backup if the backup type is not specified. This method is the standard backup method that the Tivoli Storage Manager backup- archive client uses. A full backup of new objects is followed by incremental backups. No additional full backups of are required after the first backup. There is no need to retransmit backup data that has not changed during successive backup operations. changed data unchanged data

Backup types available in TSM

Below are the commonly used backups using TSM. By using TDP, you can also archivelog backups of the database applications. By default, Tivoli Storage Manager uses the intelligent Progressive Incremental backup strategy. Only files that are changed or that are new are backed up, reducing the amount of data transfer on your network and system. With progressive backup, you do not need resource-intensive, full backups. Progressive backup provides faster restores because it needs to restore only the version of the file requested. Progressive backup backs up less data, thus saving network bandwidth, tapes, and management overhead. With UNIX symbolic links, only the path information that the symbolic link contains is backed up.

• Incremental (complete)

• Incremental by date

• Incremental (without journal)

• Journal-based

• Always backups, selective or full

• Logical volume backup

• Image snapshot backup

• Network Data Management Protocol (NDMP)

Incremental (complete) backup

The incremental backup function, also known as incremental (complete), backs up all the files that are changed since the last back up. Incremental backups also back up all of the files that are created since the last backup. The incremental backup function does not back up files that are excluded by the include-exclude list.

Incremental by date backup

A disk or volume is eligible for an incremental by date backup, if you have done at least one full incremental backup of that entire disk or volume. Running an incremental backup of only a directory branch or individual file does not make the disk or volume eligible for incremental by date backups.

To perform an incremental by date backup by using the GUI, select the Incremental (date only) option from the type of backup list menu, or use the incrbydate option with the incremental command. The following conditions apply to backup criteria:

• The client backs up only those files with modification dates and times that are more recent than the date and time of the most recent incremental backup of the file system.

• Files that the client adds after the most recent incremental backup are not backed up if their modification date and times are earlier than that backup date and times.

• Files that are renamed after the most recent incremental backup, but otherwise remain unchanged, are not backed up.

Incremental backup by date does not check for frequency, new files, deleted files, changes in file attributes, or rebinding of files.

Journal-based backup

Journal-based backups for Windows, UNIX, and Linux improve performance of incremental and selective backups by using the journal engine service to note the changed files, rather than using file scanning. When the journal engine service is installed and configured, journaling is used by default with incremental backups.

Incremental (without journal) backup

To run an incremental backup without using the journal database, select Incremental (without journal). The journal database is discussed later in the lesson.

Selective (always backup)

The selective backup always attempts to back up the objects that you select. Use a selective backup to back up specific files or directories, regardless of whether a current copy of those files exists on the server.

Image Snapshot Backups

Image backups are full volume backups for Windows clients, which use the snapshot function of Tivoli Storage Manager.

Logical volume backups

Logical volume backups improve the restore performance. Periodically, the logical volume is backed up, before the scheduled daily incremental backups. If a logical volume needs to be restored, the logical volume backup is restored first, followed by the subsequent incrementals.

Network attached storage (NAS)

Network attached storage (NAS) file system backups are enabled through the support of Network Data Management Protocol (NDMP). Tivoli Storage Manager Windows, AIX, and Solaris servers back up and restore network attached storage (NAS) file system images to automated tape drives or libraries. These libraries are attached to Network Appliance and EMC Celerra NAS file servers. NDMP support is available only on IBM Tivoli Storage Manager Extended Edition.

Journal-based backups

Journal-based backup provides an alternative to traditional progressive  incremental backup, which under certain circumstances may dramatically  increase overall backup performance. As the name already implies, journal-based backups have no effect on archive  processing. The main difference between journal-based backup and progressive  incremental backup is the method in which the list of backup candidate objects is derived.

A progressive incremental backup creates the list of files by scanning the entire file system on the client and comparing that list with a list of files on the server. For a journal-based backup, the Tivoli Storage Manager backup-archive client obtains the list of backups by examining the entries that the Journal Based Backup Daemon creates.

Journal-based backup is supported on UNIX, Linux, and all Windows clients except for clients that run on Windows Server 2003 for Itanium-based systems. If you install the journal engine service and it is running, by default, the incremental command automatically does a journal-based backup on selected file systems that the journal engine service monitors. Tivoli Storage Manager does not use the journaling facility that is inherent in Windows NTFS file systems or any other journaled file system. To perform a journal-based backup successfully, the following conditions must be met

• You must set up the journal service to monitor the file system that contains the files and directories that are backed up.

• Must run a full incremental backup successfully at least once on the file system that is backed up.

• The file space image of the file system at the server must not be modified by an administrative command since the last full incremental backup.

• The storage management policy for the files that are backed up must not be updated since the last full incremental backup.

Backup sets

A backup set is a group of active versions of files, copied onto portable media. Backup sets can be restored on the client locally, without the Tivoli Storage Manager server being available.

Although the creation of backup sets is not a client function, it is an important feature of backups. Using the most recent backup versions that the server stores, a Tivoli Storage Manager administrator can generate a complete set of client files (backup set). These backup sets are used to retain a copy of client files for a longer period of time (Instant Archive) or for recovery of a client system, without server or network, when the backup set is copied onto portable media and restored locally (Rapid Recovery).

• The generate backupset command runs as a background process on the server.

• The restore backupset command restores a backup set from the server, a local file, or a local tape device.

Day 37

Securing TSM Client data with Encryption and SSL technologies

TSM Client Data Encryption

To ensure the security of the backed up data, the Tivoli Storage Manager client implements an encryption function. With this function, you can encrypt the data before sending it to the Tivoli Storage Manager server.

The user can chooses the files that are subject to encryption with include-exclude processing. All files that match the pattern on the include.encrypt specification undergo encryption. The encryption processing is the final task on the client system before the data goes to the server. Other client operations, such as compression, occur before encryption.

Encryption uses a simple key management system, which means that the user either must remember the encryption key password during restore operation or store it locally on the client system. Encryption works for both backup and archive operations.

Encryption is a CPU-intensive process. The encryption process requires additional computing requirements on the client CPU. Carefully decide which data items need encryption, and control them by using the include-exclude statements. The administrator has the option to overwrite client selections by using client options sets.

Problems can occur with the simple key management scheme, especially in the case of long-term archiving of data. If data is archived with encryption, organizational rules must ensure that the encryption key password remains available for retrieval. Cyclic password changes or lack of external password management can cause situations in which data cannot be retrieved successfully.

When restoring data by using the scheduling function, files might not be restored because the needed encryption key password is not stored locally (PROMPT mode). Or the restored files might be encrypted with a different key than the stored encryption key password.

After the data is stored on the Tivoli Storage Manager server, it is unreadable. The include.encrypt option is the only way to enable encryption on the client.

include.encrypt /database/.../*.*

Place the encryptkey option in the client options file, either the dsm.opt file or the dsm.sys file. Use one of the following options to manage the key that file encryption uses

     encryptkey save

     encryptkey prompt

     encryptkey generate

With the SAVE option, the encryption key password is saved in the Tivoli Storage Manager client’s password file. A prompt is issued for the initial encryption key password. After the initial prompt, the encryption key password that is saved in the password file is used for backups and archives that match the include.encrypt specification.

With the PROMPT option, you manage the encryption key password. You are prompted for the encryption key password when the Tivoli Storage Manager client begins a backup or archive operation. A prompt for the same key is issued when restoring or retrieving the encrypted file.

With the GENERATE option, an encryption key password is dynamically generated when the Tivoli Storage Manager client begins a backup or archive. This generated key password is used for the backups of files that match the include.encrypt specification. The generated key password is kept on the Tivoli Storage Manager server in an encrypted form. The key password is returned to the Tivoli Storage Manager client for decryption during restore and retrieve operations.

TSM Client Data Security (Secure Sockets Layer SSL)

You can use Secure Sockets Layer (SSL) to have another level of protection beyond passwords. Secure Sockets Layer (SSL) is the standard technology to create encrypted connections between servers and clients. SSL provides secure communications for servers and clients over open communications paths through the use of digital certificates.

You can use the Secure Sockets Layer (SSL) or the Transport Layer Security (TLS) protocol to provide transport layer security for a secure connection between Tivoli Storage Manager servers, clients, and storage agents. If you send data between the server, client, and storage agent, use SSL or TLS to encrypt the data.

SSL is provided by the Global Security Kit (GSKit) that is installed with the Tivoli Storage Manager server that the server, client, and storage agent use. The Operations Center and Reporting agent do not use GSKit.

Each Tivoli Storage Manager server, client, or storage agent that enables SSL must use a trusted self-signed certificate or obtain a unique certificate that is signed by a certificate authority (CA). You can use your own certificates or purchase certificates from a CA. Either certificate can be installed and added to the key database on the Tivoli Storage Manager server, client, or storage agent. The certificate is verified by the SSL client or server that requests or initiates the SSL communication. SSL is set up independently on the Tivoli Storage Manager server, client, and storage agent.

You can set up SSL or TLS on the Tivoli Storage Manager server, backup-archive client, and storage agent to ensure that your data is encrypted during communication. You can use an SSL certificate to verify an SSL communication request between the server, client, and storage agent.

Check TSM infocenter  website for more details on how to configure Secure communication between clients and server using SSL.

Day 36

TSM BA Client Error log and Schedule Log Overview

TSM Error Log and Schedule Log Configuration

You have to provide space for error log and schedule log for each BA client and any TDP clients separately. If you are multiple schedules for one node, it is recommended to use separate log files for each schedule so that you can easily troubleshoot when problem occurs. You have to make sure that the users have permission to edit the log files. Most of the problems in TDP are because of this issue only. 

TSM Client Error Log File

The following error logging options are available, you have to mention these parameters in dsm.sys file on UNIX and dsm.opt in windows platforms.

• ERRORLOGNAME: Use this option to specify the path and name of the file where Tivoli Storage Manager is to store information about errors that occur during processing.

errorlogname /tmp/dsmerror.log

• ERRORLOGRETENTION: Use this option to specify the number of days to keep entries in the error log, and whether to save the pruned entries. The error log is pruned when the first error is written to the log after a Tivoli Storage Manager session starts and after every schedule that runs. If the only Tivoli Storage Manager session that you run is the scheduler, and if you run it 24 hours a day, the error log might not be pruned according to your expectations. You must stop the session and restart it so that the log is pruned when the next error is written. It is suggested that you set your log to be kept for 7 days.

errorlogretention 400 s

• ERRORLOGMAX: This option specifies the maximum size of the error log, in megabytes.

errorlogmax 50

• If you change from ERRORLOGMAX to ERRORLOGRETENTION, all existing log entries are retained and the log is pruned by using the ERRORLOGRETENTION criteria.

• If you change from ERRORLOGRETENTION to ERRORLOGMAX, all records in the existing log are copied to the pruned log, the existing log is emptied, and logging begins under the new log-wrapping criteria.

• If you change the value of the ERRORLOGMAX option, the existing log is extended or shortened to accommodate the new size. If the value is reduced, the oldest entries are deleted to reduce the file to the new size.

• QUIET: This option prevents messages from showing on your screen during Tivoli Storage Manager processing. For example, when you run either the incremental or selective backup command, Tivoli Storage Manager shows information about each file that it backs up. Use the QUIET option to reduce the number of entries that you must look through. When you use the QUIET option, certain error information still shows on your screen, and messages are written to log files. If you do not specify QUIET, Tivoli Storage Manager uses the default option, VERBOSE.

TSM Schedule Log File

To specify the number of days to keep log file entries in the schedule log, you can use the SCHEDLOGRETENTION option. When you run the schedule command as a service, output is directed to the dsmsched.log file and to the application event log.

By default, the dsmsched.log file is in the current directory. You can change the path and file name by using the SCHEDLOGNAME option.

schedlogname /opt/tivoli/tsm/client/ba/bin64/dsmsched.log

schedlogmax 200

If you change from schedlogmax to schedlogretention, all existing log entries are retained and the log is pruned using the new schedlogretention criteria.

If you change from schedlogretention to schedlogmax, all records in the existing log are copied to the pruned log, dsmsched.pru, the existing log is emptied, and logging begins under the new log wrapping criteria.

If you change the value of the schedlogmax option, the existing log is extended or shortened to accommodate the new size. If the value is reduced, the oldest entries are deleted to reduce the file to the new size.

The amount of detail is determined by whether VERBOSE or QUIET is set in the dsm.opt file. After scheduled work is performed, you can check the schedule log to check if the schedule is successful or not.

Options to improving the client data movement

You can control the amount of data that travels between the client and server before the server commits the data and changes to the server database. This option changes the speed at which the client works. You can also control the amount of data that travels when files are batched together during backup or during a restore procedure.

The TXNBYTELIMIT client option specifies the number of kilobytes that the client program buffers before it sends a transaction to the server. The default is 25600 KB.

The TXNGROUPMAX option in the server options file indicates the maximum number of logical files (client files) that a client can send to the server in a single transaction. The default is 4096 files. Set the TXNGROUPMAX option in the dsmserv.opt file.

The TXNGROUPMAX option specifies the number of files that transfer as a group between a client and the server between transaction commit points. Using a larger value for the TXNGROUPMAX option affects the performance of client backup, archive, restore, and retrieve operations.

You can use the TXNGROUPMAX option to increase performance when Tivoli Storage Manager writes to tape. This can considerably increase performance when a user transfers multiple small files.

Either set the TXNGROUPMAX option by editing the dsmserv.opt file on the server, or you can use the setopt command. The range is 4 to 65000 for the maximum number of objects per transaction. The default is 4096.

After the txngroupmax number is reached, the client sends the files to the server, even if the transaction byte limit is not reached.The txnbytelimit client option controls the amount of data that travels between the client and server, before the server commits the data and changes to the server database.