Domino hints and tips

These buttons link to parts of the text below

Domino Performance Tuning

IBM Notes performance tuning is complicated, because there are so many potential performance issues. Possible problem areas include the hardware, the operating system and application design. If the application design is really poor, it will be difficult to fix it by tuning the storage subsystem. If the processors or memory are underpowered, then this will affect the Indexer speed, the Replicator speed, the number of maximum possible database transactions, and the number of add-ins that can run in parallel. If the Network is overused, then this will slow down all transactions.
From the storage viewpoint, disk access rates and configurations affect the general database performance and performance of views and view indexes. So if you are attacking a problem with Notes performance from the storage angle, you need to consider, and eliminate potential problems outside the storage sphere also.

So where do you start? With the Network! Run a simple TELNET data transfer between your Lotus Notes Server and a client, and see what kind of speed you get, compared to what you would expect from your network. If your network is rated at 10GB/s you will be lucky to ever see that, but you should see speeds of about 8Gb/s or more. If you are seeing less that 8b/s then you probably have network problems. You really need a network expert to check this out, as so many things could be wrong.

If the Telnet speeds are Ok, then you need to check that your Domino systems are using the network efficiently, On Windows, check the Processor Object, then the Percent DPC (Deferred Process Calls) Time and DPCs Queued/Second. These will give you an idea of how much processor time is spent servicing the network. If the DPC queue is growing, then you have an escalating problem.

If the network is Ok, look at the performance of your Domino servers. If you have one Domino server, which is accessing one disk drive, then really all you can do is suggest that your developers tune their application. This is outside the scope of a storage website but, here's a few ideas to throw at your developers. Constructive suggestions are always appreciated!

  • Make sure that when a document loads, text is displayed first, then images, so users can read the text while the images are loading.
  • For each individual user, Notes records if a document had been read or not, for every document in a database. This can be a real performance overhead, especially if maintaining these 'read marks' does not make sense for applications like help databases. Consider switching this off.
  • By default when a database is deleted, Domino will overwrite the data on disk with a pattern, to ensure no-one can read the residual data. This is not always necessary and is an I/O performance overhead.
  • It is possible to set up 'headline monitoring', so you can automatically track specific text changes to a database. This can be useful, but is a massive overhead for a big database with lots of headline monitoring users. It can be disabled.
  • Stored Forms use memory and disk space, consider preventing their use.
  • Every document has a two fields, $UpdatedBy and $Revisions, which are used to record who has been updating a document, and the date and time of each edit. The default retentions for both these fields arre very large, consider reducing them both to save on disk space, and also improve performance.
  • It is possible to consolidate large file attachments into one central repository, instesd of holding them in individual databases. This saves on disk space, so ensure that 'Use Domino Attachment and Object Service' is selected.

If you have several disk drives, then you have some storage options. Domino is IO intensive. Make sure that the active workload is spread over all your available drives. Consider setting up disk drive partitions, and arranging the system, program, and data directories to distribute the workload across the drives as evenly as possible. You may have to experiment a bit to get this right. What you really need, is disk subsystems which have large amounts of cache, so all writes go to cache, and most reads come from cache.

back to top

Performance monitoring tools

Server.Load is supplied with the Domino install CD. It is GUI based and provides some basic performance analysis data. Its simple to use, but just provides basic statistics.

There are several good third party products which report on Domino performance. An example is RoboMon from Itheon. This runs on Windows NT or Windows 2000 servers. As well as looking for bottlenecks in the IO subsystem, it will also give you some recommendations on how to fix developing problems. It checks on database sizes, compared to preset thresholds, and warns if databases are growing at unexpected rates.

back to top

notes.ini parameters

NSF_UPDATEODS is a notes.ini setting that has been introduced with Notes 8.5.2 and tells the client to update all databases to the latest ODS (On Disk Structure). The current ODS version is 51 and improves performance compared to its predecessor by 50%. This improves the client startup process and the process of opening local databases dramatically. Just add the following line to your notes.ini file. Where is your notes.ini? Well, it is usually located on your Notes programm directory. That is where your notes.exe resides. You should close Notes before you update your notes.ini!

NSF_UPDATEODS=1

This stuff is about changing the NOTES.INI file to get better debugging information about problems. Some of the commands are undocumented, so you should probably not be using them without talking to Lotus Support. However, if you know what the options are, then you know what to discuss with them. If you decide to change NOTES.INI yourself, then please take a backup of your NOTES.INI file before you start.
Its not a good idea to change the NOTES.INI file using a text editor, the best way is to change it using the NOTES.INI Settings tab in the Server Configuration document of the Domino Directory. Some available options are -

  • By default, debugging information is sent to the server console. Use the parameter

    Debug-Outfile=filename

    to switch the trace to write to a file. That makes it much easier to analyze the trace afterwards. Its probably best to put this file into a separate partition, if you can, as its size is unpredictable. You don't want the trace log to fill up your main database partition. If the trace partition fills up, then Domino should stop tracing, but otherwise carry on processing as normal. The log file cannot be deleted while the Domino server is running; if an attempt is made to delete the file, a sharing violation will occur. If you specify a directory path for the trace file, then that path must exist. Domino will not create the path for you. Once the Debug-Outfile parameter has been added to the NOTES.INI file, the Domino server and the Server's Notes client must be shut down and restarted for logging to begin. it is important, however, to insert a carriage return after the last NOTES.INI entry.

  • To list out AdminP schedule information, specify the parameter

    Debug-AdminP=1

    and you will see a schedule dump every time the schedule is updated. This needs Domino R5 to work.

  • If you want to see the order in which documents in a database are replicated, use

    Debug-Repl-All=1

    The output looks like

    03-02-2003 15:33:29 AM REPLICA: Updating NoteID 0x206F in Roles.nsf
    03-02-2003 15:33:29 AM REPLICA: Deleting NoteID 0x205C in Roles.nsf

  • If you want information on databases which are not replicating, use

    Debug-Repl-All=2
    03-02-2003 05:31:00 AM REPLICA: ...Skipping note in Destination list
    (UNID OCFFACF7EB:E2B4F2C4-ON672320AC:OC2347C9P3)
    03-02-2003 05:31:00 AM REPLICA: ...(Con't)
    Destination has same or newer note.
    Src Time/SeqNo=03-02-2003 05:31:00 AM

  • Debug-AMGR is a setting that allows the Domino server to report Agent Manager (AMGR) activities. Several agent Manager flags can be set, here is just a few
    • '*' This means output everything! Its one to watch out for. It will need lots of disk space, and will degrade client performance a bit.
    • 'p' Generates Agent performance statistics
    • 'm' reports on Agent memory warnings

back to top

Storage Tips for Running Domino on z/OS

You can store Domino databases on z/OS in two different file systems, VSAM HFS or zFS. If you use HFS, then you need to make sure that the files do not become full. Try to keep them at about 80% utilisation. DFSMS handles HFS file integrity on update, and can lock out access to the whole HFS file for one update. For that reason, its best to avoid large HFS files to minimise lockouts, and also avoid multi-volume files.

zFS has a better caching system. It detect sequential reads and pre-stage data in buffers to speed up performance. It also has a better lock/release mechanism. It is therefore much better to use zFS files for Domino databases.

The only major performance issue is channel utilisation, as Notes uses large block files. You need to monitor disk channel usage, and take action if it becomes excessive.