KBEA-00067 - Diagnosing a slow ClearCase server

Summary

Your build is running in ClearCase and is performing poorly. The question is whether this is an ElectricAccelerator problem or a ClearCase problem and what to do about it.

Solution

ClearCase may or may not be the cause, but you may want to look into some possible issues relating to ClearCase. The ElectricAccelerator product increases the workload for a typical ClearCase server due to its faster performance. In the past there have been several incidents where the ClearCase server could not keep up with the high rate of requests.

Run your build with --emake-debug=g to get profiling information and check the profiling data at the end of the build. If active bandwidth to and from disk are poor, there may be a ClearCase problem. If the Terminator section reports a high number under "Commit by copy", you need to configure your emake temporary directories to avoid cross-VOB copying; this is most easily done by using the --emake-clearcase=vobs option.

Unix

The ClearCase logs are found under /var/adm/rational/clearcase/log. Go to that location and look at the various log files to see if there are any obvious errors/warnings. There are other potential issues though, mainly the network configuration, so check the following:

Your ClearCase hosts are all connected to the same LAN, with as few hops as possible between them.
Your DNS and NIS are correctly configured.
The servers reply reasonably fast. (An overloaded DNS server can slow everything else down around it.) Files to check include:
- /etc/resolv.conf
- /etc/nsswitch.conf
- /etc/yp.conf
- /etc/hosts
Traceroute and ping should give you round-trip times under the 0.2 ms between any two hosts, but especially between the servers (DNS, NIS, rgy, VOB, nfs storage, view).
Your disks are not mounted with a soft NFS option. Look at /etc/fstab, and /etc/exports on your file servers. /etc/auto.master, or ypcat auto.master depending on your nsswitch.conf.
Look at 'top' for the memory, swap, and load during the build.
You shouldn’t have large VOBs of hundreds of users on one small VOB server.
IBM Rational has some recommendations for the memory amounts. These are ballparks because everything depends on your usage patterns.
Take some 'mvfstime' of simple commands, and store the transcripts, for reference and comparison.

Windows

Use the ClearCase Lob Browser application to inspect logs and run the ClearCase Doctor application. The same network issues may be causing problems on Windows as well as on Unix, so check the following:

Your ClearCase hosts are all connected to the same LAN, with as few hops as possible between them.
Your DNS and NIS are correctly configured.
The servers reply reasonably fast.
Traceroute and ping should give you round-trip times under the 0.2 ms between any two hosts, but especially between the servers (DNS, NIS, rgy, VOB, nfs storage, view).
Your disks are local disks.
Look at the Windows Performance Monitor (Administrative Tools/Performance). Look for memory, swap, and load as well as individual process stats for memory.
You shouldn’t have large VOBs of hundreds of users on one small VOB server.
IBM Rational has some recommendations for the memory amounts. These are ballparks because everything depends on your usage patterns.
Take some 'mvfstime' of simple commands, and store the transcripts, for reference and comparison.

Applies to

Product versions: 4.x and later
OS versions: All