Summary
You want to report a bug and want to know which information is needed. The solution’s steps use the scientific method for investigation and analysis.
Solution
Electric Cloud recommends keeping a very detailed log of steps you perform (try to key this off of the build ID). When attempting to diagnose an issue, here is some basic information to gather:
-
What you are building
-
What you were using to do this (vanilla Electric Make (eMake), a custom version, and so on)
-
What you are trying to prove
-
What is the expected outcome
-
What happened, and why you think that happened
-
Keep the output of the step (build logs, annotation, eMake history, and so on)
Complete the following steps to retrieve detailed information that Electric Cloud requires to diagnose the issue. Electric Cloud support engineers will request this information almost immediately, so you may want to provide it proactively:
-
What OS is this happening on (be as exact as possible… version, service pack, patch level, and so on)?
-
Linux:
-
Run
uname -a
to show the OS information -
Run
cat /proc/cpuinfo
to show the processor configuration -
Run
cat /proc/meminfo
to show the memory configuration -
Run
cat /etc/issue
to show the installation version -
Run
rpm -qa
(or whatever package management tool you use) to list installed packages -
Run
ulimit -a
to show the OS limit -
Run
df -h
to list free disk space
-
-
Solaris:
-
Run
uname -a
to show the OS information -
Run
/usr/sbin/psrinfo -v
to check in the processor configuration -
Run
/usr/sbin/prtconf
to show the memory configuration -
Run
showrev -p
to show installed patches -
Run
pkginfo -l
to see installed packages -
Run
pkginfo -p
to see if there are any partially installed packages -
Run
ulimit -a
to show the OS limits -
Run
df -h
to list free disk space
-
-
Windows:
-
Run the commandline tool
systeminfo
-
Run
net use
to list mounted shares
-
-
-
What is the exact version of ElectricAccelerator that you are using (down to the build ID)?
-
Run
emake --version
on the build machine -
Check the Cluster Manager About box for the version
-
Check the Cluster Manager for the agents' EFS and Agent versions
-
-
What tools (if any) are involved in the issue/build, and what are their versions?
-
What is the exact error message?
-
How reproducible is the error (sporadic, 1 in 10 builds, always)?
-
Does the issue occur at the same point during the build each time?
Issues that move around in the build generally indicate one of the following: multiple issues, an issue with a particular machine, or an infrastructure-related issue. Issues encountered in the past:-
Overloaded file servers
-
Overloaded local network drivers (For example, AFS on Windows has an issue with too many concurrent connections.)
-
Overloaded ClearCase servers
-
Overloaded network
-
Filled up local disks on the agent or eMake machine
-
-
Does the issue occur on a specific machine, or does it occur on different machines?
-
Is the machine in question configured exactly the same as all others?
-
Can the machine is question see the same network resources?
-
A particular tool may not have been activated on the agent machine. (For example, Visual Studio requires the user to select the development environment to set up when it’s called for the first time. In a non-interactive build, this results in a popup sitting on the agent machine waiting for input.)
-
In some instances, a failure occurs because of an earlier step, so the location of the failed step is irrelevant.
-
-
Does the issue occur when running the build without ElectricAccelerator?
-
Are there any messages in the Cluster Manager system log?
-
If possible, provide a minimal test case that Electric Cloud can use to track down the issue. (It is understood that this is difficult.)
Agent Issues
-
For agent-side issues, enable full session tracing before running the builds. Refer to the following knowledge base articles for additional information gathering and troubleshooting tasks:
-
For "hanging" agents, provide a core dump (KBEA-00092 - Creating a process or machine core dump).
-
On Linux, depending on the issue, engineering may need a kernel dump (KBEA-00091 - Forcing a kernel dump).
-
In certain cases, it might be useful to run procmon on Windows to track what files/registry keys are accessed during the build.
-
Provide any messages in the Cluster Manager system log.
-
Send any memory dump files you can find:
-
Linux:
-
/core
-
/root/core
-
/opt/ecloud/i686_Linux/bin/core
-
A kernel panic will not produce a core dump in any of those locations. If a Linux system is set up to capture kernel dumps, it will create a dump in /var/crash//vmcore, or something along those lines.
-
-
Solaris:
-
/core
-
/tmp/core
-
/opt/ecloud/sun4u_SunOS/bin/core
-
-
Windows:
-
c:*.dmp
-
c:\ECloud\i686_win32\bin*.dmp
-
c:\windows
-
-
-
Provide the agent/EFS log files:
-
UNIX:
-
/var/log/ecagent*.log
-
/var/log/console*.log
-
/var/log/diskcache*.log
-
/var/log/erunner.log
-
/var/log/messages* or /var/adm/messages*
-
-
Windows:
-
c:\ECloud\ecagent*.log
-
c:\ECloud\console*.log
-
c:\ECloud\efs*.log
-
c:\ECloud\diskcache*.log
-
c:\ECloud\erunner.log
-
c:\ECloud\i686_win32\bin\ecagentsvc.log
-
Any errors or warnings in the Windows event log
-
-
eMake Issues
-
Run the build to gather more information.
-
Provide any messages in the Cluster Manager system log.
-
Provide an annotation file (
--emake-annodetail=basic,file,lookup --emake-annofile=emake.xml
). -
Provide a debug log (
--emake-debug=jnpf --emake-logfile=emake.log
). -
For parsing issues, you may need to collect a remote parser log (see ?KBEA-00041 - Collecting a remote parse log).
-
Are there any conflicts in the build?
-
Is history really being used (ensure you check the build details)?
Cluster Manager Issues
-
Provide any messages in the Cluster Manager system log.
-
Provide the Cluster Manager log files:
-
UNIX:
-
/var/log/cmlog
-
/opt/ecloud/<ARCH>/logs/accelerator.log
-
/opt/ecloud/<ARCH>/logs/ElectricAcceleratorCMService.log
-
/opt/ecloud/<ARCH>/apache/logs/access.log
-
/opt/ecloud/<ARCH>/apache/logs/error.log
-
/opt/ecloud/<ARCH>/mysql/data/*.err
-
-
Windows:
-
c:\ECloud\i686_win32\logs\accelerator.log
-
c:\ECloud\i686_win32\logs\ElectricAcceleratorCMService.log
-
c:\ECloud\i686_win32\apache\logs\access.log
-
c:\ECloud\i686_win32\apache\logs\error.log
-
c:\ECloud\i686_win32\mysql\data*.err
-
-
Install Issues
-
What version of the product were you trying to install?
-
What platform (OS and revision level) is running on the machines?
-
Describe the problem that you encountered (for example, install failed; install completed but the product did not start; and so on.)
-
Send the related installation log files, from c:\ECloud or /opt/ecloud.
Note: There is a temporary version of this log file in /usr/tmp (on UNIX), or %TEMP% (on Windows), the location is printed in the summary for the GUI install.