March 31, 2015
Solved: Disappearing c-treeACE Core Files on Linux
Note: c-treeACE became FairCom DB in October 2020.
Uh-oh…it happens. Perhaps you ran shy of memory. Or worse, you violate FairCom’s cardinal rule of transaction logs: You ran out of drive space. Your c‑treeACE process unexpectedly dies. FairCom asks to see your core file for analysis. You go to look and…it’s NOT THERE!
The Redhat daemon ABRT, Automatic Bug Reporting Tool, (abrtd process) strikes again!
Automatic Bug Reporting Tool (ABRT)
This tool, found on modern Redhat Linux systems, sets limits, in addition to ulimits, for core size, and and other problem application information. You should become familiar with this very useful utility suite. More importantly, ABRT is integrally tied to the Redhat package management system for automatic bug reporting. As such, this tool may remove unknown core files as they are dropped.
Look for messages such as the following in your system logs:
Dec 12 3:19:22 hostname abrtd: Directory 'ctreedbs-13412346-1725' creation detected
Dec 12 3:19:22 hostname abrtd: Executable '/home/FairCom/servers/bin/ace/sql/ctreesql' doesn't belong to any package
Dec 12 3:19:22 hostname abrtd: Corrupted or bad crash /var/spool/abrt/ctreedbs-13412346-1725 (res:4), deleting
This is the case for applications not installed via the Redhat package management tool. System administrators should consider adding c‑treeACE to the accepted list of applications to monitor. This is done with the following options In your/etc/abrt/abrt.conf configuration file:
- ProcessUnpackaged = yes/no
This directive tells ABRT whether to process crashes in executables that do not belong to any package. The default setting is no.
- SaveBinaryImage = yes/no
This directive specifies whether ABRT’s core catching hook should save a binary image to a core dump. It is useful when debugging crashes which occurred in binaries that were deleted. The default setting is no.
FairCom advises to also configure your ulimit to allow unlimited core files (-c) (within storage space availability, of course).