June 30, 2012
Inside c-treeACE: Operational Technology
We continue our series about the c‑treeACE database engine from the inside with its core technology. In last month’s issue we described the transactional technology of the c‑treeACE database engine that makes possible many advanced features. In this issue we continue upon that core functionality and describe the operational technology. The operational layer brings interaction with the many advanced features made possible from this core.
The Big Picture
At its heart, c‑treeACE is a sophisticated, multithreaded advanced indexing engine composed of multiple subsystems:
- The Core Engine, which comprises three main services: the Thread Manager, the Disk I/O Manager and the Memory Manager, besides the data itself;
- The Transactional and Operational Management layer, which includes all low-level functions to control and maintain transaction integrity, and includes the framework for the data access interfaces, such as SQL access;
- The Low Level API, which provides the interface for applications and programs to access the data structures, through a basic set of functions, such as insert, delete, etc.
- The Communication Layer, which provides the high level integration interfaces for access to c‑treeACE data through industry standard APIs, such as JDBC, ODBC, ADO.NET, etc.
When an application connects to c‑treeACE, it must identify itself as a valid user. The identifying code is called the User ID. To gain access to c‑treeACE, the User ID seeking access must be authorized as a valid User ID.
Users can also be part of user groups: by establishing and using groups, the Administrator can offer file-level operation control to selected groups of users. The Administrator can establish groups of any sort and associate each User ID to as many as 16 of these groups.
Each file created by c‑treeACE has an owner. The User ID in effect when a file is created is automatically made the owner of the file, however, the Administrator can later change a current file owner to any other valid User ID. The concept of file owner is important because it can be used with the file permission mask, a set of controls over who can do what with a given file.
Data backups are critical to ensure data integrity as part of disaster recovery and business continuity planning.
The c‑treeACE dynamic dump feature provides a safe, secure method of backing up data while the server is operational. An administrator can schedule a dump of specific files, which may be all files necessary for recovery or a subset of them. The dump runs while the server is carrying on normal activity and processing transactions and is transparent to users.
The server performs the dump at the first opportunity on or after the scheduled time. When beginning a scheduled dump, c‑treeACE temporarily stops new transactions from beginning and starts the actual dump as soon as all active transactions are complete (or after aborting transactions that do not complete during a pre-set delay period). After the dump begins, there are no restrictions on transactions.
Note: The dynamic dump and recovery processes are intended primarily for files under transaction processing control.
Quiesce and File Blocking
There may be times where an administrator may wish to open a maintenance window for system level management. A system administrator may need exclusive access to the physical files on disk, for example, initiating a hardware-based file backup at the disk volume level. Completely stopping c‑treeACE is a challenging task in high availability settings with large numbers of users. Coordinating this down time can be a frustrating experience. What is needed is functionality to halt c‑treeACE operations cleanly, while allowing user applications to remain connected. This function is frequently referred to as a “quiesced” state.
The c‑treeACE Quiet function can quiesce c‑treeACE operations and later re-enable them. This allows c‑treeACE administrators to perform maintenance or other on-demand activities without having to stop critical applications. Files are then readily accessed for activities such as backup. Users are temporarily held back from operations for as long as the external activities require. This is especially useful for hardware-based disk snapshot utilities.
There are situations where it would be extremely advantageous to completely block access to an open data file by current users. For example, consider the case where an administrator requires immediate access to a data file to compact the file since the file system was determined to be near capacity.
Typically in this situation, the application is brought down, as the compact operation requires exclusive access to the data file and indexes. With c‑treeACE, an interactive approach can be enabled to simply hold users out of the file while the operation takes place and allow them to “re-open” the file afterward. The c‑treeACE File Block API call gives you this ability.
Message Queues and Notification
With c‑treeACE server-based queues, client programs, connected to the same server, can pass information amongst each other, emulating a powerful inter-process communications mechanism that is portable, scalable and distributed.
c‑treeACE-based queues, also referred to as system queues, extend the concepts of c‑tree inter-thread queue routines by providing two additional features:
- System queues operate across the client/server boundary.
- System queues can be named so that different clients connected to the same c‑tree Server may share the same queue.
Server-based queues provide the foundation on which the c‑treeACE file notification feature is built.
File notification gives your c‑tree application the ability to respond to a variety of events at the ISAM level. Using the server-based queue feature, events can be placed by the server for defined events on selected files into an established queue – similar to an SQL trigger. With the proper handling of these events, you can enable such robust features as syncing data between servers in a replicated fashion, or triggering security and auditing actions based upon a specified file event.
Each notification message includes the following details:
- The type of operation (add, delete, or update)
- For a transaction-controlled file, the transaction number in which the change occurred
- The record offset of the modified record.
The following optional information can also be included in notification messages:
- The unique key value involved in the operation. (An update may return both old and new key values.)
- The record image involved in the operation. (An update may return both old and new record images.)
The node name of the client that performed the operation.
Statistics and monitoring
c‑treeACE tracks and reports a wealth of performance-oriented statistics. The server’s performance SNAPSHOT capability enables the capture of performance monitoring data using a combination of configuration file options and the c‑treeACE SnapShot API function. Monitoring the health of c‑treeACE is key to ensuring the best performance for your database applications.
Performance data can be captured automatically at specified intervals or on demand in the following ways:
- The c‑treeACE Statistics Utility, ctstat, provides a command-line interface to the SnapShot API function, supporting output of server statistics at the system, user, file, and function levels.
- Configuration options support automatic performance snapshots by specifying an interval between automatic snapshots and the contents of the snapshots. Performance data captured by automatic snapshots are written to the server’s system event log (SYSLOG) files. Configuration options can also be used to capture the automatic system snapshot data to the human-readableSNAPSHOT.FCS file.
- The SnapShot() API function can control automatic snapshots, overriding configuration options (if any), and can capture on-demand performance data.
- The c‑treeACE Monitor graphical utility provides a human-readable interface to read SNAPSHOT data generated by c‑treeACE.
Watch for next quarter’s eNewsletter where we’ll discuss the extensive menu of Interface technologies (APIs) for fast, flexible application development.