Inside c-treeACE: Transactional Technology

Note: c-treeACE became FairCom DB in October 2020.

In last month’s issue we introduced the c‑treeACE database engine from the inside with its core technology. In this issue we build upon that core functionality and describe the transactional technology. The transactional layer makes possible many advanced c‑treeACE features such as automatic recovery, replication, and dynamic backups.

The Big Picture

At its heart, c‑treeACE is a sophisticated, multithreaded advanced indexing engine composed of multiple subsystems:

• The Core Engine, which comprises three main services: the Thread Manager, the Disk I/O Manager and the Memory Manager, besides the data itself
• The Transactional and Operational Management layer, which includes all low-level functions to control and maintain transaction integrity, and includes the framework for the data access interfaces, such as SQL access
• The Low-Level API, which provides the interface for applications and programs to access the data structures, through a basic set of functions, such as insert, delete, etc.
• The Communication Layer, which provides the high level integration interfaces for access to c‑treeACE data through industry standard APIs, such as JDBC, ODBC, ADO.NET, etc.

c‑treeACE Transactional Technology

The Transactional Technology layer includes the low-level functions to control and maintain transaction integrity, guaranteeing complete recovery of data should an unexpected outage occur, such as hardware or power failures. With the c‑treeACE transaction logging facility, many advanced features can be enabled such as dynamic dumps with complete roll forward capability and complete index data backup; replication to failover sites; and transaction history for auditing.

Transaction control provides two extraordinarily powerful tools for your applications – atomicity of operations, and recovery of data. In fact, there are four key components of any complete transactional system, collectively referred to as ACID:

Atomicity – Grouping of database update operations into all-or-nothing commits

Consistency – The database remains in a valid state due to any transaction

Isolation – Concurrent transactions do not interfere with each other outside of well-defined behaviors

Durability – Committed transactions are guaranteed to be recoverable, even if the data and index files are inconsistent

c‑treeACE is fully ACID compliant and as such, c‑treeACE provides the highest levels of transaction integrity.

c‑treeACE transaction control can also be enabled for many legacy applications allowing many additional advanced features to be available. Automatic transactions, while not providing complete atomic operations, do allow full recovery, and an ability to add replication for fault tolerant systems and dynamic dumps of index data.

Atomic Operations

Atomic operations allow an all-or-nothing approach to database updates. Grouping operations together provides for both business unit cohesiveness, as well as performance. Multiple files take part within a single transaction. Should any operation within the transaction fail, this entire sequence can be immediately aborted. Compare this to maintaining individual file updates under your own control.

To enhance the all-or-nothing atomic operation, complete savepoint and rollback features allow you to quickly move within a transaction as conditions change. Errors can be handled and operations adjusted without completely starting over.

In addition, c‑treeACE offers extremely fine grained locking mechanisms both within and between transactions. The ability to keep locks from before a transaction begins, and maintain locks after a transaction commits gives c‑treeACE developers control unavailable in most other database technologies. This yields a key advantage to high performance and acutely tuned applications.

Journaling

FairCom’s transaction management system maintains files that record various states of information necessary to recover from unexpected problems. Information concerning ongoing transactions is saved on a continual basis in these transaction log files. A chronological series of transaction logs is maintained during the operation of the application. Transaction logs containing the actual transaction information are saved as standard files. They are given names in sequential order, starting with L0000001.FCS (which can be thought of as active transaction log, number 0000001) and counting up sequentially (that is, the next log file is L0000002.FCS, etc.).

By default, the transaction management logic saves up to four active logs at a given time. When there are already four active log files and another is created, the lowest numbered active log is either deleted or saved as an inactive transaction log file, depending on how c‑treeACE is configured.

Every new c‑treeACE Server session begins with checking the most recent transaction logs (the most recent 4 logs, which are always saved as “active” transaction logs) to determine if any transactions need to be undone or redone. If so, these logs are used to perform automatic recovery.

The c‑treeACE transaction logs serve as the ultimate control point for many important c‑treeACE features:

• Automatic Recovery on startup
• Dynamic Backups
• Roll forward/backward capabilities
• Replication
• Change History

Enhanced Index Backups with the Dynamic Dump

The dynamic dump takes full advantage of transaction processing when available. By having files under complete transaction control, you can back up both your data and index files, and be assured of a consistent state after restoring without costly time-consuming index rebuilds. Additionally, keeping transaction logs available since the last backup provides for full roll forward capability, such that you won’t lose any data since the last backup up to the time of the last committed transaction. These features are only available for files configured for full transaction control.

Transaction History

The transaction history feature accesses audit logs of transaction controlled files to add valuable historical capabilities in any application. This allows the programmer to examine the changes performed on each unit of information at a very detailed level. One such use of this technology is to track the changes made to a manufactured device as it proceeds through an automated fabrication factory. The exact history of each individual manufactured device can be determined from the transaction log files. The following types of information can be gathered from the transaction logs:

• A particular record position of a specified data file
• A particular, unique key value of a specified index
• All updates to a specified file
• Updates from a particular user ID and/or node name
• Some combination of the above

Replication

The c‑treeACE database engine provides a flexible set of facilities for data replication between servers. c‑treeACE implements this replication mechanism through transactional replication. Transactional replication is based upon the transaction logs maintained by a c‑treeACE database engine. Changes are read from the transaction logs of one c‑treeACE database engine and applied to another.

c‑treeACE replication supports:

• 1-way replication for “warm” backup servers
• Bi-directional replication for “active-active” systems
• Synchronous updates for local data for enhanced read performance
• Source code to Replication Agent for maximum control

Transaction Grouping

The c‑treeACE Server supports grouping transaction commit operations for multiple clients into a single write to the transaction logs. Transaction grouping, or commit delay, is a high performance optimization for environments with large numbers of clients sustaining high transaction rates. Commit delay takes advantage of the overhead involved in flushing a transaction log. The performance improvement per individual thread may result in only milliseconds or even microseconds, however, multiplied hundreds of times over hundreds of threads and thousands of transactions per second, this becomes quite significant.

Effect of commit delay on transaction commit times for multiple threads is shown in the accompanying figure.

FairCom has supported commit delay logic for many years and continues to optimize and enhance the effectiveness of the transaction grouping logic. For example, in recent c‑treeACE editions, the commit delay is self-adjusting and numerous heuristics and statistics have been added to constantly monitor and report the effectiveness of the current commit delay state.

Distributed Two Phase Transactions

Two-Phase transaction support allows, for example, a transaction to span multiple servers. This is useful for updating information from a master database to remote databases in an all-or-nothing approach. With distributed applications, the two-phase transaction commit enables synchronous updates to vital master database tables for real-time availability of data.

See Also

• Inside c‑treeACE: Core Engine Technology
• Inside c‑treeACE: Transactional Technology
• Inside c‑treeACE: Operational Technology
• Inside c‑treeACE: Interface Technology