In this Webinar, we’re going to focus on maximizing high-performance database throughput using NoSQL technology, while still being able to use SQL over the same single instance of NoSQL data. We’ll start off with a quick overview on FairCom’s unique approach to NoSQL technology, and how this approach provides not-only High-Performance and High Availability, but also affords for a nice blending of NoSQL and SQL over the same data. We’ll then finish with some real-world use cases of how this technology has been successfully deployed in performance sensitive environments.
Presenter
Randal Hoff – VP of Engineering Services, FairCom
As FairCom’s Vice President of Engineering Services, Randal Hoff oversees the technical support, training and overall product satisfaction of the FairCom customer base. Working directly with an advanced team of skilled engineers, Hoff keeps their focus on providing the best possible technical services for FairCom customers. His technical career, which spans more than 25 years, includes progressively responsible positions at FairCom and General Electric. Hoff holds Bachelor of Science degrees in Computer Engineering and Electrical Engineering from the University of Missouri, Columbia.
Slides
Transcript
Thank you for attending today’s webinar on High-Performance, High-Availability for Mission-Critical Applications.
My name is Randal Hoff, and I’m the VP of Engineering Services for FairCom.
In this Webinar, we’re going to focus on maximizing high-performance database throughput using NoSQL technology, while still being able to use SQL over the same single instance of NoSQL data. We’ll start off with a quick overview on FairCom’s unique approach to NoSQL technology, and how this approach provides not-only High-Performance and High Availability, but also affords for a nice blending of NoSQL and SQL over the same data. We’ll then finish with some real-world use cases of how this technology has been successfully deployed in performance sensitive environments.
Why Use NoSQL
For todays discussion we’re going to be focusing on FairCom’s NoSQL implementation, which is most closely associated with a Key-Value Database. This type of implementation is successfully used in many industries where application performance is paramount. A few sample industries include manufacturing, telecom, financial applications, and so on. Basically any place you need to store data quickly, and/or be able to later retrieve it quickly is a good candidate for a Key-Value Database.
So what is it about NoSQL that makes it appealing to performance-sensitive applications? Our customers tell us repeatedly that it’s Performance, Control, Footprint, and Reliability. By control, you empower the developer to work outside of the restrictions imposed by the SQL standard. Next, performance is driven by providing APIs that allow you to get directly to the data, without going through parsing, optimizers, etc. A smaller footprint generally means smaller code which results in faster throughput. And lastly, Reliability is an absolute for any mission critical system. So these attributes at a high level help drive performance within the Key-Value NoSQL world. We’ll elaborate more on these attributes throughout this presentation.
So NoSQL is good at providing fast performance – but why do I need SQL? Many applications, including some of the ones we are going to focus on in the case study segment of this presentation benefit not only from the performance of NoSQL, but also with the added advantage of SQL access.
The ability to share data with other applications in the enterprise, share with web applications, connect mobile devices, and perform on-the-fly ad-hoc reports through SQL, all provide compelling reasons to utilize a database capable of bridging NoSQL and SQL over the same live data.
At the heart of our product offerings is our “c-treeACE” database, which is the most mature Key-Value NoSQL solution available. On top of this NoSQL database we’ve incorporated a full ANSI-Compliant SQL database engine.
So with c-treeACE, we don’t make you choose between NoSQL and SQL. Instead, we let you choose any combination of APIs, all for use concurrently, over the same single instance of your data. One way we like to think about this is we empower the database developer with a true multimodel database, providing seamless support for both NoSQL and SQL.
This diagram shows a typical application using the c-treeACE database. By using c-tree’s flexible approach to database API’s, you can concurrently access your application data from any combination of the supported APIs. Here we are showing a sample application using our native non-relational interfaces and available SQL APIs, all accessing the same single instance of the data, all at the same time. Additionally, a 3rd party application like Excel, or your favorite reporting tool, could also easily access the same core data concurrently.
One of the secrets of how we are able to leverage SQL on top of NoSQL data, is by implementing many of the database fundamentals down within the NoSQL APIs. For example, a full ACID-compliant transaction processing engine with roll forward and roll-back support, row-level and key-level locking, and automatic deadlock detection to mention only a few.
By implementing these database fundamentals down at the lowest common denominator API, the c-treeACE Advanced Core Engine API, which is one of our NoSQL APIs, we’ve been able to allow concurrent access to the application data through any combination of the SQL and NOSQL APIs listed in this diagram. You can see there is a wide array of APIs supported here, including PHP and Python, JDBC, ODBC, C# and a very unique Java NoSQL API that implements a native Java API for accessing your data below the relational stack – providing up to 40% performance gains over JDBC. This wide array of control for the application developer, provides great opportunities for extreme performance gains.
This has made c-treeACE a popular choice in many markets, but especially successful in the high-performance high-availability markets.
Case Studies
Let’s change directions now and begin discussing real-world uses of this technology.
There are many industries that can leverage the blending of SQL and the NoSQL technology. Some of them are well-established industries that have used this type of technology for many years, while others are not known for being “NoSQL centric”.
But the one item most industries have in common is the fact they cannot afford the luxury of giving up SQL when implementing NoSQL data processing. We will cover an example from the Financial Industry and an example of Smart Metering, or Consumption Monitoring. Let’s start with Financial Services.
Financial Case Study
The first example is within the credit card industry more specifically: credit-card transaction authorizations. At the center of such systems is a module called a Payment Gateway, or simply a Switch, which is responsible for connecting the point-of-sale terminal in a store with both the credit-card issuer and the credit provider, usually a bank.
The process starts with the switch receiving the authorization request and then validating the request with the credit-card issuer, testing if the card is valid, what type of card has been presented, and other parameters. If the credit provider approves the transaction, the transaction is then sent to the credit provider, again typically a bank. Both entities need to approve the transaction in order for the purchase to be completed. This approval process is done through multiple standard, pre-established networks, such as Cirrus or Pulse. In the meantime, the switch needs to keep the transaction alive, indicating that the connection has not been lost, and that it is still verifying the information.
If the authorization returns approved from the bank, the switch sends it back to the merchant, completing the transaction. The complexity of this system is that it needs to be able to do this in milliseconds! And handle tens of millions of transactions per day! Let’s take a closer look at how they solve these challenges using NoSQL and SQL.
Here is a diagram representing a typical switch system. You can see multiple elements like the consumer, the merchant, the credit-card issuer, and the banks. Current Service Level Agreements to process a credit card authorization are within the range of 200ms to get an approved or declined answer from the bank from the moment the switch acquired the transaction.
This requires extremely fast lookups for information regarding where to route the transaction. Such processing cannot be done under such tight SLAs with all the overhead that a typical SQL query injects, therefore many of these systems rely on the use of a key-value NoSQL database, specially because they provide ultra-high speed lookup capabilities, with sophisticated indexing systems that can efficiently connect the store to the bank, thereby allowing the switch to accomplish these transactions within the very short SLAs.
Another important thing is High Availability. When acquiring the transaction, the switch needs to know where to go in case the primary server fails. The requirement of a fully ACID compliant transaction is very important to avoid situations like “the husband and wife syndrome”, where you have two people are using the same account to make a purchase at the same time, in different places. ACID prevents the accidental approval of a transaction that could result in an account overdraft. Most NoSQL systems are eventually consistent, but this unacceptable for mission-critical systems like a Payment Switch, where ACID is mandatory.
Now that we’ve walked through the credit-card approval component, let’s think about generating reports, billing information, balance consolidation, etc. Reporting and sharing data with other systems, including web sites and mobile devices, are typically done through SQL. Usually this type of information is transmitted in batch processing, and has no need to be in real-time. SQL capabilities are quite important for keeping these systems integrated with the rest of the world.
Utility Case Study
Another interesting combination of NoSQL and SQL is on the smart metering system or Consumption Real-time Monitoring.
The cost of energy, the increasing population crisis, and political factors are demanding more and more innovation from the utilities companies, feeding an entire industry of start-ups with focus on this new money.
This leads to development of new solutions that go far beyond the meter, which is still a very analog device in many countries and naturally is evolving to become digital. This increases the amount of information that can be monitored, and adds value to the energy bill that goes beyond the amount that is owed: it includes consumption patterns, saving strategies, better resource allocation, and a lot more. Let’s take a look at how such systems are using the combination of SQL and NoSQL to address this problem.
Here is a typical electricity consumer network. In most cases today, this network is disconnected, and the only way to gather the consumer information is by visiting each house. Meters are analog, or semi-digital, but have no communication features, which makes the whole process of managing energy a batch operation, with no real-time decision capabilities.
Improvements in the energy industry are rapidly developing and include: A more efficient distribution of energy based on predictive consumption patterns; designing educational campaigns based on an understanding of consumption habits; and offering savings reward programs during peak times. With the digital revolution in this industry, smart meters are demanding integration capabilities to connect sensors everywhere from the home, to the office, to manufacturing buildings and warehouses and even to the smart devices on our person.
The result of all of these new sensors is millions of messages being collected, and most of the time, locally stored, to be centralized and processed by the utility. This means that now they can have information in real-time. It can be used to support decision processes to adjust the energy distribution grid proactively, preventing loss and maximizing efficiency.
There aren’t presently technologies to store surplus energy in an effective way, which creates a problem both for the overload, when there isn’t enough energy being produced to satisfy the demand, and the case of over production, where there is no good way to store excess energy. In either case, having real-time analysis capabilities is critical to enable the utility to provide real-time adjustment of their energy production. As we saw in the financial industry, analyzing all the historical data and extracting useful information from it, are typical tasks that can be managed with SQL tables, because of the ease of joining tables and creating multiple relationships.
SQL is also very important to provide feedback reports to the consumer, as well as the traditional billing information, and other typical relational data results. This shows again the power of the combination of SQL and NoSQL, to handle both the complexity of data sets, the performance needed for real time data collecting, and the advantages of seeing information in a structured, relationship-like manner.
As we’ve explained in this webinar, more and more companies are turning to NoSQL databases to drive the performance of their applications. The data management flexibility and control that NoSQL solutions provide yield performance that is simply not achievable with a commodity SQL database. Yet, these same companies also realize they cannot be a “Not SQL” shop. SQL is a vital interface in some areas — reporting and integration to name two – and you should not be limited by an either/or approach when it comes to choosing SQL or NoSQL.
The use-cases we looked at today were intentionally diverse. What we hope you learned was that there are environments that cry-out for a solution that is “Not Only SQL” where both NoSQL and SQL are needed to achieve success. Our c-treeACE advance core engine brings you the best of both worlds—relational and non-relational, at the same time, over the same data—and allows you to solve your application-specific needs.
As we’ve explained in this webinar, more and more companies are turning to NoSQL databases to drive the performance of their applications. The data management flexibility and control that NoSQL solutions provide yield performance that is simply not achievable with a commodity SQL database. Yet, these same companies also realize they cannot be a “Not SQL” shop. SQL is a vital interface in some areas — reporting and integration to name two – and you should not be limited by an either/or approach when it comes to choosing SQL or NoSQL.
The use-cases we looked at today were intentionally diverse. What we hope you learned was that there are environments that cry-out for a solution that is “Not Only SQL” where both NoSQL and SQL are needed to achieve success. Our c-treeACE advance core engine brings you the best of both worlds—relational and non-relational, at the same time, over the same data—and allows you to solve your application-specific needs.
In Closing
In closing, I’d like to briefly introduce you to FairCom, so you have a better understanding of where we are coming from.
FairCom has been working in the Database industry since the late 1970s. Our specialty is helping application developers solve their high-performance data storage and retrieval needs. We’ve been doing this worldwide—with customers in over 100 countries. We are very proud of the fact that we have over 40% of the Fortune 100 using our technology.
We have our corporate headquarters in the United States just outside of Kansas City Missouri, an office in Salt Lake City Utah, and offices in Europe and South America.
Lastly, we’re very pleased to note that we were included in the latest Gartner Magic Quadrant for Operational Database Management Systems again in the fall of 2015, previously known as Online Transaction Processing Systems. With hundreds of database solutions on the market, it is a great honor to be selected by Gartner as one of a handful of vendors to include in their latest magic quadrant.
Finally, I’d like to invite you to visit our web site at faircom.com to learn more about our c-treeACE product line, our company, and our customers. While we’ve focused today on our core database technology, we also have unique solutions for those of you using COBOL and Btrieve with our c-treeRTG product, and our real-time replication component, c-treeAMS. If you’d like an engineer to contact you with more detailed information about our technology, feel free to select the Contact option near the upper-right corner of our web site.
Thank you for your interest in FairCom’s NoSQL + SQL High Performance Database Technology, and for watching this webinar.