MultiValue Framework vs. NoSQL/Relational Database

When we call an MV server a database we are identifying the core aspect but we are leaving a lot of the story untold. Our MV servers are full-fledged application development environments. They include a database, an application development language, a powerful scripting language, and a natural language reporting system. Telling the whole story communicates the value more clearly.

I'd like you to consider referring to it as The MultiValue Technology Framework. This better communicates that our "database" is really several important pieces that are seamlessly integrated together. I know I'm asking you to make quite a perspective shift, but our systems should get the credit they deserve.

There is a lot that the MV servers provide right out of the box. I've mentioned a few in the opening paragraph, here are some more:

• Flexible Data Modeling (Relational Data Model and NoSQL Data Model)
• Enterprise Class Features
• Business Application Layer

Let's untangle some terms that may help us to recast management's view of our tools.

Data Store vs. Database

Data store - a repository of persistent storage and management for collections of data. This can be anything that holds data electronically. CSV is a valid type of data store.

Database - includes the tools (and means) to organize, search, index, retrieve, combine, parse, and/or update data found in a data store.

These two terms are often thrown into the Relational vs. NoSQL debates because not all NoSQL data stores are databases, but all databases (NoSQL and Relational) can be used as a data store. The key difference: The database's ability to provide tools in addition to storage.

SQL vs. Relational Data Store

SQL (Structured Query Language) is a query language which is the API (Application Program Interface) that a relational database uses to provide an interface to their data structure. Relational databases were designed specifically to optimize the use of SQL by forcing the data store to keep the data in 1NF (First Normal Form). This caused the terms relational database and SQL database to be treated as interchangeable.

Hence the confusion when someone asks, "Are you using an SQL database?" Well, no, MultiValue databases are not databases that store information in a format that is optimized for 1NF. At the same time, yes, because nearly every MV database does support the SQL query language.

MultiValue query (sometimes called Access, English, ReTrieve, etc.) is a query language and MultiValue BASIC is an API, SQL is a tool for CRUD (Create, Read, Update, and Delete). None of them define their data storage format. They just leverage it in an optimized fashion. Several MV flavors allow you to select a back-end data store without any change to your queries or programs. Dan McGrath did an excellent presentation at the last Spectrum show on how to program in mvBASIC but CRUD in Big Table, which is Google's NoSQL data store.

The key idea here is that while mvBASIC and MultiValue query both allow easy interaction with the MultiValue files, they are not what define the MultiValue data store.

Flexible Data Modeling

For a long time flexible data modeling was looked at as a cheap hack and was frowned on by the traditional database administrator. That is no longer the case due to the rising popularity of NoSQL databases. Spun one way: SQL is well structured to enforce uniformity of methods while NoSQL (including MultiValue) is undisciplined. Spun the other way: SQL is inflexible and does not model the real world while NoSQL (including MultiValue) offers the option to model different data differently, either as an aid to clarity or for optimization.

MultiValue is still quite unique in our data modeling compared to other NoSQL databases. We can do one or more (or all) of the following in the same database, even sometimes in the same database table:

• Non-structured data (schema-less/application-based schema)
• Structured data (1NF, MultiValue attributed)
• Virtual data structures (data structures imposed on non-structured data)
• Mixed data structure (mixing structured and non-structured data in the same database)

Evolution From SQL to NoSQL

Relational databases are designed to enforce data structure and consistency on developers. That really is its whole design: Impose data requirements, restrictions, and structure. The critical assumptions are that data types are the most important aspect of any field. It assumes that single values are the only approach to data architecture.

NoSQL, including MultiValue, is designed to store, distribute, and access data using the best method for each case. NoSQL technology was originally created and used by Internet leaders such as Facebook, Google, Amazon, and others because SQL has trouble scaling in complexity. These giants require database management systems that can write and read data anywhere in the world, while scaling and delivering performance across massive data sets and millions of users. In the world of big data, these are some of the biggest.

NoSQL has evolved to meet the needs of high-speed transactional input/output. It is predicated on the idea of constantly evolving data models. And it does that while still keeping the enterprise-class features associated with relational databases. In this way, MultiValue is absolutely a world class NoSQL database.

If you don't think we've been evolving with the times, go to your vendor of choice and ask for the last few years worth of version change notes. Just because your shop may not be using the features doesn't mean they aren't there. If your shop isn't using the newest features, maybe they should be. There's a lot to gain in the newer releases.

MultiValue vs. NoSQL Data Store

Conceptually, most NoSQL data stores are not much different from what MultiValue developers have used for many years. Believe it or not, we have been ahead of the innovation curve for years. Sometimes we do a poor job of explaining (bragging) about MultiValue because we don't realize how hard other data stores and query engines have been working to keep up with us.

Let's look at a couple of examples. For reference, here's the SQL 1NF approach to data:

Key	Name	Dollar
129k	Nathan	$1234.00
128p	Nathan	$9881.00
515q	Nick	$9881.00

NoSQL Key-Value Data Stores

Key-value data stores emphasize the read/write of non-transactional data. The data is stored as a non-structured data element with a single value, and are accessed via a unique primary key. This type of data structure is best for applications that need fast input/output of information and don't care about structure of the value stored in the key. The term single value is misleading because the value might be one document or one JSON string, not just one number or a single word.

Key-value example:

129k.dollar	$1254.00
129k.name	Nathan

NoSQL Document Stores

Document stores emphasize the storage of JSON or XML. Similar to Key-value except that it provides query and indexing on popular fields found within the JSON and XML data elements. This allows developers to store JSON information natively, but still provide a schema to create a stored data model that can be queried.

Document store example can be found below.

129k	{"dollar":" $1254.00","name":"Nathan"}

NoSQL Column Stores

Column stores are best stylized as an inverse of a relational row/column data store. While not originally a NoSQL data store, but more of a relational indexing feature, it has grown into a data store that is popular for specific types of data models.

Instead of each column being a field of information within a row, the column is the primary concept, and the row is secondary.

Column store example:

Name		Dollar
ID	Value	ID	Value
129k,128p	Nathan	129k	$1254.00
515q	Nick	128p,515q	9881.00

NoSQL Graph Stores

Graph stores are really one of the most innovative data storage methodologies in the NoSQL world, but it is relatively limited in its applications. Graph data stores use structures that link the relationships between the data of each record. This is a very specialized format that allows very fast BI, analytics, and data mining. The relationships need to be defined as they would be in a traditional schema, but the data is categorized, reduced, exploded, and processed so that the inter-relationships are easily queried, instead of querying the data itself.

Graph data example can be found below

Enterprise-Class Features

Any LOB (Line-Of-Business) application will need some if not all the enterprise-class features. This has been a major argument against NoSQL. Fortunately, this is one of the places where MV is ahead of the other NoSQL players. MultiValue databases have had enterprise class features built-in for many years.

The features that are often looked at are:

Security

Access and update security provides the ability to control individuals, and/or applications, providing a sufficient granularity to the security of the data, processes, and endpoints. Most existing relational databases excel at table and field access security but lack strong endpoint security. Modern application developers have since moved beyond the need for security on individual fields within tables due to the ongoing evolution of application design. Since most databases provide access to the data through CLI (Command-Line Interface) endpoints and direct SQL query statements, only restricting access to fields and tables at the query level is no longer a winning strategy. This is part of the problem with SQL: It only supports one way to do things. And that way was defined in a previous age. Modern hardware, speed, programming languages, and user-driven demands are all moving away from that aging model.

I always loved this misguided quote from Ben Rossi, arguing for field security: "And even where security can be added to applications, this puts a huge burden on the application developers, not to mention the additional cost and time implications." < http://www.information-age.com/putting-enterprise-nosql-acid-ambiguity-out-123458126/ >

The reality is that most modern application developers are placing the security into the application regardless of the database security. The main reason for this is that most applications have to use security for the presentation model, rather than the rights to the data itself. It is about what the users can see, access, or update, not about what the applications can access or update.

If there is security on the database tables, then the application developer has to address the security violations and present UI elements to notify the user of issues. As an application developer, that makes database security a burden more often than it is an asset. Again, since most applications embed CLI scripts into a full blown GUI, the complexity of application development has evolved beyond that level of granular security.

Encryption

Database storage encryption is rising in importance because enterprise systems are routinely in danger of being hacked. This is a feature that you should implement as a must. Payroll and any other sensitive data that can be used for user identity theft should be routinely encrypted. The same goes for anything which might be a target of industrial espionage.

Data theft is no longer limited to hacking users' accounts; mining the raw bytes of your hard drive for interesting information is becoming easier. In additional to that, our backups are often not encrypted, but are routinely being saved to hard drives on the network. Those trusty old backup tapes weren't network accessible.

ACID Compliance

This has always been a stickler for database comparisons, and a requirement for the enterprise checklist. It is also something that relational databases have always been good at due to the fact that they are designed to enforce data requirements on developers. When working with 1NF, it is very difficult to do something incorrect with your data from a compliance standpoint. This enforcement is often an issue with developers that have constantly evolving data models, but until the NoSQL database started being used, they didn't know there was any other way.

So naturally, if a database didn't enforce ACID, it was wrong ! That was the traditional way of thinking about systems. The interesting thing about ACID in modern transactional applications is that the code takes care of that anyway. If a primary key doesn't exist in the database, the application has to respond to the error and correct it or notify the user. So to keep from having to address the data error after the fact, application developers are moving to control the front end of the application process, instead of the back, where ACID would normally be enforced. The requirement for ACID is more about "we have always done it like that," than any specific application requirement.

For many developers, the primary reasons for ACID were dead before they started in the industry. It comes from a time when databases existed on a single machine, and issues with multi-nodes or servers never had to be considered. It was built to help with transactions boundaries, but when you get into applications development, transaction boundaries aren't always clear. ACID's attempt to manage consistency imposes delays on database updates when you have more than one node or machine in the mix. Because it isn't optimized for the realities of modern networks, ACID tests must check all the nodes, or servers, to verify that the information exists. If it doesn't, ACID issues a roll-back to the application because of a perceived sync error.

Since ACID across clusters, nodes, and servers is very hard and can cause bigger issues, it is often turned off or ignored except for the simplest cases; for example: Does a customer record id exists? Many ACID shops are ACID in name only. Full implementation can put you out of business.

Replication

Even though ACID interfaces are supposed to include support duplication and replication, replication is often looked at separately from the ACID requirement. And it should be. Replication is often more important than ACID in modern applications. Business nowadays require 24x7x365 on their servers. Replication makes that possible.

It isn't always about backups though. Replication is also a way to support reporting servers and data warehouses. That means that replication is not always about identical data, and that's another reason ACID's day has passed.

Get Close to the Data

Most NoSQL databases and relational databases have had a problem that MultiValue does not impose on its application developers. Non-MV systems make it a challenge to get as close to the data as possible. There are a lot of reasons we want this closeness, but the main reason is performance. That may be hard to fathom if you grew up in MultiValue. Think about it. Why would you design a database that doesn't prioritize ease of access for developers?

Any large or complex application requires interaction with large volumes of data and the relationships between that data and other data within the database. Most of these relationships are not schema defined, but instead are dynamic. They have to be calculated on-the-fly.

Contrary to what database administrators often believe, the traditional database is just a place to persist data for use by an application at a later date. The database by itself will not keep the data fresh and up-to-date without an application and/or user using it. If it doesn't get used, it gets old and stale.

As data volumes grow in the database, the amount of processing that has to be done by the application grows as well. If you have to work with large datasets, then processing the data as close to the database as possible decreases the amount of time it takes to get an answer to a specific business process.

Relational databases tried to address this by creating transaction scripting (T-SQL, PL-SQL, etc.) to facilitate data processing as close to the data as possible, but transaction scripting is still limited by the fact that relational databases are primarily focused on storing and retrieving data, not processing and using the data. As I keep saying, SQL was designed based on outmoded assumptions. Unfortunately, if you look at many of the NoSQL databases currently being used, their transaction scripting is still just as primitive, if not more so, as their relational database counterparts.

They both depend on other application development languages to do the processing of the data. This means that the data has to be transmitted from the database to the application, processed, and then returned to the database with changes, updates, or additional queries based on the processing. Literally speaking, they are required to handle data in a highly inefficient way. This is not a failure of the SQL model or the various NoSQL models. They could implement more robust close-to-the-data tools. We've had them since the inception of Proc and mvBASIC.

The performance of accumulating the datasets and transmitting the datasets to an application becomes a bottleneck. Even with tricks like view, cursors, and transaction scripting, the database is still relegated to just retrieving the data and pushing it (in bulk) to somewhere else to be processed, and then pulling it back to be stored.

This processing bottleneck was one of the problems that NoSQL databases were trying to address, by changing the way the data was stored. They are predicated on the belief that doing the wrong thing (shoveling data) faster was the solution. The retrieve time is decreased, compared to SQL, so the application doesn't have to wait as long to process the remaining data. But, as you can see, NoSQL databases only address one aspect of that performance bottleneck, the retrieving and storing of the data itself. That still requires the data to be offloaded to the application in order to do additional processing, parsing, or combining that wasn't built into the database schema.

MultiValue does not suffer from these performance problems. It never has.

Tiered Application Architecture

We can see some of the value provided on the database storage side of things. Let's now look at the application architecture that the MultiValue framework offers.

We have all heard of the N-Tier application architecture. This is how most legacy applications and all modern applications are developed. Most developers usually choose between a 2-Tier or a 3-Tier architecture.

The two tiers in a 2-Tier architecture are the database and a client application, which work together[ see below ]. The client application generally includes all the business rules, workflow, and the data processing code. It exchanges information with the database. This is the simplest architecture, but it creates a lot of traffic between the database and client.

The three tiers in a 3-Tier architecture are: database, application processing (containing the business rules and workflow), and a client application (for presentation). This is the preferred architecture for most LOB applications that use relational databases[ see below ].

The 3-Tier architecture allows developers to put the business processing applications on the same machine as the database. By doing this, they get the majority of the application as close to the database as possible, so the processing time of pulling and pushing the data in and out of the database is decreased. The end result of the application processing layer is then transmitted to the client to be presented or interacted with.

When building modern MultiValue applications (GUI, Web, Mobile, IoT, etc.), we have a tendency to choose what we think of as a 2-Tier architecture. You really aren't creating a 2-Tier application, though. The third tier is inherent in the MultiValue model. Instead of the traditional independent layer pulling and pushing the data in and out of a database, our third tier is on top of the data, and therefore closer to it.

The MultiValue Database Becomes an Application Framework:

This is what sets MultiValue above being a database and qualifies it as an application framework. MultiValue acts as a tool to supply the structure and templates for constructing an application. And that's the very definition of a business-centric framework.

Any application framework generally allows all of the following:

• Skeletal support for applications
• Ability to create reusable components for an application
• Abilities to retrieve, process, and update data
• Interaction with input and output for users, applications, and processes

The MultiValue framework is held together by mvBASIC, but also supports query, reporting, PROC (Procedural Control Language), and Paragraphs (Batch Scripts and macros). Many of the database management tools provided with the MultiValue system are written using mvBASIC.

What is included in the MultiValue framework:

• Database data I/O
• Database schema management
• CLI (Command-Line Interface) I/O
• Math and string functions
• Background scheduling
• Printer management
• Reporting Tools
• Multi-user/multi-threaded support
• Encryption and hashing
• Data triggers and indexing

The mvBASIC environment's features aren't news to MultiValue developers. We use them daily to create LOB (Line-of-Business) applications. But its real power is the fact that you can't get a full blown programming language any closer to the database than this.

This means any calls done to endpoints into the MultiValue framework will process, retrieve, and update the data without having to transfer it in and out of the database. Since the MultiValue framework also includes a process scheduler, a developer could run processes in a multi-threaded/parallel processing model. This is why MultiValue developers can write 2-Tier applications which provide the value of optimized 3-Tier applications.

Framework Endpoint

The MultiValue framework has built-in support for several different endpoints in order to provide applications developers different interfaces and protocols.

CLI (Command Line Interface)

Most databases have a command-line interface for database management, but that is all it's designed for. The MultiValue framework supports full user interfaces at the CLI level. Input, output, data validation, and many other features are supported through Telnet, SSH, and O/S shell scripts.

Web (REST, SOAP, HTTP)

The MultiValue framework has additional support for interfacing with web software frameworks, as well as the ability to directly present data through web services. MultiValue frameworks are not limited to just presenting data for other web frameworks to use. It can also be used to generate HTML, CSS, and JavaScript directly, using CGI as an alternative for the CLI user interface.

RPC (.NET, JAVA, PYTHON)

RPC calls are supported, allowing the MultiValue framework components to be accessed through .NET, JAVA, and Python. These interfaces are not limited to just interacting with the data, but are best suited to interact with existing MultiValue BASIC routines, which can act as efficient pre-processors.

SQL Mirror (Access/update SQL databases)

Even though the MultiValue framework includes its own database, and is optimized to work with that storage structure, it is not limited to the MultiValue database files. It can also interact with SQL data sources as natively as its own internal storage structure.

ODBC/SQL

As stated before, SQL is a query language, which is also supported through the MultiValue framework.

Conclusion

Stop trying to explain MultiValue as a database. It is so much more than that. The real power resides in the application framework it provides to developers.

MultiValue, in one line: A MultiValue system is an enterprise class business application framework with a built-in application development layer for 3-Tier programming that combines the power and stability of a relational database with the flexibility of a NoSQL database.

Featured:

Mar/Apr 2017