“Good artists copy, great artists steal” (Pablo Picasso). I think that sums up the current state of affairs in programming languages. Ever longed for a missing feature in your favorite programming language? A good chance exists that it will be added within a few years, stolen from another language. I’m not saying that’s wrong, but what about real innovation?
Let’s take three random examples. Type inference has proven to be very convenient, but it was already invented decades ago as the Hindley–Milner algorithm. It was first introduced in ML in 1973 and has been used in functional programming ever since. The more and more popular map() and filter() methods can save you a lot of lines of code. But, I remember using collect: and select: way back with Smalltalk, which also dates back to the 1970s. And, an even more recent thing like co-routines was already described in a paper by M. E. Conway in 1963.
Although taking decades, all Third Generation languages (3GL) seem to converge towards the same set of features. Will syntax, runtime, and ecosystem be the only distinguishing attributes?
In the ’90s, the future of programming was expected to be 4GL. Although this category of languages, coexisting with RAD (Rapid Application Development) and CASE (Computer-Aided Software Engineering) tools, was certainly not strictly defined, the main ingredients were: ease to use, along with better or tighter integration with the underlying database. Unfortunately, the 4GL movement did not survive the subsequent Internet technology (HTML clients), and—being a very vendor-proprietary development—it quickly lost its market to more open systems and 3GL languages.
Obviously, 4GL exactly as it was then, is not the holy grail. But what about the philosophy behind it? What about a language being integrated with client/server communication and database access?
With almost no exceptions, every information system has to deal with three major aspects:
- Data Persistency
With 3GL languages, we only seem to focus on the processing part. Whether you create an object, add it to a list, calculate new values, or pass around objects, everything we do in a 3GL language is just manipulating bits in memory. As soon as we need to store anything in the database, we glue strings together to form a query, and we have to rely on APIs. The same is true for client/server communication. Maybe with the help of frameworks, but still. As I wrote before in “We Need Better Languages, Not More Frameworks”, frameworks will never be able to add fundamental new abstractions to a language.
We can look at this in a few other ways. The Von Neumann architecture (see Figure 1) shows us a Central Processing Unit talking to memory, but the model does not say whether this memory is either internal or external.
Figure 1 – The Von Neumann Architecture
And think about it, what if there was no distinction between the two? Imagine that external memory was as fast as internal memory. It might then have been very natural for a program to directly access persistent data (maybe Intel’s 3D XPoint memory or something similar can ever achieve this, who knows).
Originally, SQL databases were heading against having business logic inside the database, close to the data—for example, in the form of stored procedures and functions. It is the introduction of what we now call an application server that created the separation between the business logic and the database. The trouble that resulted from this separation was even given a name: the object-relational impedance mismatch. I’m not saying that stored procedures as they were designed decades ago are the way to the future. But, the principle of having business logic close to the data, like in an object model, makes a lot of sense to me.
This would enable the runtime to optimize the data access depending on the exact client request that came in, without a programmer interfering by manually having to write queries at some arbitrary level halfway.
Also, remember that the original idea that databases could be accessed by multiple applications is not regarded as a valid strategy anymore. So in most architectures, we already bind a given database to a given application server. The logical next step is to technically integrate the applications server and database again.
So what’s holding us back from this vertical integration and having programming languages directly access the database?
I think a major reason is the fact that the separation between the application server and database (a three-tier architecture) is regarded to be a law of nature—like a dogma. One of the major drawbacks of two-tier architectures in the 4GL era was the difficulty to enforce schema changes against client applications. But, that can be solved in other ways—for example, by supporting a multi-versioned database schema.
Also, due to this three-tier dogma, developers tend to either invent new application servers to potentially support all database engines out there, or they implement new database concepts that can potentially be used by any application server. The result is a lowest common denominator solution for the communication between the two. It does not matter how fancy that communication is. As long as both tiers live in complete separate worlds, an impedance mismatch will always be present, even with NoSQL.
Finally, technically merging the concepts of an application server and a database engine also requires us to rethink the way source code is processed and managed. Accessing persistent data does not really fit the traditional idea of a compiler directly translating our code into some sort of executable form. Instead, we need a strategy in which the code is analyzed against the underlying database structure and data statistics trying to determine the most efficient access paths, like a query optimizer does with SQL (ignoring the SQL language itself, which is not what I propose).
Interested in reading more about the deeper reasoning behind this story and what solutions I propose? Have a look at my book about Vertically Integrated Architectures: https://amzn.to/2ROBf9l (expected to be released January 2019).