Design Walkthrough

InfoQ Article

Note: we have published an article on InfoQ that gives an overview of Obevo and some technical details.

The InfoQ article has a slightly more up-to-date and cleaner representation of some of the information on this page. Feel free to look at the InfoQ article first before proceeding on this page.

Obevo Problem Statement - A Compiler for Stateful and Stateless Code Classes

Design goal of Obevo - to give idempotent deploy semantics onto a platform target that does not have it by default, the keys problem to solve being:

1. There is no existing mechanism to take a full binary of code and deploy it to upgrade an environment
2. Some objects within the environment are stateful, thus requiring incremental commands to modify them (and making #1 difficult)
3. There is no default ordering mechanism for object changes to the environment; must be manually defined by users.

Databases happen to hit all 3 points, notably for applying SQL DDL changes to DB schemas, per the overview

• #1: There is no mechanism currently defined by DBMS providers to define a full schema definition in source code and apply the results to a database that can work both against a blank and existing db
• #2a: Table modifications cannot be expressed by simply presenting a full view of the object definition. Instead, one must provide an incremental modification statement to alter the table, and changes should only be applied once, lest an exception is hit or data applied in duplicate.
• #2b: At the same time, other DB objects are stateless (unlike tables), where one can simply drop and add objects to recreate them. Such objects like stored procedures and views share semantics more similar to application code.
• #3:When deploying a full database, order must be considered to deploy dependency objects before the ones depending on them (e.g. deploying a table before the view that depends on it.

Noting point #2, each object type (stateful vs. stateless) have different ways to maintain them, and a tool should ideally be able to handle them both. This is the premise and value-add of Obevo.

Boiling down the core problem and design - it actually does not involve SQL or DB constructs at all. Instead, it is about managing code files in the stateless and stateful manners and applying changes to them to the target environment. The end result is a “mini-compiler” of sorts; hence, the title of this section.

This page details the generic pattern here on compiling stateful and stateless code for an underlying system that only accepts incremental changes. As of today, the implementations have been around RDBMS, but we’ve already started to venture into NoSQL where not all operations need SQL (Hive), and depending on the creative minds out there, we can apply this to more.

So yes, this is a solution looking for a problem :) - but an interesting problem it is!

Problem Terminology

Let’s lay out the main actor involved in a System, at least with regards to this problem statement:

Requirements for Stateful objects / run-once statements

For any given deployment activity, a set of statements need to be applied to a target environment.

As mentioned earlier, statements on stateful objects must only be run once, and the state of an environment over time is the accumulation of such statements applied to it.

e.g.

• Binary Version 1 - apply statements 1, 2, 3
• Binary Version 2 - apply statements 4, 5
• Binary Version 3 - apply statements 6, 7, 8

We present two ways to do this:

1. The Deploy Team notes which statement/s from the Source Code need to be applied for that migration
2. The Deploy Tool takes the full set of statements from the Source Code and figures out which ones need to be applied to the environment

Option 1 was the way teams did this without automation. Tooling-wise, it is quite simple; but it is a riskier endeavor as it leaves more responsibility on a human operator.

Option 2 is what most database deployment tools do. We detail that now.

Deployment Algorithm

Source Code Format

We have the algorithm in abstract now - the question becomes: how best to represent the Changes in source code?

We present 2 alternatives - the first is what most DB tools support that works best with stateful objects, and the second is what Obevo uses to work with both stateful and stateless changes.

Sorting Changes in the file-per-object format

The file-per-object format that Obevo chose has many benefits, but it adds some complexity in the tool’s implementation, namely: how to define the order of the changes?

With the file-per-migration format, it is simpler to just order by the files; but the file-per-object format would likely require some ordering constraints across changes. In this section, we discuss how this is handled.

[Developer Guide note - this section is implemented in com.gs.obevo.impl.graph.GraphEnricher]

Let’s look at the source code example above, redrawn as a graph:

1. Each filled box represents a Change
2. The stateful table* files consist of many changes, and the stateless view* file only has a single change.
3. The arrows represent order dependencies among the changes, notably:
4. view1 depends on table2 and table3 per the SQL definition (an explicit code dependency)
5. table3’s foreign key depends on table1 to be created (an explicit code dependency)
6. The Changes listed within a stateful object must be done in order (an implicit dependency based on the order in the file)
7. Note that changes not related via arrows (whether directly or transitively) don’t have any dependency order, e.g. table1.init and table2.init can be executed in any order.

Given that graph, an acceptable deployment order can be obtained usingtopological sort. Please read the link for more information.

[Developer Guide note - the topological sorting is implemented (with the help ofJGraphT) in com.gs.obevo.impl.graph.GraphSorter]

For the example above, we have a number of acceptable topological sorts:

1. table1.init, table1.alter1, table1.index, table1.alter2, table3.init, table3.alter2, table3.fk1, table2.init, table2.ind1, view1
2. table2.init, table2.ind1, table3.init, table3.alter2, table1.init, table1.alter1, table1.index, table1.alter2, table3.fk1, view1

For easier supportability by clients, we will tune the topological sorting algorithm to give a consistent sorting.

Defining and discovering dependencies in Source Code

One detail left for this algorithm - how is the dependency graph created from the source code? Specifically the explicit code dependencies, which require some sort of parsing of the code. Parsing code is not a trivial problem; to do it correctly, we would need ASTs for the coding language of the text, and that may not be easily avaiable or easy to implement.

Part 1: The simplest methology is to avoid parsing entirely by defining a metadata attribute on each Change for the dependencies. This would technically work, however it will not scale well for developers if there are hundreds (or thousands) of code files to maintain. But this method will still prove useful; we will com back to this.

Part 2: We can discover the dependencies in text using a rudimentary technique:

1. We have the object names (based on the directory structure) and the text of the code
2. For each Change, search for references to those object names. Whichever object names are found in the text are counted as dependencies.

[Developer Guide note - this technique is implemented in com.gs.obevo.impl.text.TextDependencyExtractor]

From the example above:

1. The object name list is: table1, table2, table3, view1
2. The text for view1 is as follows, with the obect names in red and italicized

1. Excluding the object’s own name (view1), we find that table2 and table3 are dependencies.

Part 3: What if our rudimentary technique doesn’t work? We may find false positives or false negatives.

In this case, fall back to the metadata tag option mentioned in part 1. We either let clients:

1. Override the programmatic calculation by defining their list
2. Supplement the programmatic calculation by specifying dependencies to add (fixing the false negatives)
3. Supplement the programmatic calculation by specifying dependencies to remove (fixing the false positives)

Putting it together - integrating DB Deployments and Other Platforms

Thus far, we have not had any specific DB or SQL imprints in the algorithms (The text parsing algorithm could work with any language).

The points where DB-specific (or any other platform-specific) log would go are:

• “Apply Changeset to Platform” - this point in the algorithm takes the Change as input; how to actually apply the change is platform-specific
• Deploy Log implementation - this should ideally co-exist w/ the environment (e.g. a table in the DB for DB platforms), but that is not an absolute requirement.
• “Read Changes from Source Code” - the structure to impose for a platform on users can be defined as an implementation desires.
• “Calculate Changeset” and Discovering Dependencies - there are a few minor details for each platform implementation to fill in (e.g. handling case-sensitivity)

[Developer Guide note - com.gs.obevo.api.platform.Platform and com.gs.obevo.api.platform.ChangeType classes are the key interfaces to implement to customize the behavior for each platform.]