In my previous bubbles post I explained about silos and why it’s important to “pop them bubbles”. I will continue the history where I left off, with the introduction of the best of breed systems. This is also the premise of this post: companies have just invested heavily in these monsters (sorry, systems), so they are very reluctant to get rid of them. Popping bubbles is plainly out of the question at this time in automation-history.
The realization is growing, however, that certain company-data is used in multiple systems. It seems like a waste of effort to make employees enter the same data multiple times. Furthermore, people tend to make mistakes. Isn’t there some way to automate this, to connect systems so that data is entered only once and then magically appears in other systems?
Well, you can make a batch in one system that dumps some data and then make another batch in another system that reads that data, all during the night. Nightly batches are still very popular. My favorite type of nightly batches are those that read directly from the database of another system. Of course, all sorts of problems immediately come to mind. Being dependent on data models of other systems is a big no-no in Computerland. You always want to hide the internal working of another system so as not to become dependent on life cycles of these systems. (so no, not really my favorites, but I always have to laugh when I encounter them)
Another disadvantage of nightly batches is the not-real-time availability of changes. Especially in banking this became apparent and it’s there that EAI became popular first.
EAI explained
Having systems exchange information (near) real time is indeed what Enterprise Application Integration (EAI) is all about. The term says it all: Integrating Applications within an Enterprise. But that’s easy: just have a system poop out its information and have all other systems scoop it up!
Even in its simplicity, that solution assumes quite a lot. Especially the near real time bit. It at least means that:
- Each system needs a triggering mechanism that sends out data as soon as it is entered;
- Each system needs a way to send out data;
- Each system needs a ways to receive data;
- Each system needs a way to check for data that another systems has sent;
- Each system needs to be up and running or data may be lost;
- The way one system sends data should be compatible with the way each receiving system can receive it (e.g. a boat signaling with flags to a boat using Morse-beeps is useless);
- The language of each system should be the same (French OR English);
- The definitions of each object should be the same (‘telephonenumber’ OR ‘phonenumber’);
- The format of each object should be the same (20190726 OR 7/26/19);
Now what you have to understand about these monoliths (sorry, systems) is that they are specialized to do the task for which they were made: ERP or CRM or what have you. It’s just that ‘the outside world’ does not exist for these systems, especially back in the day. With any luck there maybe was one manner in which data could be sent out of these systems besides a logfile. SAP for example had (and still has) the eDoc format. Another popular format is the CSV (comma-separated value) file. Luckily, along comes Integration Software that solves our problems.
EAI patterns
Most of the issues mentioned above could be solved with integration software that implemented EAI patterns. Issues 1 through 4 have to be solved by the systems themselves obviously, even though I have once or twice made a solution that used a database trigger to read from a database table of one system to send a message (and then have the guts to call it a custom adapter). A lot of Integration Systems came with out-of-the-box adapters for the big systems that probably did something similar, but now at least it was properly tested by the manufacturer (ahem…). Using adapters is a start towards hiding the quirks of a system from others.
Quite a number of these issues are solved by patterns. Patterns to bridge communication protocols (6 – translate waving flags to Morse code), translate language (7 – data format transformation. Not really French, but rather e.g. plain text to XML), and translate definitions (8 – data model transformation)
The rest is solved by consultants. Since I first started in this field, in 2004, by far the most trouble has been caused by Date. Not only is there the d/m/y versus m/d/y mismatch, but using Dates in combination with DateTimes and then marching into Daylight-Saving time is guaranteed to shift birthdays by a day. This is one of many issues, some of them stubbornly resurfacing every few years, that you need true Integration Consultants to solve or even recognize.
And we are connected!
Wonderful! All our problems are solved now. OK, not really. We have now used very expensive software to connect one systems to one other system, burned time and money by hiring expensive consultants, just to move the address of a customer from the CRM system to the billing system. Why then are we not done?
Well, there’s also 20 other systems and 300 other data objects that need to be moved. In order to avoid spaghetti between all these systems we use the rest of the patterns for reliable messaging, routing and correlating. (and much more)
All this is still EAI. It does solve most of the problems it set out to solve: moving data between silos. So why the claim that it fails to pop the bubbles? My question is: are the bubbles still there? To which the answer of course is: Yes! We only managed to build some bridges between them. The bubbles still only speak their own language, use their own processes that sometimes overlap and are still hardly aware of other bubbles. Integration has only amplified the differences: we need this and this data from them. Why can’t they perform? And furthermore, we have introduced a new silo, but have disguised it as a solution.
What? Introduced a silo? How dare you?
Well, this realization came to me once I stepped back from my own behavior and looked at how I perceived all other silos: “can’t even think outside their own system”, “never even heard of correlation”, etc. So full of my own jargon and how others couldn’t grasp Integration principles that I became my own silo.
Conclusion
While it does solve a lot of issues, EAI is still a far cry from dismantling the silos. What we need now is business involvement. Maybe the colleagues from the business have something that can make the silos understand they really are part of the same company, that they share a common goal. The silos should be made part of the automation of business processes (BPM).
Next post: popping bubbles: BPM.
