This is a belated follow-on on from “The big 5 areas to nail when moving to the cloud“. It’s been a long time between drinks. Nothing like a new consulting gig to disrupt one’s writing habits.
I caught up with an ex-colleague for lunch recently. We’d both been working on Integration projects and were wondering why good integration capability is difficult and rare.
Integration often manifests itself in a shared asset like a bus or a broker. One issue is that IT is run by projects (with their own selfish interests) so often integration that is scalable, re-usable, loosely coupled etc. gets jettisoned just to deliver the project. This results in point-to-point integrations, managed file transfers and shared DB connections; whatever is easiest for the project to understand and implement. There’s nothing necessarily wrong with using any of these integration options in isolation, but over time the environment becomes unmanageable and ‘orrible.
If a mature integration capability has already been established you stand half a chance, but even then integration teams can be seen as slow, fussy and expensive pedants, to be worked around. (Not by me of course. I love you integration guys)
As a business grows and becomes more complex understanding how applications communicate becomes very difficult. At some stage an organisation invests in a discrete integration capability. (or DIC? Sorry for that moment of immature hilarity). The rise of mobile, cloud and outsourcing (the last being the least sexy of the three) has made integration even harder. It’s a cliché that information needs to be accessible anywhere, any time on any device and not just between systems housed in a single data centre. How do you meet these demands?
Integration – already complex – has become more complex. Security and integration teams must be proactive, forward-thinking and nimble to respond. The diagram shows common integration methods in a today’s organisation.
Sharing a DB between applications (the red arrows) is great for speed of exchange, but your applications need to always agree on the data format (which never happens with commercial software) and change/release management needs to be in lock step. I’ve never heard of an internal and external application sharing a direct database connection before.
Point-to-point integrations (purple) are fine in isolated sub systems where the likelihood of adding a third application to the mix is low. You could integrate an internal and external application this way, possibly with some bespoke format translation, but it’ll end up being some crude hole in the firewall.
File transfers (light blue) are quick to implement once you’ve agreed on the file format. Typically file transfers are done in batch and therefore not real time. You can make your batch transfers more regular but after a while the files are moving so often and are so small, you might as well look at messaging or web services. Externally file transfers can occur using a file transfer gateway device.
Messaging and queuing systems (light green) are great for moving data between systems where exchange is not time critical, delivery is guaranteed, and where systems have different data formats and standards. If using this method externally there are different messaging technologies and governance standards to manage. For example an organisation might use MQSeries internally, but their mobile app partner only has experience with Amazon SQS. You could start with a handcrafted adaptor if likelihood of re-use is low but it’s not going to scale.
Web Services (dark blue) have a distinct advantage because they use http(s) and can integrate with an API product such as Apigee. Web services have become central to the API economy (”…the design of APIs has become as important as the design of the user interface”). The cloud is in many respects a cloud of APIs. Request / Response integration has its disadvantages – no guaranteed delivery, polling etc. – but most things can integrate with it securely.
Real-time integration between in your inner and outer worlds is the future. Managing things like data confidentiality, service level guarantees, access management, transaction traceability across multiple environments and organisations will become increasingly difficult. Contract management will have to play a part.
The tools required will be provided – at significant cost – by the big established integration vendors. They’ll provide you with the tools, but you’ll have to build and run it. Same as it ever was.