Why you NEVER want to integrate - Part 1

It’s not easy to connect multiple systems to each other.  In fact it’s downright hard.  Truth be told, connecting the systems themselves isn’t really the challenge - that’s just a software project - the challenge is almost always dealing with the people that use the two systems you’re connecting.  Because you aren’t just connecting systems, you are connecting business processes, and you are connecting teams of people involved in those processes.  Teams of people who sometimes seriously dislike each other.  Who have spent years (sometimes decades) frustrating each other, blaming each other, and outright avoiding each other.

They aren’t the only reasons, either.  Here’s a longer list (this post covers points 1 and 2, the rest will be covered in future posts):

1. Integration shows how poorly people are doing their job

2. Automation (via integration) takes mundane work away from good staff… but it’s work they probably weren’t really doing anyway (see point #1)

3. All ‘upstream’ issues are suddenly visible and easy to identify… making upstream teams look bad and giving them a lot more issues to fix

4. Hidden crap data becomes visible crap data.  It takes effort and experience to clean up that mess, and who is going to pay for that?

5. The best defence is offence - when a new system highlights poor data, poor processes, and sub-par teams, then you bet the first stone will be thrown at the new system and the project team!

6. COTS systems can be easy to ‘turn on’ and intuitive to use, so they’ve got to be easy and intuitive to integrate with… right?  Our developers who have never seen the system before will pick it up straight away… right?

Of course this article is meant to be read ‘tongue in cheek’ - Integration is one of the most important elements to a successful digital transformation that makes the most of the distributed cloud systems and data already in place within the organisation.  

But these are real challenges that have prevented many integrations from going live, and the only way to successfully counter them is to understand them.

If you’re an executive trying to bring your teams together, or a project manager tasked with delivering an integration project, or an IT manager responsible for all integration systems, then this list is for you.  Good luck, sucker.

One customer we worked with did feel responsible, and did genuinely want to improve performance.  They had a central team that was responsible for setting policies on purchasing and vendor management (and ensuring the policies were followed).  However actual purchasing and vendor management activities were often completed by other staff in other teams who were both outside their hierarchy / jurisdiction, and who also had many other responsibilities.

Not only were the relationships strained (akin to the relationship between ex-inmates and the parole officers they report to!), but it was quite impossible for the central team to oversee all purchasing activity, and they didn’t have authority to intervene when processes weren’t being completed correctly.

On the other side, the real purchasing staff felt a lot of the policies were a burden, getting in the way of getting real work done.  These were good purchasing administrators: staff who served their own managers and were valued by their own teams.

The new purchasing system was intuitive and made many processes easier.  But integrating all of the disparate datasets immediately identified how poorly some of the central policies had been followed.  Data wasn’t recorded properly, and information on supplier insurances, contract end dates, even contract values were wrong.

What a nightmare!  The purchasing officers were clearly one of the most important stakeholders for the new system - if they didn't use it, then the project would be a failure - and yet it immediately alienated them by making it visible to everyone (including their managers), how poorly they had been following central processes.

To win them back, the project team had to cross that divide, empathise with these stakeholders, and advocate for real process improvement.  This required change management (not just running a few training workshops!): it meant bringing those teams together to negotiate new ways of working in a better system which could automate a lot of the policies that were getting in the way of the real work.

Staff costs are normally one of the highest costs to organisations, and the one most commonly looked to for efficiency gains. Mundane tasks that can be easily automated freeing up staff time are a no-brainer for efficiency gains, and many organisations have tackled these opportunities as low hanging fruit.  Subject matter experts should be used for process improvement where they can tackle real world problems, not menial tasks that add little value to the organisation.

Human error is a thing, it happens more often than not, regardless of how good the individual is at their job.  Integration as one tool used to automate the mundane action of moving data from one system to another, in a consistent and efficient manner.  It is one of the best actions to improve data quality where inter-system data is one of the problems, as business rules can be consistently applied, and escalated when exceptions are encountered.

Data corruption can occur at input where a system is unable to apply the appropriate level of validation, and one place to identify these issues is performed as part of the integration, where alerts can be raised to identify and seek manual correction data in an automated fashion.

The importance of data driven decision making has never been more important, and getting the right data at the right time is king.  Operating with old data is no longer acceptable in most businesses, and timely information is required to make the right decision.  Integration can help if done correctly here, but having a good understanding of the business needs is critical, along with how data flows through the organisation.  Some organisations look to address this with a business intelligence solution, however without good process and business rules are just kicking the data quality issues further down the track.