In Part 2 of our series, we provide a framework to help you ask the right questions before embarking on a complex integration project. If you haven't already, be sure to read Part 1 first.

Understanding that integrations can balloon the complexity of a project, you may be asking yourself: why do it this way? That’s a valid question!

Why Integrate? Why not consolidate?

With all the potential complexity surrounding an integration, it may be worth it to ask the question: is there an alternative option?

There are three main factors to consider when looking at the value of a potential integration:

1. Where does the data live / who owns the data?
2. What business logic or outcomes do we need from the data?
3. Does the data need to be exposed to any other systems apart from the one integration we’re considering?

Often, the balance of integrating versus bringing an external service “in-house” (or into the platform) comes down to these three factors.

Data Ownership / Storage

It may well be that you cannot bring an external system into your platform natively (forcing an integration) because you don’t own the data you want to integrate with. This might be the case with any paid access to a database like a list of service providers or tweets. Someone else owns that data but allows you to expose it (or use it) on your site. You cannot bring that data into your platform natively, requiring you to go and fetch it from them.

Sometimes data ownership issues get complicated, too: just because your organization collects the data doesn’t necessarily mean you own it. This is sometimes the case with proprietary systems. Your organization may have fed the data into their system, but part of your license assigns them ownership of your data after you’ve submitted it. It is important to understand who owns the data you’re integrating with and where it ultimately can and cannot be stored.

Business Logic

Integrations usually bring with them some meaningful experience: either data to enhance some functionality or some processing of information for an outcome. For some examples, you could think here of a commenting platform that is used widely: having an account with that commenting platform may enable you to comment on many different sites on the Internet rather than needing to create an account on each site. Or, a small ecommerce site may integrate with a credit card processor that you’re already familiar with, so you can trust when you click “Buy now” and are taken to a familiar checkout experience, that the retailer is using solid technology to handle your private financial information.

In both of these examples, the meaningful outcome of interacting with this data isn’t owned by the platform itself: it is business logic that is owned by another company. The service you’re getting by integrating with them is that data transformation. The credit card processor takes a credit card number and turns that into a deposit in your bank account. The commenting platform stores all of the comments, their approval status, and who made them, so you don’t have to. Drawing bright lines around this functionality and understanding what the processes are that you’re relying on is key to understanding how hard or easy it may be to bring a particular piece of business logic to your platform, rather than outsourcing it.

Data Exposure

A system that makes the right data available to the right people at the right time can be difficult to set up and maintain! Even if you own your own data and you can replicate the business logic you need to perform on the data, you still need to evaluate whether or not other systems apart from yours rely on that information to be available in a particular format. If there are other such systems, then you’ll need to make the data available from your platform if you choose to bring that integration “in-house.”

This might mean setting up and hosting an API yourself. It might mean moving a database behind your corporate VPN. It might mean needing to comply with data protection regulations that you didn’t have to worry about before (like HIPAA or GDPR).

It is certainly possible to do (and sometimes it is very beneficial), but it does require careful examination of the whole system.

How to plan an integration

With all that context, let’s set up a scenario: you’ve been asked to plan an integration. Your company is planning to start selling widget online and would like to use a third-party shopping cart and credit card processing. How do you start planning for this discrete but complex set of integrations?

Define the Benefits

Like most plans, it helps to start by defining your desired end state. When this integration is done, what will your users be able to do? Will there be a new button on your website? Ten new buttons? Text entry fields? What is the value or benefit to the customer?

Write it down!

Put All the Cards on the Table

Next, investigate what information will flow through the system. Document it in as much detail as you can. Will there be products? If so, they probably have lots of data with them (prices, quantities, sizes, availability, etc). What about users? What is a user (a username/password combo? An email address? A shipping address and credit card)? Try to be as specific as you can be about both the data and the composition of the data you’re going to be working with. (If you’re integrating with a good third-party system, they may already have documentation about the data and data composition that you can use.)

Draw a Flowchart

A simplified view of the flow of information through a system. This documents requests that can happen, storage locations, and data transformations to make the system complete. This documents five integrations that are codependent.

Once you have a good understanding of both the outcomes you’re aiming for and the pieces you have, you can start to line up the pieces sequentially in the order they’ll be used/consumed by the systems. Block out areas for where there will be business logic or transformations (these typically slot in right before storage, before display, or after initial read).

As you go through this process, make sure you write down your questions. If you don’t know what data looks like at some point during the process, note it! That is an area that could drive up complexity (or you might be able to discover the answer to that question and disambiguate the system a little more).

Your flowchart will drive your understanding of the complexity of the integration and now it’ll be a whole lot less ambiguous, too!

Conclusion

Integrations can be scary! They’re inherently complex: you’re dealing with data transportation, information ownership, data models, business logic, user experience, and display logic. Until you have a good framework for breaking down that complexity into some core component pieces, it can be really hard to wrap your head around how they all fit together.

By taking this approach, we at Palantir are able to estimate the complexity of particular integrations quickly during both our sales estimating and project planning phases of our projects.

Integration by Christer licensed under CC BY-NC-ND 2.0.

Strategy
Industries
Government
Healthcare
Higher Education