Author: Michael Klotz
Sponsored by eBI Solutions LLC  

Introduction

In this paper, the author attempts to shed some light on the realities of Enterprise Integration projects, which, not unlike many big ERP and CRM implementation have a high failure rate or do not deliver the benefits originally anticipated.  After examining the common misconceptions and mistakes made before, during and after an integration project, a set of guidelines that will all but guarantee that such projects are successful and deliver on their promise.

Figure 1 - Integration Scenario (Single Site)

Integration � A Necessity

Integrating disparate systems within many organizations as well as connecting to business system of customers, suppliers and other business partners has become a necessity. The reasons are different for almost every type of organization, but some of the common drivers are:

• Need for accurate, real-time information
  

• Reduced transaction costs
  

• Elimination of human error (re-keying)
  

• Speed of transaction processing (e.g. to reduce inventory, improve cash flow, increase customer satisfaction)
  

• Need for customer and employee self service (24x7)
  

• Enabling new ways of doing business (e.g. EDI or RosettaNet with large partners)
  

• Automation of entire business processes across multiple systems (business process automation)

While these needs are very widespread and many organizations have undertaken some type of integration project, the failure rate is still very high. Some research puts the failure rate at well over 50%.

The first step to assure your organization is not in the group that has failed is to identify the root causes for failing implementations.

Why Integration Projects Fail

An integration project can fail in different ways. For someone who is looking for �big cost savings�, the project may be a failure if the expected ROI is not realized. From a technical perspective, the project is a failure if the attempted integration simply doesn�t work. For others, including users, the project may be a failure if it creates new problems, or if the system is not adopted the way it was anticipated.

The underlying causes of these types of failures include the following, rather common, types:

• Goals and Scope Are not well Defined
  If there was no clear understanding of the expected outcome, benefits, timeline and budget, it is almost impossible to gauge the success and often, these projects stall for a lack of direction.

• Integration is Primarily Considered a Technology Problem
  While technology is an important part of the puzzle, the fact is there are a number of products and tools to address almost every integration problem. This propels other aspects of an integration project, especially cross-system expertise, to be the most critical factors.

• Complexity is Underestimated
  Because an integration project often touches different applications, systems, platforms and stakeholders (in the case of B2B integration even external ones), the complexity of the project can easily be underestimated. Therefore, timelines and budgets are unrealistic right out of the gates.
  
  

• Impact on Users is Underestimated
  Often, integration projects are considered purely a �technical� undertaking. At first sight, that seems to be very true, but the reality is that the impact on stakeholders often reaches farther than rolling out a new application.
  
  

• Isolated System Knowledge
  Often, projects are staffed with system owners and experts, who only know one particular system to be integrated, while others only understand integration software. However, when it comes to designing the more detailed aspects of the integration, problems arise out of the fact that none of these domain experts speak the same �language.� Business process owners know the flow and transactions very well. They have an intrinsic understanding of the business drivers and day-to-day activities, but are typically not technical enough.
  
  Then there are technical experts, who know a certain system (SAP, for example) very well, but don�t necessarily understand the data or technology in a web site, which is owned by another domain expert. Put a third expert, an EAI specialist, who knows the package well, but does not understand your business process, and the systems that need to be connected and chances are, you are headed for trouble, or, at the very least, serious investment in time and money to bring everyone up to speed.
  
  

• Trying to Stick with a One-Vendor Solution
  While there are projects where the products of one vendor can address all the technology needs, in the vast majority of cases, this is not true. Vendors as well as the marketers of the numerous products on the marketplace, tend to want organizations to believe that one product can address all integration needs. Usually it�s not quite that simple.

Organizations often require a detailed ROI analysis to determine the viability of a given project. While this is an important thing to do, a few unforeseen slips in schedule or budget may turn a viable project into a money loser.

A number of these problems arise from the fact that Enterprise Integration is a considerably young discipline and, therefore, best practices have yet to spread and mature like they have with some other expertise domains. Also, because of the complex and heterogeneous nature of integration projects, it is more difficult to develop a detailed, step-by-step approach, which has been used in implementing certain ERP solutions with considerable success.

This should not suggest that it is not worth making a strong effort to develop and mature such an approach, nor should it lead you to believe that no one has developed any methodology frameworks for integration.

Rather than describe a specific implementation of such a methodology, we will explore some interesting realities of integration, followed by a series of guidelines for integration project success in the following sections.

Some Lesser Known Facts And Misconceptions

With many technological advances in today�s complex world, it is not necessarily what we don�t know, but what we believe to know that can be found as the root cause of problems. The following concepts may seem obvious, yet we may not necessarily consider them when approaching an integration project:

• ROI can be an Elusive Concept
  Unlike what we experience when we invest in Government bonds, an IT investment, does not have an ROI that can be accurately determined up front. It can, and absolutely should be calculated up front, though. If you are a CFO and it doesn�t exactly turn out the way the numbers said it would, don�t be surprised.
  Because of all the factors that can impact the timeline and budget of a project, the return you will realize may not be as high as expected, or it may take longer to realize it.

• Technology is Not the Biggest Challenge Anymore
  As discussed above, integration requires some advanced technology and is often viewed as a �technology project". There are many products available if used appropriately, will address the technical challenges today that required extensive custom development just a few years ago. While custom development may be required in certain cases and might be the best option in others, selecting the right combination of products (based on a solid foundation of requirements) becomes a critical factor today.

• Data Mapping is Your Biggest Task � By Far
  This probably requires some explanation: this is not to suggest that data is bad, incomplete or unfit for the business needs. Just like your domain experts speak different languages (e.g. SAP vs. Oracle vs. J.D. Edwards, etc.), so is the underlying data and the idiosyncrasies of APIs and interfaces unique and quite different in each major system.
  Consider the following: Across a number of projects, we have been tracking and classifying issues encountered. In each case, the majority of issues that required clarification, debate, compromise, and in some special cases special logic to translate or correlate data between different systems These type of issues made 85% of all the issues on one project alone.

• What Vendors Sell vs. What You Buy
  In today�s market, and in the competitive Integration market in general, software vendors are to a certain extent forced to sell the concept that their system addresses all your requirements. This is partially due to the buyers� unrealistic expectations  of the product. The reality is that these vendors built a product for many customers, not just for one organization.
  
  This should not imply that the software systems on the market today are not valuable to an integration effort. On the contrary, many products have matured tremendously and provide much value to an enterprise.
  Nor do we mean to imply that each product is the �same�. Based on an organization�s unique combination of size, systems, dynamics and anticipated future needs, there is probably one system that fits best.
  But don�t make the mistake and expect one software package will do everything you need right out of the box.
  
  Look for vendors who have a track record with their product as well as alliances with other vendors to complement or complete their offering. This may sometimes come under an OEM arrangement, so it may not be obvious at first glance.

Keeping these ideas in mind and applying common sense when exploring how to implement an integration solution will significantly improve the success of your project.

Strategies and Tools for Integration Project Success 

This section contains some strategies that have come from ongoing experience and refining an approach to successful EAI implementations.  Experience has shown that there is no such thing as a �doomed project�. With the right approach, discipline and patience, each integration project can be successful. The following sections outline strategies that have been used to consistently mitigate risks, meet deadlines and budgets and deliver the desired solutions.

Avoiding the �ROI Letdown�

Because of the many variables involved with technology projects and the �soft� nature of attributing value to some of the benefits, determining ROI on integration projects is not an exact science. However, there is a strategy to ensure that the ROI is either realized quickly or the project can be halted.

Typically, an integration project is comprised of a number of �integration points�, i.e. a number of different transactions or other data movements that  are similar in nature to many others. It is usually easy to pick which integration point should yield the highest ROI. This typically is an integration point that requires one or more of the following:

• High transaction volume

• Extensive manual effort

• High cost of failure or error

• Significant benefits from real-time transactions

Implementing this integration point first allows the project team to measure the realized return against what was expected and expectations for the remaining integration points can be adjusted and are usually accurate.

Focusing on the Real Work

We have determined earlier in this paper that technology is not the largest challenge anymore. Statistics across multiple projects we have implemented shows that data semantics and data type compatibility make up bulk of issues (up to 85% of all issues on a given project). Therefore, it is critical to structure the project accordingly and ensure that resources working on the project have the skill set to work on these types of issues. While knowing technologies and standards like relational database systems, XML, EDI, etc. are very important, it is the ability to understand data in the business context across these standards and technologies that will get problems resolved and projects done.

Dealing with Complexity

While integration projects are almost always complex, their realization does not have to be complicated and risky. Many integration projects are doomed to fail before they start due to unrealistic expectations. These problems can be manifested by unrealistic demands from the project sponsor, but often are rooted in underestimating the complexity of the task at hand.

To implement and integrate systems well takes time and money, but if approached right, can almost be guaranteed to be on time, on budget and deliver the ROI, given the expectations at the beginning of the project have been realistic. The following section on project management shows tactics to accomplish that.

Managing the Project

There are a number of different ways to structure and execute projects that increase the chances for success significantly. The project manager should use as many of the following as possible:

• Analyze Well
  

• Detail the desired results

• Understand which systems are involved and how

• Have a good understanding of the impact on stakeholders � this will allow you to get the right people involved  and ensure that there are no loose ends.
  

• Plan Well

• Be detailed in the planning phases and tasks (to a practical level) and plan in more detail for the short time-horizon (next 3-6 weeks)

• Build in some flexibility. Think of what-if scenarios and anticipate different paths of resolution ahead of time. This will save time, money and, most importantly, the faith of the sponsors and stakeholders in the viability and success of the project.
  

• Break the Project into Workable Units

• Divide the work into different phases. This is done by following a methodology, which breaks down tasks into different stages

• Within a phase, break down tasks into discrete units of work (tasks), that can be assigned to individuals and tracked for progress.
  

• Build Prototypes

• Helps uncovers issues that may not be discovered otherwise

• Problems and changes are much easier addressed on a small scale
  

• Stagger Rollout

• This allows the project team to deal with one set of functionality and/or one group of stakeholders at a time, therefore keeping the adoption process very manageable

• Allow the team to apply lessons learned to subsequent stages of the rollout

Be Smart in Selecting Technology

When selecting technology for the integration project, do your homework. Use external help from an unbiased expert. Make sure to consider the following factors when selecting technology:

• Short Term Goals
  Ensure that the solution addresses what you need to accomplish immediately.
  

• Long Term Goals
  Spend time documenting the vision for your organization�s long-term integration strategy and needs. This will let you avoid having to implement expensive one-off solutions later on or even having to throw out the short-term solution completely.
  

• Systems to be Integrated
  Understand the different systems that need to be integrated today and which ones need to be integrated in the future.
  

• Platforms in the Enterprise
  Be sure to consider the different hardware platforms, operating systems and databases used in the organization when selecting integration technology.
  

• Existing Skill sets in the Enterprise
  Skill sets are an important asset and are expensive to build. Make sure to weigh the amount of skills that exist in the enterprise as part of the total cost of ownership.
  

• Budget (ROI)
  While budgets should not be the driving force behind selecting technology, the ROI needs to make sense. More importantly, select technology that is right for the size of your organization.
  If budget constraints exist, look for technology that is modular and can be expanded in functionality and geography over time.
  

• One-Vendor Solutions May not be Best
  While often desirable, it may not be in your best interest to choose only one vendor for your software solution. Needs for specialized adapters or the ability to use existing messaging systems are good reasons to mix technology.
  It is important to keep a balance between the number of technologies and your ability to maintain a system in an economic fashion when making this consideration.

Avoiding Costly Mistakes

The approaches discussed above significantly reduce the risk of failure and surprises.  It is always good to implement risk-mitigating steps. The following list summarizes risk-mitigating measures, some of which have been discussed above:

• Implement transactions with best ROI first
  

• Analyze requirements thoroughly
  

• Design thoroughly
  

• Involve users, other stakeholders
  

• Implement in phases
  

• Implement the following:
  

• Proof of Concept
  

• Prototypes
  

• Pilot Rollouts
  

• Risk and Issues Tracking

The sections so far have elaborated on the issues involved with integration projects and ways to deal with them. The following sections present a structured approach to integration projects that can be adopted for any project.

Putting it all Together

This final section is intended to summarize the pieces that you need for an integration project to be successful.

Building Blocks for Integration Success

There are a number of factors that make a project successful and probably too many to be listed here. The following items should be considered important and cannot be missed on any project:

Unless senior management explicitly sponsors a project and emphasizes its importance to the business, the project has a high chance of failure. Resources then get diverted, users and stakeholders do not fulfill tasks and little issues become reasons for major hold-ups. It needs to be ensured that senior management clearly supports a project and keeps showing interest in the progress and success for the duration.

As previously discussed, many stakeholders, including end-users are affected by the integration either directly or due to business process changes. It is critical for the successful implementation and adoption of a new system to ensure users have input into the project and buy into the solution.

A methodology will serve as a road-map through the project. Unless the entire team knows the end-goal and understands what they need to do next, and why, the project will suffer. The methodology should not be a bureaucratic burden, but an important tool to help the team be more effective and efficient.

To execute the project successfully, the project manager has to use critical tools that help him or her keep track of progress, critical issues, trends and developments. Such tools include issues logs, status reports, risk tracking, budget tracking, Gantt Charts, and statistics (e.g. bug-count trend), especially on larger projects.

Milestone reviews should be done at the very least, during the end of every project phase and more often if the scope of the project is very large. During milestone reviews, the project sponsors and the manager need to review progress, issues and deviations from the original plan. It is critical to review any potential changes that may affect the project when the project lasts for a long period of time. Assumptions that have been made at the beginning of the project may not hold true nine months later.

While this sounds logical, many projects get in trouble due to the lack of these two components. Often, external pressures or internal impatience lead team members to cut corners and act as �cowboys.� It is on the project sponsors and the project manager to understand these factors and resist the easy fix. In the vast majority of cases, shortcuts end up costing multiples in time and money instead of what was intended to be saved.

While all the above are critical components, the most important one is definitely having the right talent on the project. Unless the team members have the expertise, the interpersonal skills and other critical skills, the project will not be successful.
It is important to secure the right talent, from the very beginning (before the project kicks off) all the way to project completion. Having the right users as champions and competent administrators running the system is just as invaluable as having the right consultant during implementation.
As a general rule, the most successful project teams have a small number of dedicated project team members, each of which has knowledge and experience across multiple expertise domains (e.g. ERP, CRM, Integration and web sites).

In the next and final section, a simple framework is presented, which has been used repeatedly to successfully complete integration projects of varying size and nature.

Example of an Implementation Framework

This framework overview illustrates one way of addressing the challenges typically encountered when integrating different types of systems in a systematic and organized fashion. Because of the widespread use of the de-facto industry standard Unified Process and the Unified modeling language (UML), the example is based on the Unified Process and UML.

There are four main phases in the methodology, which will be executed sequentially:

• Inception Phase (Envisioning)
  In the inception, or analysis, phase, the project team works closely with all the affected business and system owners to collect all the requirements, document them and to coordinate scheduling and approach to the realization of the project.
  
  

• Elaboration Phase (Design)
  In the elaboration, or design phase, the project team plans the detailed technical implementation while making sure that all issues and potential problems are addressed before they begin building the system. Towards the end of this phase, the team will potentially build a 'Proof of Concept' system, which will allow stakeholders to see that the solution works conceptually, early in the project.
  

• Construction Phase (Implement and Code)
  Based on the design, the system is implemented and tested for functionality. At the end of this phase the project team delivers a functional system.
  
  

• Transition Phase (Roll-out, Hand-off)
  Finally, in the transition phase, the project team conducts 'integration testing,' where the stakeholders ensure that all the requirements have been met. At the end of this phase, the solution will be deployed to the production environment and the system turned over to the users and administrators with all the necessary documentation and training.

Conclusion

Integrating different systems, data sources and even external trading partners has become a necessity in many organizations today. Integration projects are often complex and can fail for a variety of reasons.  However, if approached correctly, integration projects can consistently be successful and deliver on the business benefits that merit undertaking such projects in the first place.