Mapping and Transformation – A critical integration platform function

One of the most overlooked functions of an ESB Designer toolset is the ability to do mapping & transformation of data between a source document and a target document. As much as Application Integration is about connecting two or more application interfaces together, the end goal is to provide a seamless way to synchronize data or at least view data as a single source of truth. In order to be able to do so, when one application’s data needs to be synchronized with another, a series of data augmentation / transformation steps need to be done. And that is the mapping & transformation function.

Going back to my first statement, I did mention that this is an often overlooked function primarily because customers tend to think that this is a very standard featureset amongst all integration middleware vendors. When evaluating ESBs across multiple vendors, customers need to spend a good amount of time in this topic since they may end up spending most of their time doing integration work in this area. Let us take a couple of examples and understand why ease of use is such a big deal with this function.

Let us assume we are mapping customer order information from a CRM system to an ERP system. There may or may not be a canonical based on your needs and how you have structured your document types. As a side note, it is highly advisable to use canonicals to make your modeling and mapping highly extensible and reusable. Back to the example, the CRM order document may have customer name as two fields – First Name and Last Name. If that is how it is represented within the ERP document as well, our map just became easier. Just drag and drop First Name in the source doc to the First Name in the target doc. Evaluation check time – some vendor tools do not even offer drag and drop capabilities to connect up the two documents. Imagine doing a mapping exercise for over 200+ fields without drag and drop capability!

Changing our example slightly, let us say that the ERP doc just had Customer Name as one field. Now, we have the need to combine (or concatenate) the two fields (First Name and Last Name) in the source doc and then map the result of that operation to the Customer Name field in the target doc. As simple as it sounds, you may find it interesting that this simple use case can be implemented with a varying range of complexity across multiple vendor tools. Some vendors have a multi-step process to accomplishing this. Some vendors make you write code (java, javascript, groovy and other languages) to do this operation. And some that offer pre-packaged concatenation functions to do this operation instantly. So, spending some quality evaluation time in this topic will help you make well-informed decisions on how long your resources are going to spend creating integrations between two endpoints.

Key features to understand when evaluating an ESB for this capability –

  • Ability to understand a wide range of document formats (XML, flatfiles, CSV, EDI etc)
  • Support for canonicals
  • Drag-and-drop capability between source and target documents
  • Pre-built transformation functions to implement common data transformations quickly
  • Ability to create custom transformation services quickly
  • Ability to create reusable maps / segment maps that can be repurposed across multiple maps
  • Minimal or no code to do mapping / transformation
  • Ability to parse complex document structures (such as nested or referenced documents)
  • Ability to do looping over a collection of documents to extract and map data into another document

With that, I will leave you with one more pointer. Pick your most complex (yet often used) document type and use it for the evaluation. That will be a good reality check for you and your team. Adios!

About Dinesh Chandrasekhar

Dinesh Chandrasekhar has written 27 posts in this blog.

Dinesh Chandrasekhar has more than 16+ years experience in Application Architecture, Integration and Implementation across multiple industry verticals. He has special interest in on-premise / cloud integration, iPaaS solutions, high-speed messaging and solving complex integration problems. He is currently a Sr. Manager of Global Product Marketing at Software AG, responsible for the Application Integration product line.

Leave a Reply