Help! My Data is Up There…

This is a true incident which happened a few months ago at an upcoming Bay Area-based wireless device manufacturer. I was there to meet with their IT director to discuss their enterprise integration strategy. They had recently acquired a SaaS CRM to replace their on-premise CRM application. They seemed happy and content about their decision. The IT director could not stop talking about the savings in terms of infrastructure and operational costs with this decision. He claimed that even the business folks were happy!

A few weeks later, he called me again and this time, he sounded perplexed. Business was not able to get a single view of the customer; Sales reps were taking longer to enter opportunities; Reporting was still a challenge. Problems galore! It was as if the cloud move did not quite help at all. Ironically, the SaaS application had become yet another data silo in their organization!!

Before you get your head in the clouds....So, before you have your head up in the clouds, here are a few data challenges you want to consider for data management in the cloud:

Your data has left the building - Once you adopt a SaaS application in a public cloud model, your data is no longer in-house. Be prepared to deal with data residing outside your DMZ (firewall). This is a major concern for a lot of companies that are considering cloud adoption. Specifically, if your company is in an industry that has a higher level of scrutiny and regulations, this becomes a sensitive topic. However, it does not mean that you cannot go cloud at all. Appropriate measures need to be taken to protect your data even if it is on the cloud – Think secure storage and the possibility of even encrypting your data in dire situations.

Security is a key concern – A related topic to the point above. Data is invoked and consumed via interfaces or services. So, make sure that you have a tight access control mechanism set in place. Authentication and authorization models need to be strictly enforced to protect sensitive data from getting exposed accidentally. More importantly, put together a data governance model. The more you adopt SaaS applications, the more you will end up using services to work with the applications and inherently, their data. So, plan for a SOA Governance model as well throughout the organization.

How often should I synchronize my changes with the rest of my systems? As simple as that question seems at the outset, it is actually tricky to answer. Almost all of the SaaS subscriptions charge you based on number of service invocations or at least limit you to a tier of calls that you can make within a month. Considering such limits, it is prudent for you to measure how often you access your SaaS application since each call has a cost factor. The other factor to consider is latency. Your network bandwidth will definitely play a key role on how quickly you can access your SaaS data. So, if you are thinking of synchronizing your cloud data with your on-premise data as quickly and as often as possible, cost and performance are going to hit you right in the face. Rather than re-inventing the wheel, leverage a leading ESB (Enterprise Service Bus) solution to solve this problem for you.

Data integrity and consistency issues – But then, you cannot stay away from synchronization due to the issues stated above because soon enough, you will run into data integrity & consistency problems. It is not really a catch-22. You just need to strike the right balance. This situation is no different than an on-premise integration problem involving a mainframe, for example. What you need to figure out is your system of record. Is it the one on the cloud or an on-premise data store? Once you figure that out, you can sort out the push/pull pattern of data, its frequency and update mechanisms. Moreover, this is where a really robust integration middleware solution can help you in keeping your cloud data and on-premise data in perfect sync.

Other key data questions to be answered – Apart from the above factors to consider, there are still more questions to answer before you decide how to handle your data on the cloud. For instance, here are a few:

  • Should you treat your transactional vs non-transactional data differently? If so, how?
  • If your system of record is on the cloud, do you need a copy of your master data to be made available elsewhere?
  • Is the SaaS application data represented in XML or some proprietary format?
  • Should data be pulled or pushed from/to the cloud-based systems?

Next Steps – So, if you are planning to integrate a SaaS application into your existing IT landscape, start planning for data challenges immediately.

  • Create a quick checklist based on the factors discussed above.
  • Set integration goals / objectives on what needs to be achieved from a business standpoint. For example, Business objective – Sales reps should not enter opportunity data in multiple places. IT goal – Sales reps will enter opportunity information only on SaaS CRM, which should then sync data automatically with on-premise ERP and lead-tracker database.
  • Data flow diagrams are always efficient to figure out which application integration patterns to use.
  • Engage your enterprise architect and your data architect to come up with a plan to help meet your goals.
  • Leverage a integration middleware solution such as an ESB to enable connectivity between your cloud and on-premise application.
  • Plan for data and SOA governance.

Until the next post, adios! In the meanwhile, talk to me about other data challenges you have faced in a situation like this.

About Dinesh Chandrasekhar

Dinesh Chandrasekhar has written 27 posts in this blog.

Dinesh Chandrasekhar has more than 16+ years experience in Application Architecture, Integration and Implementation across multiple industry verticals. He has special interest in on-premise / cloud integration, iPaaS solutions, high-speed messaging and solving complex integration problems. He is currently a Sr. Manager of Global Product Marketing at Software AG, responsible for the Application Integration product line.

1 Comment

Leave a Reply