Enhancing data transport efficiency and real-time synchronization

NOS, a Portuguese telecommunications and entertainment group, is driven by its mission to connect people, companies, and institutions seamlessly by leveraging advanced technologies. NOS unifies fixed and mobile communications through intelligent and advanced networks. NOS needed to leverage the full potential of its data to continue improving the customer experience, optimize internal processes, and mitigate fraud and cyber risks. The Integration team at NOS plays a crucial role in capturing, monitoring, and simplifying data for the business to achieve its mission and strategic goals.

NOS faced a critical digital transformation challenge of efficiently integrating data across diverse systems without high costs. The absence of a streamlined data transport system impacted the business’s ability to identify problems, analyze data, and make real-time decisions, potentially delaying its overall digital transformation goals. The NOS Operations team needed support ingesting massive amounts of data from legacy applications, internal applications, middleware, frontend logs, and API gateway logs. The Integration team also identified the need for streaming data ingestion and replication from the cloud to local servers. This added another layer of complexity to operations and complications in linking data for a unified view. 

Since NOS was already using StreamSets across various business units, the Integration team knew it would be easy to implement and allow for faster development. This included the opportunity to leverage native connectors, such as Kafka, Pulsar, JDBC, and JMS; reuse pipeline fragments; customize with Jython and JavaScript; and have flexible installation options via a Docker image, Tarball (local installation), or the cloud.

With the evolution of a new CRM system using Salesforce, the Integration team started Project Replicate to ensure the migration of data from the new technology ecosystem to existing NOS systems. Project Replicate utilized StreamSets Control Hub to facilitate real-time data transport and replication, bringing together their diverse data sources. The team was able to carry out data validations and transformations and execute complex SQL queries with ease. They quickly realized how easy it was to leverage pipeline fragments to develop and change a pipeline to get the data they needed. This reduced the overall implementation time for the team, establishing a standard framework for data transport and manipulation.

Using StreamSets, NOS has experienced exponential growth in the number of pipelines within the project’s scope and continues to see growth in its transformation journey. The team has 124 batch and streaming pipelines, 117 jobs, and is expanding to more projects with a forecast of 40 new pipelines in 2024.

Marketing automation achievements so far include redefining customer experiences within NOS’s digital channels, managing advanced and personalized marketing communications, and ensuring real-time data synchronization. The Marketing team can now customize customer newsletters based on interests, personalize SMS pushes and emails, and execute cart abandonment or next-best offer campaigns.

Data synchronization has allowed the team to improve their equipment sales process. Price and stock divergence is automatically updated across NOS systems. This allows for more accurate sales forecasting, improved inventory management, and enhanced customer satisfaction.

The team values StreamSets for its ease of adaptation, development, and the ability to add custom code if necessary. “The average delivery time would take around 2 to 3 weeks to deliver value to the customer, depending on the complexity and development. With StreamSets, I now have everything I need ready and tested in a week,” says Oliveira.

Software delivery time from development to production has been reduced. Developments that used to take 4 to 5 days now only take 2 to 3 days using the integration framework. Changes can now be implemented in 1 to 2 days.