Top managers at companies moving forward with their digital transformation programs would all agree that their projects can’t succeed unless their teams can make sense of all the company’s data so they can then turn it into actionable product initiatives that generate revenue.
None of that can happen effectively unless companies can produce clean and accurate data that consumers can trust. Unfortunately, the sheer quantity of data and the task of managing it all has become so challenging that too often, sludgy, unqualified data works its way through data pipelines and into the analytical applications, negatively impacting everything from insights and outcomes to customer experiences and bottom lines.
What is DRE?
That’s where data reliability engineering (DRE) comes in. A subset of site reliability engineering (SRE), DRE consists of a set of tools and practices designed to help organizations achieve high data quality, freshness, and availability across the data life cycle.
As part of the work, DevOps, SRE best practices, and new technology are leveraged, but George Philip, practice head for digital insights at Hitachi Vantara, says that managers should think of DRE as more of a collaborative business philosophy than a discrete technology.
“The legacy way of doing data reliability doesn’t work in the fast-paced real-time environments of today where companies are moving products to market fast,” Philip says. “Today, companies have to do DRE earlier in the process of modernization. Rather than thinking about it after the fact, they should do DRE earlier on in the process of modernization.”
The Path to DRE
Companies can get started with DRE by getting developers to shift left. Businesses have to start thinking about producing clean data at the source and it all starts with developers building security and reliability into their application workloads right from the start.
It’s also important to practice organizational change management. As Hitachi Vantara’s Philip explains, DRE requires a change in mindset.
“For DRE to work, everyone in the pipeline, from developers to application support managers to marketing and sales have to be involved in the same conversation around data reliability and quality,” Philip says.
Companies also need to expand on existing data management best practices. In traditional monitoring, a data problem get identified, but a data engineer then must spend time to discover the root cause. He or she then must fill that gap.
Hitachi Vantara’s Philip explains that data observability tools went a step further and let engineers observe the data, identify the problem and recommend a fix. But engineers still must go in and remedy the problem.
That’s where self-healing engineering come in: Self-healing tools developed by HitachiVantara let data engineers initiate automated reconciliation of an incident, dramatically reducing the time needed for repair and moving the data faster into the analytical application.
So here’s the essence of DRE: build more reliable code from the start, practice organization change, and leverage new technology strategically to save time and deliver faster insights based on clean, reliable data.