The DataOps Revolution

Delivering the Data-Driven Enterprise

Simon Trewin

• Publisher: Auerbach Publications
• Year: 2021
• Print ISBN: 9781032062969
• eBook ISBN: 9781000462104
• Edition: 1
• Topic: Computer Science
• Subtopic: Data Mining
• Index: Computer Science

DataOps is a new way of delivering data and analytics that is proven to get results. It enables IT and users to collaborate in the delivery of solutions that help organisations to embrace a data-driven culture. The DataOps Revolution: Delivering the Data-Driven Enterprise is a narrative about real world issues involved in using DataOps to make data-driven decisions in modern organisations. The book is built around real delivery examples based on the author's own experience and lays out principles and a methodology for business success using DataOps. Presenting practical design patterns and DataOps approaches, the book shows how DataOps projects are run and presents the benefits of using DataOps to implement data solutions.

Best practices are introduced in this book through the telling of a story, which relates how a lead manager must find a way through complexity to turn an organisation around. This narrative vividly illustrates DataOps in action, enabling readers to incorporate best practices into everyday projects. The book tells the story of an embattled CIO who turns to a new and untested project manager charged with a wide remit to roll out DataOps techniques to an entire organisation. It illustrates a different approach to addressing the challenges in bridging the gap between IT and the business.

The approach presented in this story lines up to the six IMPACT pillars of the DataOps model that Kinaesis (www.kinaesis.com) has been using through its consultants to deliver successful projects and turn around failing deliveries. The pillars help to organise thinking and structure an approach to project delivery. The pillars are broken down and translated into steps that can be applied to real-world projects that can deliver satisfaction and fulfillment to customers and project team members.

Part I
Scoping and Defining

Chapter 1

Setting the Vision

Key Concepts

When building data and analytics solutions, a large factor determining your success is the business sponsor and the work that you are carrying out meeting their needs. It is also likely that in low-trust environments prior to implementing DataOps, a shared vision of the direction of travel does not exist or is not accurate. Without understanding the true destination, you cannot be sure that the increment you are working on is taking you in the right direction. Successful projects are due to having a clear understanding of that destination or fighting to gain sight of it where it is not clear. Many people mistake agile for a free for all, where the destination of travel is determined every two weeks. In part this is true with regards to how you get there, but there needs to be a clear statement of the business goal to provide purpose. Many parts of the framework for a data project must be established over the longer term, and the shape of the framework is determined by the ambition of the vision.

JENNIFER WAS targeting the key meeting with Brian Catts to build rapport and start setting up the foundations for creating collaboration between business sponsor, the users, and IT. A key building block of this was for them to have a clear sense of a defined, realistic, and achievable scope. Jennifer would need to use all her skill to be able to define this over the coming weeks with the business users, business analysts, and her team. Tomorrow was less about this and more about really understanding Brian’s underlying target vision and to start to share the challenges that they would need to overcome in getting there. In Jennifer’s experience, it was sometimes hard to get to this point in one meeting, given the amount of noise around the day-to-day activities. Understanding the noise would help her to get a picture of the issues and the challenges in the relationship, but the priority for the meeting with Brian was to talk about the big picture. To shape the meeting, she put together her agenda to send to Brian to contribute to.

• Introductions
• Background—Brian
• Immediate needs
• Scope for department / business
• Housekeeping—regular meetings, communication plans, etc.

She sent the agenda through to Brian and quickly turned her attention to reading the existing documentation for the project. This would take the rest of the day and some of her evening to read through. She needed to understand the status so she could prevent the meeting with Brian getting sidetracked and to instill confidence that she was coming up to speed quickly.

She downloaded the documentation onto her laptop locally to prevent her having to do this remotely, and she printed out some A3 sheets containing the project plans. She would head home early and then set up residence in her office and work through the information.

• • •

The morning came around quickly, and Jennifer arrived bright and early. The business floor was situated in a large, open-plan office on the second floor. The floor was laid out in a series of desks in groups forming logical team working spaces. Brian’s desk was over on the south side with views out the window into the London skyline. Next to his desk was a glass office set out for team meetings and conference calls. The bank had global operations, and there was a need to regularly coordinate with colleagues in Asia and the Americas. The conference call with Asia was just ending, and the London attendees were filing out of the room to on with their work.

Several faces were familiar to Jennifer as they filed past. Brian came out at the end and greeted Jennifer warmly. “Hi, Jennifer, great to have you onboard. I hope you have settled in OK. Could I get you a coffee or some water?”

Jennifer had had several coffees in the morning, so she thought a glass of water would be required over the course of the next hour. They quickly collected the drinks and made themselves comfortable in the office.

“Hi, Brian, thank you for taking the time with me this morning. I started in the role as IT manager for regulatory delivery yesterday, and I am greatly honored to be able to lead this programme. I look forward to working with you and the team over the coming weeks, months, and years. I believe that we can really move the business forward and help to integrate modern practices into the work that you do.

I sat down with Mark Denby yesterday, and he made me aware of some of the challenges that you have had over a period. I always like to meet with the business sponsor first, before anything else, to get a clear idea of the vision and scope of what we are looking to achieve. I will then join that up with the work on the ground. I find this allows me to understand the issues and to put them into context with the high-level vision. This is of real assistance and will help me to determine the best path forwards. Does that fit with your expectations?” Jennifer asked.

“That makes sense, although I have had this conversation with your predecessors lots of times in the past and we do not seem to make much progress. The team here are really losing patience with this process—they seem to talk to IT, then some grand plan gets launched, and then when it comes to deliveries nothing of substance comes,” Brian said, struggling to contain his frustration.

Jennifer considered this for a couple of seconds and then responded. “The way that we need to set this up is to create a process of delivery that demonstrates value at intervals through the project lifecycle. We can then build back up the belief, trust, and momentum in the project,” Jennifer replied.

“Ah, that would be agile. One of the project leads brought in that a few years ago, and little seemed to change,” Brian said.

Jennifer realised there was going to be some work to do to build back the trust with the business. “I find that quite a lot with data projects. A standard agile approach does not work. What I suspect has happened is that the project team have not realised that there are only certain aspects of the project that can run agile. If you do not organise things correctly, then the agile process very quickly resorts back to a waterfall project. Part of the work that I will be doing initially is to identify this within the project plan then organise it to reintroduce agile. For this I will need to present to you and the team how this can work so that you understand our constraints and you can therefore help us to work with them. With agile there is a requirement for disciplined engagement and decision making and for us to be transparent with the blockers and the issues. Are you OK to work with me and the team in this spirit?” Jennifer asked.

Brian knew he needed results to be able to lift the morale of his team again. What had gone before had not worked. “Jennifer, of course I will work with you on this. We need to change approach, because what we are doing is not working!” he said bluntly.

“I will need a couple of weeks working with the team to understand where we are. In the meantime, they will be making progress on the immediate milestones, as this is key to make sure that we do not lose time against plan. Once I have had time to assess the current state, I will be in position to present back a set of recommendations and work through these with yourself and the other senior managers to prioritise them,” Jennifer explained.

“I understand,” Brian

Jennifer sensed that Brian had calmed down from his initial position and was willing to move forwards, so she decided to move the conversation on. “Reading through the documentation there is a business case that states that the goal of the project is to build up a reliable source of data to be able to meet future demands and to improve the reporting process. Would it be possible to elaborate on this more so I can understand this from your perspective?” Jennifer asked.

“One of the big challenges that we have on my team’s side is that there is a huge demand for information about the state of the business. This is coming from the regulators and senior management who are on the line for making sure there are no surprises. Passing each of these to IT is not feasible to meet the demands, even for new reports from existing data. We have employed many additional staff over the past few years to be able to meet the new demand, but the increase in costs is noticeable. I am having to reduce headcount whilst needing to produce more! Our attempts to build a central store and reporting engine has absorbed more resources than created efficiencies. I have resource dedicated to defining dictionaries, semantics, and business rules, but none of it seems to be making a difference.

“The IT team have produced a system, and it meets the requirements of one regulation, but when asked to add another, they seem to need to start again. On top of that, the data quality of what is produced is poor—meaning that my team must extract the data from the system and then massage it to be able to produce the reports that they need.

“These are the issues that are causing us the most problems, so my vision is to have a system that my team can use and work with that has source data and the means to cleanse it that is understood and easy to extend for additional requirements. The system should enable them to author reports and analyse data in a reliable and fast way. It needs to support full transparency and lineage to provide the management team with confidence to sign off on the results. This seems too big a step from where we are today, so we tend to go from requirement to requirement. If I gave this big mandate to IT, they would no doubt disappear for three years, and I would be left on my own to meet the immediate requirements,” Brian said.

“I understand your perspective. I have found that if IT do not understand the target vision, they will build a system based on half the information available. When they do this, the first iteration of the system is not able to cope with the second requirement, because the architecture of the system has not considered where the requirements are going. This means they have to do lots of work to be able to turn the implementation around,” Jennifer explained.

“So, you’re saying that the delays in the IT system are my fault then?” Brian asked a little indignantly.

“No, that is not what I am saying! I believe that there needs to be a target that is out in the distance that provides a compass to the actions and iterations of the here and now. This is not aiming fault; it is just being able to set a course and then to define waypoints on the journey to that destination to measure if you are on track. As an analogy, think of a journey that you have taken in the past. If the destination were local shops, you would set up for this in a totally different way to if you are heading to the Arctic Circle on an adventure. The number of supplies and the choice of vehicle, clothing, and sleeping arrangements are totally different.

“If you tell someone that they are going to the local shops, and when they get there, you tell them that they are heading to the edge of town, and then when they get to the edge of town, you say they are heading to the next town, and so on, eventually you will run out of ability to go any further. This is where a lot of agile projects fail. They start with a list of current needs with no end in mind, and then the iterations that they take are not heading in a consistent direction, which means that they look like a random walk. This is OK if you have infinite resources and infinite time, but that is not the case. Each iteration must move you forwards towards your goal.

“We therefore need a business vision and the IT vision to line up with each other, and then we need to establish the route to the target, meeting the goals of the business and IT on the journey. At each step to achieve a goal, we may want to make a conscious choice to build something tactically, but there should be as much reusable collateral to move us forward at each step to enable the next phase to be easier,” Jennifer said.

“I understand what you are saying, but how do we do this? Everything we do seems to get too big really quickly,” Brian said.

“Firstly, we need to identify what can iterate quickly and what takes longer. Each part of a system will have different cadence or volatility, and you need to respect that cadence and work with it to be able to operate effectively. If you put a high volatility part of the system in the core of the framework that is supposed to be stable, then you end up with a lock, which means the system appears to grind to a halt,” Jennifer explained.

“I am not sure I understand,” replied Brian.

“OK, by way of example, in the reporting system the report formats change quite frequently—for example, the layout or the groupings in the report. If you put the definition of these format at the core of the system that other parts of the system rely on, then you cannot change them without all parts of the system downstream of these being tested. This takes considerable time and effort, so therefore only happens once every three months.

“If you put the definition of these as close to the end of the pipeline and near to the reports, you can change these frequently without needing to test anything other than the report itself. The secret of data systems is understanding the scope of data and then put it in the right place. Going back to the vision, to be able to make meaningful iterations, then we need to understand these things, organise them correctly, and then make sure all the placeholders are in position. This way we can give you what you need today and build up the picture towards the vision over time,” Jennifer said.

“OK, so what you are saying is, we need to lay out the full vision for what we are going to need as a business and the roadmap of defined deliverables. You and your team will break that down into logical deliveries that satisfy the immediate deadlines, and then this builds to meet the target solution,” Brian echoed back to confirm his understanding.

“Yes, and the breakdown of the system will be what we present back to you. There is a discipline that is required that recognises that if we need to take a shortcut for a deadline, then we are granted the time to reengineer the shortcut into a more robust solution. Typically, this is where it gets hard—if we are not allowed to do this, over time, we will slow down our deliveries, and the costs will go up,” Jennifer explained.

Jennifer paused for a few moments to check if Brian had further questions before moving back onto topic. “To clarify the vision for the future, there is a number of questions that I need to ask you. Are you OK if I work through these?” Jennifer asked.

“Yes of course,” Brian responded.

“How do you see the business working with the new system?” Jennifer asked.

“As you know, our primary focus is the delivery of information to support the control teams to oversee and help to facilitate the business. The emphasis in recent years, for obvious reasons, has been more on the control of the business— providing infor...

Table of contents

• Cover
• Half Title
• Title Page
• Copyright Page
• Table of Contents
• Part I Scoping and Defining
• Part II Delivery
• Part III Appendix