DevOps Paradox

The truth about DevOps by the people on the front line

Viktor Farcic

• Publisher: Packt Publishing
• Year: 2019
• ISBN:
  9781789138801
• Edition: 1
• Topic: Computer Science
• Subtopic: System Administration
• Index: Computer Science

Discover DevOps secrets from leading experts. Viktor Farcic interviews DevOps industries voices including Mike Kail, Greg Bledsoe, Jeff Sussna, James Turnbull, Kohsuke Kawaguchi, Liz Keogh, and more.

Key Features

• Leading DevOps experts share their insights into modern DevOps practice
• Engage with the real-world challenges of putting DevOps to work
• Strengthen your DevOps practices now and prepare for future DevOps trends

Book Description

DevOps promises to break down silos, uniting organizations to deliver high quality output in a cross-functional way. In reality it often results in confusion and new silos: pockets of DevOps practitioners fight the status quo, senior decision-makers demand DevOps paint jobs without committing to true change. Even a clear definition of what DevOps is remains elusive.

In DevOps Paradox, top DevOps consultants, industry leaders, and founders reveal their own approaches to all aspects of DevOps implementation and operation. Surround yourself with expert DevOps advisors. Viktor Farcic draws on experts from across the industry to discuss how to introduce DevOps to chaotic organizations, align incentives between teams, and make use of the latest tools and techniques.

With each expert offering their own opinions on what DevOps is and how to make it work, you will be able to form your own informed view of the importance and value of DevOps as we enter a new decade. If you want to see how real DevOps experts address the challenges and resolve the paradoxes, this book is for you.

What you will learn

Expert opinions on:

• Introducing DevOps into real-world, chaotic business environments
• Deciding between adopting cutting edge tools or sticking with tried-and-tested methods
• Initiating necessary business change without positional power
• Managing and overcoming fear of change in DevOps implementations
• Anticipating future trends in DevOps and how to prepare for them
• Getting the most from Kubernetes, Docker, Puppet, Chef, and Ansible
• Creating the right incentives for DevOps success across an organization
• The impact of new techniques, such as Lambda, serverless, and schedulers, on DevOps practice

Who this book is for

Anybody interested in DevOps will gain a lot from this book. If you want to get beyond the simplistic ideals and engage with the deep challenges of putting DevOps to work in the real world, this book is for you.

Introducing Kevin Behr

As CSO of PraxisFlow, Kevin Behr spends his time working with clients who seek to develop their DevOps process. His 25 years of experience have been driven by a passion for engaging with the complex problems that large IT organizations face, and how we can use DevOps to solve them. You can follow Kevin on Twitter at @kevinbehr.

The journey to DevOps

Viktor Farcic: Hi Kevin, you've been involved with many topics that have become central to DevOps since your early childhood working with your father. How did your father's work prepare you for DevOps?

Kevin Behr: Well, it's exactly 30 years since I first got formally involved in the world of computing. In my earlier years, I had the fortune of growing up with my father, Harold Behr, one of the cofounders of the Association of Field Service Managers, or AFSM. For those who don't know, AFSM was one of the first global groups dedicated to global service managers. AFSM would discuss topics that are still related to DevOps today, such as how mainframe computing was going to be serviced, as well as discussing availability and continuity of value for customers.

I was seven years old when I started building small digital computers, working on vacuum tube equipment. I was about ten years old when I started working with midranges and mainframes, in the context of repair. My dad ran a team that would charter jets to fly to their customers whenever their mainframes went down, and they would fix them at night, so that they'd hopefully be ready and working by the time morning came. If they had an outage happen on a Friday, I would often go with them if they flew out in the evening. Even back then, it was a fun thing that I could do with computers and with my father.

Viktor Farcic: You were fixing mainframes at ten years old?

Kevin Behr: My job would be to hold the solder and heat sinks. This was a time back when you could actually service and fix these beasts! One guy would be on the phone with IBM Armonk, or whatever mainframe company they were dealing with, and they'd be getting traces to test for certain voltages and impedances on the boards. Then, they would solder and replace the bad components. I was better at soldering than most of them, because I had small hands so I could get into places, but they mostly had me hold on to heat sinks and make RS–XXX cables while they chain smoked, muttered fresh obscenities, and squinted through reading glasses while soldering.

Viktor Farcic: And by the time you graduated high school, you were in business with your father servicing mainframes. How did all that sit with your education commitments?

Kevin Behr: Yes, when I was about 18, we were taking vacations in Moab, Utah. But, like a lot of places where you'd go for a vacation, there was no work. In our case, that meant that there were no computer services or consulting companies or anything like that. So, my dad and I started a small computer consulting company. We went out to businesses, government and schools, and we built computers! It was just becoming possible back then to build cloned computers for the first time, and so we went right ahead and manufactured our own computers. And we serviced them, right along with any mainframes and minis that needed servicing in the area. I also picked up some work with a company that had a contract with the state. I had a pager, and when they gave me a call, I would go fix the mainframes and Wang OIS systems.

A few years later, my CS professor asked me how much I made in those years; I told him it was anywhere between $35,000 to $40,000, which was pretty good in the 1980s. My professor then grabbed me by the arm and said, "Leave and get out!" When I asked him why, he said something important to me:

"I'm not saying this because you're a bad student, Kevin, you're an exemplary student. And I'm not saying this because you're asking a lot of questions about who is going to manage all these people we are teaching. I'm saying this to you, Kevin, because you're right: somebody needs to go and write this curriculum. But to do that, they have to do it from empirical experience. Somehow, Kevin, you have to work your way through these organizations and write your learnings."

And while I didn't set out with that purpose, I did drop out—and I've taken the exact path that my CS professor advised!

Viktor Farcic: What did you do right after dropping out of college?

Kevin Behr: Over the next several years, I held every job in an IT operation. I got to know what it's like to be a network engineer, and what it's like to be a system administrator, and what it's like to be the lowly guy who checks the disk array fault lights, the fans, and the filters on the air-conditioning. From rotating the backup tapes to programming firewalls. I did all those jobs.

I also went to school to develop software. I'm a lazy and slow developer, but I made sure that I understood everything, from the bottom of the stack to the top of the stack. I started with the B language as we used to joke—as in assembler—which means staring at a lot of binary, which is hard for us dyslexics.

I found during this period that the more I worked, the more disillusioned and confused I became about the folks who were managing the technology. It seemed like companies were just promoting technical people who had been there for a while up into management positions. In many cases, those people were not very good at what they were doing. They were not trained to do those things, and they often didn't want to be managing those things.

"The more I worked, the more disillusioned and confused I became about the folks who were managing the technology."

—Kevin Behr

Viktor Farcic: But if you want a raise in your salary, then you need to become a manager. It's like for a long time you might want to continue being a coder but then, five years later, you need more money, so you think about becoming a manager.

Kevin Behr: But back then, there was literally no help or support for the people going into technical management positions. There was nobody to mentor these technology managers, nobody to answer their questions, and there was no documentation for them to read. There just wasn't anything for them at all, and I found this very strange, especially when you reflect for a moment on how much emphasis there is around most executive positions to prove competency, education, and experience; and to provide training and documentation to ensure professional standards.

This was very strange, and it affected my view of CIOs profoundly, because I didn't see CIOs making any decisions on their own. I stopped seeing it as an equal partnership between CEO and CIO as the parents of a company. The CIO job looked more like a babysitter than a parent.

Viktor Farcic: That's a great way to describe it.

Kevin Behr: In my view, the CIO wasn't a peer with a real position in most companies during the 1980s and 1990s. The CIO essentially worked for everybody else.

The management of information systems during the early 1980s involved a lot of finance people, and, of course, technology originally came into businesses through finance—to help them calculate numbers and construct books and records. The first computer from IBM was a time clock that was designed to track people's working hours. Technology solutions had always had the backing of finance groups.

It was therefore very interesting and very curious when finance proceeded in the 1980s and 1990s to kick technology out of finance! I remember seeing this happening when the PC first came out. At that time, I was a mainframe guy, so was biased, but, like a lot of people at IBM back then, I believed that desktop PCs were just business cards for the mainframes. So, I just sat in front of them every day. Computer. IBM. Computer. IBM. I didn't believe that PCs would amount to much.

Then, suddenly, we had client-server computing in the 1980s and 1990s. As far as I'm concerned, client-server destroyed computing and set us back 40 years. The issue with client-server was that we already had all those capabilities in mainframes, but they worked better, faster and were actually less expensive by the time you counted all the people, weird contractors, and vendors that you would need. But finance made the mistake of only looking at the purchase price of the computer.

Viktor Farcic: You could say that finance became its own worst enemy. But a mainframe cost a lot more than a PC, so the client-server idea must have been very attractive?

Kevin Behr: Yes, sure, but then there was another false assumption: that you could run the PC all by yourself because... it's personal. The reality is that when you have 100,000 personal computers, it is not personal anymore. You then need to manage all those PCs, and they are all distributed!

So, I kept seeing this disconnect between technology and organizations, and the disconnect between CIOs and CEOs, become greater and greater. It was not until some years later, through the 2000s and 2010s, that DevOps was working to heal this disconnect.

Bridging the CEO–CTO gap

Viktor Farcic: It's interesting how those first phases of your career related to a history before DevOps, including those tensions and disconnects you talk about that DevOps tries to address of course. Did your next career step, as CTO at IP Services, take you closer to DevOps as we talk about it today?

Kevin Behr: Yes, in the 2000s, I was the CTO at a company called IP Services, which could best be described as an early MSP and outsourcer for infrastructure. It provided mission-critical infrastructure for large fortune and global 500 companies. While I was at IP Services, we had to develop ways to manage across various systems of control, because we would have auditors from every client wanting to come in and inspect our operations.

At this time, I started collaborating and working with Gene Kim, another kindred DevOps mind. We were both CTOs reporting to a CEO, and we both experienced a very specific process of adopting and adapting our thinking to meet the challenges in our work.

Viktor Farcic: Did this experience help the disconnect you mentioned earlier, between CTOs and CEOs in organizations?

Kevin Behr: Yes, we noticed how CEOs often describe things with word pictures, using primary colors, and numbers from 0 to 9. On the face of it, this CEO language can feel super-reductive and oversimplified, and that's certainly how it would sometimes feel for Gene and myself, because we were both engineers at heart. And the point here is that, as CTOs, it took a lot of work for us to learn this CEO language and its associated CEO mental frameworks. But that's what it takes sometimes.

I also remember Gene and I agreeing how humor can help heal a disconnect. Gene found this great book called Throwing the Elephant, by Stanley Bing, and together we began to appreciate how Bing discussed "managing up" in a tongue-in-cheek way, like humor, from a Zen perspective.

Listening and finding common links with other people was another important lesson for us during that time. Gene and I would often meet at a restaurant/bar called Pazos in Portland, Oregon, where we would each describe common scenarios about our respective executives and clients. We found that we had a lot of passion and many common questions about our industry.

Viktor Farcic: Such as?

Kevin Behr: Well, we might say "How come Client A has all of these problems?" They have the same amount of money, and a lot of the same talent as Client B; and yet here we are, with Client B doing so much better. Why?

Gene and I were very passionate about these types of questions, and we convinced our bosses to let us put our pith helmets on. As Gene used to like to say, we were like old explorers cataloging plants and animals for the first time. Our world was business of course, and so we would study high-performing companies to see what they did differently.

We shared a lot of what we learned at the first Security and Audit Controls That Work workshop in 2003 that Gene and Stephen Northcutt chaired, and I gave the talk Blood, Sweat and Visible Ops, which was later memorialized in a book with Gene Kim and George Spafford called Visible Ops, which came out in late 2004.

Viktor Farcic: Why did you decide to use ITIL in your Visible Ops book?

Kevin Behr: We decided to use the language of ITIL because ITIL was a standard process language that a lot of people understood. We'd also mapped into ITIL all the actions that we'd been watching those different companies doing.

Our objective was to be able to compare the patterns of activity between successful and less successful companies using ITIL. What we discovered was that a lot of companies were doing things completely differently from the others—most critically, around how they managed risk and change. The more successful companies usually had the most effective change management processes.

A great example of the positive effect of good change management was at a client where we went in to change what was called a WAR, a work authorization request. The management of this client didn't like change because it was dangerous—and they happened to run one of the largest financial institutions. But the funny thing was that this client made way more changes than low-performing clients, and I was like wow! Their risk surface was much greater, and yet, they had almost no failed changes. Or, if they did, they were reversed very quickly and there was almost no impact to production.

We saw such high-performing clients as this one, and we saw low-performing clients, where both would be similarly skilled, and have similar budgets. The ITIL analysis showed that the key difference was the way that different clients were managing the change process that was integrated with release and incident processes. It turned out that 80% of failures were caused by things people did, and so the incidents are the results, and the changes are what we intended. We therefore started measuring things such as change success rate, and how do you know your process works? Is it successful?

But one of the things that we found about high performers is that they tended to have fewer controls than low performers. That was a big surprise. We were like "Hey! Wait a minute here!"

Viktor Farcic: Do you mean less control over people?

Kevin Behr: No, it was all about fewer process controls, such as from a management intersection or audit standpoint. So, we're there thinking our client has like 15 controls in here, while the other client has almost 40 from COBIT. And I'm like... this doesn't make any sense at all!

As we looked harder at this, we saw that the people with fewer controls were building purpose-built processes: they knew what their process had to do and where the risks really were. Meanwhile, low performers were reading best practices and they thought more controls were better for the auditors. And the lower performers treated every change the same way: they'd get a bunch of people in the room, and they'd talk about it, but that didn't make the outcome any more reliable.

The high performers were seeing changes as releases. They were looking at their whole infrastructure as if it were a platform; and as though they were releasing a new piece to this platform. They were looking at everything more holistically, and so they would track the interdependencies. They were doing a lot of things that were really and simply just in the change process, the incident management process, the release process. They had these processes integrated in such a way where you knew the outc...