Operations Anti-Patterns, DevOps Solutions

Jeffery Smith

• Publisher: Manning
• Year: 2020
• ISBN:
  9781638350798
• Topic: Computer Science
• Subtopic: Project Management
• Index: Computer Science

Operations Anti-Patterns, DevOps Solutions shows how to implement DevOps techniques in the kind of imperfect environments most developers work in. Part technology tutorial, part reference manual, and part psychology handbook, this practical guide shows you realistic ways to bring DevOps to your team when you don't have the flexibility to make sweeping changes in organizational structure. Summary
Operations Anti-Patterns, DevOps Solutions shows how to implement DevOps techniques in the kind of imperfect environments most developers work in. Part technology tutorial, part reference manual, and part psychology handbook, this practical guide shows you realistic ways to bring DevOps to your team when you don't have the flexibility to make sweeping changes in organizational structure. Purchase of the print book includes a free eBook in PDF, Kindle, and ePub formats from Manning Publications. About the technology
To some extent, all organizations—even yours—suffer from poor development practices, garbled communications, and outdated legacy systems. The good news is DevOps can help you improve your processes. First, however, you'll need to recognize the core issues holding you back. This book empowers you to deliver DevOps with limited resources while navigating the office politics and entrenched mindsets that are all too common in actual workplaces. About the book
Operations Anti-Patterns, DevOps Solutions offers clear steps for transforming development and communication. Using jargon-free language, this book describes incremental techniques that pay off immediately. Streamline your workflow, manage unplanned time, and build operational metrics. Whatever your issues, this book holds the keys to organizational success. What's inside Turn failure into opportunity
Drive change through culture
Break down knowledge silos
Settle middle management turf wars About the reader
For team leaders and managers. About the author
Jeffery D. Smith has been in the technology industry for over 15 years. He has managed DevOps transformations at the ad-tech firm Centro and the online ordering platform Grubhub. Table of Contents 1 The DevOps ingredients2 The paternalist syndrome3 Operational blindness4 Data instead of information5 Quality as a condiment6 Alert fatigue7 The empty toolbox8 Off-hour deployments9 Wasting a perfectly good incident10 Information hoarding: Only Brent knows11 Culture by decree12 Too many yardsticks

1 The DevOps ingredients

This chapter covers

• Defining DevOps
• Introducing the CAMS model

It’s 11:30 p.m. on a Friday, when John, the IT operations manager, hears his phone ring. The ringtone is distinct, one that John has programmed so that he can instantly recognize a call from the office. He answers the phone, and on the other end is Valentina, one of the senior software developers at John’s office. There’s a problem in the production environment.

The last software release included additional functionality that changed how the application interacted with the database. But because of a lack of adequate hardware in the testing environments, the entire application couldn’t be tested prior to release. Around 10:30 this evening, a scheduled task that runs only quarterly began executing. The job was missed during the testing phase, and even if it wasn’t, there isn’t enough data in the test environment to create an accurate test. Valentina needs to stop the process, but she doesn’t have access to the production servers. She’s spent the last 45 minutes searching through the company intranet site to find John’s contact information. John is the only person Valentina knows who has the production access she needs.

Killing the scheduled task isn’t straightforward. The task usually runs overnight and wasn’t designed to be stopped midway through processing. Because Valentina doesn’t have production access, her only alternative is to dictate a series of cryptic commands to John over the phone. After a few missteps, John and Valentina finally manage to stop the task. The two plan to regroup on Monday to figure out what went wrong and how to fix it for the next quarter. Now both John and Valentina must stay on guard over the weekend in case the behavior repeats itself with another job.

Chances are this story feels familiar to you. Having production code that hasn’t been properly tested feels like a scenario that could have been avoided, especially when it interrupts a team member on their off-time. Why is the testing environment insufficient for the needs of the development group? Why wasn’t the scheduled task written in such a way to make stopping and restarting it straightforward? What’s the value of the interaction between John and Valentina if John is just going to blindly type what Valentina dictates? Not to mention the two probably skipped the organization’s change approval process. Nothing raises the safety of a change like five people approving something they don’t understand!

The questions raised here have become so commonplace that many organizations don’t even think to examine them in detail. The dysfunction detailed is often accepted as inescapable, due to the difference in roles between development and IT operations teams. Instead of addressing the core issues, organizations continue to heap more approvals, more processes, and tighter restrictions onto the problem. Leadership thinks that they’re trading agility for safety, but in reality, they’re getting neither. (When was the last time you said, “Thank goodness for change control”?) These negative and sometimes wasteful interactions between teams and processes is exactly what DevOps is attempting to solve.

1.1 What is DevOps?

These days, “What is DevOps?” feels like a question you should ask a philosopher more than an engineer. I’ll give you the story and the history of DevOps before presenting my definition. If you ever want to start a fight at a conference, though, you can ask the “What is DevOps?” question to a group of five people, and then walk away and watch the carnage. Luckily, you’re reading this and not talking to me in the hallway, so I don’t mind putting my definition out there and seeing what happens. But first, the story.

1.1.1 A little DevOps history

In 2007, a systems administrator by the name of Patrick Debois was consulting on a large data center migration project for the Belgium government. He was in charge of the testing for this migration, so he spent a fair amount of time working and coordinating with both the development and operations teams. Seeing the stark contrast between how development and operations teams functioned, Debois got frustrated and started thinking of solutions to this problem.

Fast-forward to 2008. Developer Andrew Clay Shafer, attending the Agile Conference in Toronto, proposes an ad hoc discussion session called “Agile Infrastructure.” He received such poor feedback on his proposal that he didn’t even attend the session himself. In fact, only a single attendee joined the session, Patrick Debois. But because Debois was so passionate about discussing this topic, he tracked Shafer down in the hallway, where they had an extensive discussion about their ideas and goals. Directly out of those conversations, they formed the Agile Systems Administrator Group.

In June 2009, Debois was back in Belgium, watching a live stream of the O’Reilly Velocity 09 conference. At this conference, two employees from Flickr, John Allspaw and Paul Hammond, gave a talk titled “10 Deploys per Day: Dev & Ops Cooperation at Flickr.” Debois, moved by the talk, was inspired to start his own conference in Ghent, Belgium. He invited developers and operations professionals to discuss various approaches to working together, managing infrastructure, and rethinking the way the teams worked together. Debois called this two-day conference DevOps Days. A lot of the conversations about the conference were happening on Twitter, which then limited the number of characters per message to 140. To save as many precious characters as possible, Debois shortened the conference’s Twitter hashtag from #devopsdays to just plain #devops, and with that, DevOps was born.

Definition DevOps is a set of software-development practices that combines a software development mentality with other functions in the organization. DevOps puts a heavy emphasis on shared responsibilities across all teams throughout the software development life cycle. The edges of job functions soften, as operations team members take on tasks that were traditionally more developer-focused, and development team members do the same. The term DevOps is most commonly associated with development (Dev) and IT operations (Ops), but the approach can be extended to other groups as well, including but not limited to security (DevSecOps), QA, database operations, and networking.

It’s been more than 10 years since that fateful meeting. Since then, DevOps has moved beyond small web startups and has begun to penetrate larger enterprises. The success of DevOps, however, has brought the most cantankerous enemy of any movement: market forces.

According to LinkedIn Talent Solutions, in 2018 the most recruited job overall, not just in tech, was DevOps engineer. Considering we’ve defined DevOps as a set of practices, it’s strange how a style of work quickly became a job title. You’ve never heard of an Agile engineer, because it just sounds silly. As transformational as DevOps is, it couldn’t escape market forces. With that much demand, the job title of DevOps has led to scores of candidates rebranding themselves as DevOps engineers.

Product marketers are looking to cash in on the DevOps craze. Simple products like metrics and monitoring get rebranded into “DevOps dashboards,” further diluting the meaning of the word. With the market pulling the term “DevOps” in different directions, it has splintered into different meanings for different people. I could spend an entire chapter arguing about what DevOps should and shouldn’t mean; instead, I’ll use the definition that I proposed previously. But if you ever see me at a conference and want to see me go on a tirade, ask me what it’s like being a “DevOps manager.”

1.1.2 What DevOps is not

Ironically it might be easier to define what DevOps is not rather than what it is. Thanks to market forces, these details will probably fall on deaf ears, but since this is my book, I figure I might as well go for it! For starters, it’s not about tools. If you purchased this book hoping to learn about Jenkins or Docker or Kubernetes or AWS, you’re going to be sorely disappointed. I don’t do refunds, but you can feel free to scream into the ether with your disdain.

DevOps isn’t about tools, but about how teams work together. Technology is definitely involved, but, honestly, the tools are less important than the people. You can install the latest version of Jenkins or sign up for CircleCI, but if you don’t have a solid test suite, it’s useless. If you don’t have a culture that considers automated testing valuable, the tool doesn’t provide value. DevOps is about people first, then process, then tools.

You need the people on-board and ready for change. Once the people are on-board, they need to be involved and engaged with creating the process. Once a process is created, you now have the necessary input to pick the right tool!

So many people focus on the tool first and try to work backward from there. This is probably one of the top DevOps follies. You can’t choose a tool and then tell the people that they have to change all their processes. Our brains are wired to immediately be hostile to that type of approach. When tools are launched like that, the tool feels like it’s happening to them, not through them. That approach differs significantly from the way people accept new ideas. You have to have buy-in.

In addition, when you get excited about a new tool, you begin applying it to problems you never had. When you buy a new table saw, suddenly everything in your home becomes a construction project. It’s the same thing with software tools.

All this is to say that the major focus of this book and DevOps is about people and their interactions. While I may reference specific tools here and there, the book avoids giving specific examples based on architecture. Instead, the examples focus on capabilities, regardless of which tool provides that capability. To highlight this approach, the DevOps philosophy is structured on top of the CAMS model, which aims to place people first when addressing problems.

DevOps as the “new” systems administrator

When I attend technology events, I’m often greeted by someone who believes that the popularity of DevOps means certain doom for the “traditional” systems administrator. With the rise of virtual machines, software-defined networking, and API access for creating infrastructure, it is no surprise that software development skills are becoming increasingly important for systems administrators and, in many companies, is already a strict requirement. This push toward more development-focused systems administrators has led many to speculate that DevOps is the beginning of the end for systems administration.

But the demise of the operations function has been greatly exaggerated. The way operations teams go about their work is definitely in a state of flux, but it has been since about 1960. I agree that developers will take more of a role in operations work, but operations work will continue to be separate and distinct from the work that developers do on a daily basis.

Regardless of who does that work, tasks like infrastructure architecture planning, capacity planning, operating the system at runtime, monitoring, implementing patches, overseeing security, developing internal tools, and managing the platform will continue to exist. Operations engineering will continue to be a specialized form of engineering. There is no doubt that system administrators have a new set of skills that they’ll...