Red Hat Enterprise Linux Troubleshooting Guide

Benjamin Cane

• Éditeur: Packt Publishing
• Année: 2015
• ISBN:
  9781785283550
• Édition: 1
• Sujet: Computer Science
• Sous-sujet: Operating Systems

Identify, capture and resolve common issues faced by Red Hat Enterprise Linux administrators using best practices and advanced troubleshooting techniques

About This Book

• Develop a strong understanding of the base tools available within Red Hat Enterprise Linux (RHEL) and how to utilize these tools to troubleshoot and resolve real-world issues
• Gain hidden tips and techniques to help you quickly detect the reason for poor network/storage performance
• Troubleshoot your RHEL to isolate problems using this example-oriented guide full of real-world solutions

Who This Book Is For

If you have a basic knowledge of Linux from administration or consultant experience and wish to add to your Red Hat Enterprise Linux troubleshooting skills, then this book is ideal for you. The ability to navigate and use basic Linux commands is expected.

What You Will Learn

• Identify issues that need rapid resolution against long term root cause analysis
• Discover commands for testing network connectivity such as telnet, netstat, ping, ip and curl
• Spot performance issues with commands such as top, ps, free, iostat, and vmstat
• Use tcpdump for traffic analysis
• Repair a degraded file system and rebuild a software raid
• Identify and troubleshoot hardware issues using dmesg
• Troubleshoot custom applications with strace and knowledge of Linux resource limitations

In Detail

Red Hat Enterprise Linux is an operating system that allows you to modernize your infrastructure, boost efficiency through virtualization, and finally prepare your data center for an open, hybrid cloud IT architecture. It provides the stability to take on today's challenges and the flexibility to adapt to tomorrow's demands.

In this book, you begin with simple troubleshooting best practices and get an overview of the Linux commands used for troubleshooting. The book will cover the troubleshooting methods for web applications and services such as Apache and MySQL. Then, you will learn to identify system performance bottlenecks and troubleshoot network issues; all while learning about vital troubleshooting steps such as understanding the problem statement, establishing a hypothesis, and understanding trial, error, and documentation. Next, the book will show you how to capture and analyze network traffic, use advanced system troubleshooting tools such as strace, tcpdump & dmesg, and discover common issues with system defaults.

Finally, the book will take you through a detailed root cause analysis of an unexpected reboot where you will learn to recover a downed system.

Style and approach

This is an easy-to-follow guide packed with examples of real-world core Linux concepts. All the topics are presented in detail while you're performing the actual troubleshooting steps.

Red Hat Enterprise Linux Troubleshooting Guide

---

Table of Contents

Red Hat Enterprise Linux Troubleshooting Guide

Credits

About the Author

About the Reviewers

www.PacktPub.com

Support files, eBooks, discount offers, and more

Why subscribe?

Free access for Packt account holders

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Downloading the example code

Errata

Piracy

Questions

1. Troubleshooting Best Practices

Styles of troubleshooting

The Data Collector

The Educated Guesser

The Adaptor

Choosing the appropriate style

Troubleshooting steps

Understanding the problem statement

Asking questions

Tickets

Humans

Attempting to duplicate the issue

Running investigatory commands

Establishing a hypothesis

Putting together patterns

Is this something that I've encountered before?

Trial and error

Start by creating a backup

Getting help

Books

Team Wikis or Runbooks

Google

Man pages

Reading a man page

Name

Synopsis

Description

Examples

Additional sections

Info documentation

Referencing more than commands

Installing man pages

Red Hat kernel docs

People

Following up

Documentation

Root cause analysis

The anatomy of a good RCA

The problem as it was reported

The actual root cause of the problem

A timeline of events and actions taken

Any key data points to validate the root cause

A plan of action to prevent the incident from reoccurring

Establishing a root cause

Sometimes you must sacrifice a root cause analysis

Understanding your environment

Summary

2. Troubleshooting Commands and Sources of Useful Information

Finding useful information

Log files

The default location

Common log files

Finding logs that are not in the default location

Checking syslog configuration

Checking the application's configuration

Other examples

Using the find command

Configuration files

Default system configuration directory

Finding configuration files

Using the rpm command

Using the find command

The proc filesystem

Troubleshooting commands

Command-line basics

Command flags

The piping command output

Gathering general information

w – show who is logged on and what they are doing

rpm – RPM package manager

Listing all packages installed

Listing all files deployed by a package

Using package verification

df – report file system space usage

Showing available inodes

free – display memory utilization

What is free, is not always free

The /proc/meminfo file

ps – report a snapshot of current running processes

Printing every process in long format

Printing a specific user's processes

Printing a process by process ID

Printing processes with performance information

Networking

ip – show and manipulate network settings

Show IP address configuration for a specific device

Show routing configuration

Show network statistics for a specified device

netstat – network statistics

Printing network connections

Printing all ports listening for tcp connections

Delay

Performance

iotop – a simple top-like I/O monitor

iostat – report I/O and CPU statistics

Manipulating the output

vmstat – report virtual memory statistics

sar – collect, report, or save system activity information

Using the sar command

Summary

3. Troubleshooting a Web Application

A small back story

The reported issue

Data gathering

Asking questions

Duplicating the issue

Understanding the environment

Where is this blog hosted?

Lookup IPs with nslookup

What about ping, dig, or other tools?

Ok, it's within our environment; now what?

What services are installed and running?

Validate the web server

Validating the database service

Validating PHP

A summary of installed and running services

Looking for error messages

Apache logs

Finding the location of Apache's logs

Reviewing the logs

Using curl to call our web application

Requesting a non-PHP page

Reviewing generated log entries

What we learned from httpd logs

Verifying the database

Verifying the WordPress database

Finding the installation path for WordPress

Checking the default configuration

Finding the database credentials

Connecting as the WordPress user

Validating the database structure

What we learned from the database validation

Establishing a hypothesis

Resolving the issue

Understanding database data files

Finding the MariaDB data folder

Resolving data file issues

Validating

Final validation

Summary

4. Troubleshooting Performance Issues

Performance issues

It's slow

Performance

Application

CPU

Top – a single command to look at everything

What does this output tell us about our issue?

Individual processes from top

Determining the number of CPUs available

Threads and Cores

lscpu – Another way to look at CPU info

ps – Drill down deeper on individual processes with ps

Using ps to determine process CPU utilization

Putting it all together

A quick look with top

Digging deeper with ps

Memory

free – Looking at free and used memory

Linux memory buffers and caches

Swapped memory

What free tells us about our system

Checking for oomkill

ps - Checking individual processes memory utilization

vmstat – Monitoring memory allocation and swapping

Putting it all together

Taking a look at the system's memory utilization with free

Watch what is happening with vmstat

Finding the processes that utilize the most memory with ps

Disk

iostat – CPU and device input/output statistics

CPU details

Reviewing I/O statistics

Identifying devices

Who is writing to these devices?

ps – Using ps to identify processes utilizing I/O

iotop – A top top-like command for disk i/o

Putting it all together

Using iostat to determine whether there is a I/O bandwidth problem

Using iotop to determine which processes are consuming disk bandwidth

Using ps to understand more about processes

Networ...