Hands-On System Programming with Go

Build modern and concurrent applications for Unix and Linux systems using Golang

Alex Guerrieri

• Publisher: Packt Publishing
• Year: 2019
• ISBN:
  9781789803365
• Edition: 1
• Topic: Computer Science
• Subtopic: Computer Science General
• Index: Computer Science

Explore the fundamentals of systems programming starting from kernel API and filesystem to network programming and process communications

Key Features

• Learn how to write Unix and Linux system code in Golang v1.12
• Perform inter-process communication using pipes, message queues, shared memory, and semaphores
• Explore modern Go features such as goroutines and channels that facilitate systems programming

Book Description

System software and applications were largely created using low-level languages such as C or C++. Go is a modern language that combines simplicity, concurrency, and performance, making it a good alternative for building system applications for Linux and macOS.

This Go book introduces Unix and systems programming to help you understand the components the OS has to offer, ranging from the kernel API to the filesystem, and familiarize yourself with Go and its specifications. You'll also learn how to optimize input and output operations with files and streams of data, which are useful tools in building pseudo terminal applications. You'll gain insights into how processes communicate with each other, and learn about processes and daemon control using signals, pipes, and exit codes. This book will also enable you to understand how to use network communication using various protocols, including TCP and HTTP.

As you advance, you'll focus on Go's best feature-concurrency helping you handle communication with channels and goroutines, other concurrency tools to synchronize shared resources, and the context package to write elegant applications.

By the end of this book, you will have learned how to build concurrent system applications using Go

What you will learn

• Explore concepts of system programming using Go and concurrency
• Gain insights into Golang's internals, memory models and allocation
• Familiarize yourself with the filesystem and IO streams in general
• Handle and control processes and daemons' lifetime via signals and pipes
• Communicate with other applications effectively using a network
• Use various encoding formats to serialize complex data structures
• Become well-versed in concurrency with channels, goroutines, and sync
• Use concurrency patterns to build robust and performant system applications

Who this book is for

If you are a developer who wants to learn system programming with Go, this book is for you. Although no knowledge of Unix and Linux system programming is necessary, intermediate knowledge of Go will help you understand the concepts covered in the book

Section 1: An Introduction to System Programming and Go

This section is an introduction to Unix and system programming. It will help you understand what components Unix operating systems have to offer, from the kernel API to the filesystem, and you will become familiar with system programming's basic concepts.

This section consists of the following chapters:

• Chapter 1, An Introduction to System Programming
• Chapter 2, Unix OS Components
• Chapter 3, An Overview of Go

An Introduction to System Programming

This chapter is an introduction to system programming, exploring a range of topics from its original definition to how it has shifted in time with system evolution. This chapter provides some basic concepts and an overview of Unix and its resources, including the kernel and the application programming interfaces (API). Many of these concepts are defined here and are used in the rest of the book.

The following topics will be covered in this chapter:

• What is system programming?
• Application programming interfaces
• Understanding how the protection ring works
• An overview of system calls
• The POSIX standard

Technical requirements

This chapter does not require you to install any special software if you're on Linux.

If you are a Windows user, you can install the Windows Subsystem for Linux (WSL). Follow these steps in order to install WSL:

1. Open PowerShell as administrator and run the following:

Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Windows-Subsystem-Linux

2. Restart your computer when prompted.
3. Install your favorite Linux distribution from the Microsoft Store.

Beginning with system programming

Over the years, the IT landscape has shifted dramatically. Multicore CPUs that challenge the Von Neumann machine, the internet, and distributed systems are just some of the changes that occurred in the last 30 years. So, where does system programming stand in this landscape?

Software for software

Let's start with the standard textbook definition first.

System programming (or systems programming) is the activity of programming computer system software. The primary distinguishing characteristic of system programming when compared to application programming is that application programming aims to produce software that provides services directly to the user (for example, a word processor), whereas system programming aims to produce software and software platforms that provide services to other software, and are designed to work in performance-constrained environments, for example, operating systems, computational science applications, game engines and AAA video games, industrial automation, and software as a service applications.

The definition highlights two main concepts of what system applications are as follows:

• Software that is used by other software, not directly by the final user.
• The software is hardware aware (it knows how the hardware works), and is oriented toward performance.

This makes it possible to easily recognize as system software operating system kernels, hardware drivers, compilers, and debuggers, and not as system software, a chat client, or a word processor.

Historically, system programs were created using Assembly and C. Then came the shells and the scripting languages that were used to tie together the functionality offered by system programs. Another characteristic of system languages was the control of the memory allocation.

Languages and system evolution

In the last decade, scripting languages gained popularity to the point at which some had significant performance improvement and entire systems were built with them. For example, let's just think about the V8 Engine for JavaScript and the PyPy implementation of Python, which dramatically shifted the performance of these languages.

Other languages, such as Go, proved that garbage collection and performance are not mutually exclusive. In particular, Go managed to replace its own memory allocator written in C with a native version written in Go in release 1.5, improving it to the point where the performance was comparable.

In the meantime, systems started to get distributed and the applications started to get shipped in containers, orchestrated by other system software, such as Kubernetes. These systems are meant to sustain huge throughput and achieve it in two main ways:

• By scaling—augmenting the number or the resources of the machines that are hosting the system
• By optimizing the software in order to be more resource effective

System programming and software engineering

Some of the practices of system programming—such as having an application that is tied to the hardware, performance oriented, and working on an environment that is resource constrained—are an approach that can also be valid when building distributed systems, where constraining resource usage allows the reduction of the number of instances needed. It looks like system programming is a good way of addressing generic software engineering problems.

This means that learning the concept of system programming with regards to using the resource of the machine efficiently—from memory usage to filesystem access—will be useful when building any type of application.

Application programming interfaces

APIs are series subroutine definitions, communication protocols, and tools for building software. The most important aspects of an API are the functionalities that it offers, combined with its documentation, which facilitates the user in the usage and implementation of the software itself in another software. An API can be the interface that allows an application software to use a system software.

An API usually has a specific release policy that is meant to be used by a specific group of recipients. This can be the following:

• Private and for internal use only
• Partner and usable by determined groups only—this may include companies that want to integrate the service with theirs
• Public and available for every user

Types of APIs

We'll see that there are several types of APIs, from the ones used to make different application software work together, to the inner ones exposed by the operating system to other software.

Operating systems

An...