Information Theory Tools for Visualization

Min Chen, Miquel Feixas, Ivan Viola, Anton Bardera, Han-Wei Shen, Mateu Sbert

• Publisher: A K Peters/CRC Press
• Year: 2016
• Print ISBN: 9781498740937
• eBook ISBN: 9781315352237
• Topic: Computer Science
• Subtopic: Computer Graphics
• Index: Computer Science

This book explores Information theory (IT) tools, which have become state of the art to solve and understand better many of the problems in visualization. This book covers all relevant literature up to date. It is the first book solely devoted to this subject, written by leading experts in the field.

CHAPTER 1

Basic Concepts of Information Theory

CONTENTS

1.1 Entropy

1.2 Relative Entropy and Mutual Information

1.3 Information Specific to a Particular Symbol

1.4 Entropy Rate

1.5 Jensen–Shannon Divergence

1.6 Information Bottleneck Method

1.7 Summary

Information theory was founded in 1948 and described in Shannon’s paper “A Mathematical Theory of Communication” [107]. In this paper, Shannon defined entropy and mutual information (initially called the rate of transmission), and introduced the fundamental laws of data compression and transmission. In information theory, information is simply the outcome of a selection among a finite number of possibilities and an information source is modeled as a random variable or as a random process. While Shannon entropy expresses the uncertainty or the information content of a single random variable, mutual information quantifies the dependence between two random variables and plays an important role in the analysis of a communication channel, a system in which the output depends probabilistically on its input [33, 134, 148]. From its birth to date, information theory has interacted with many different fields, such as statistical inference, computer science, mathematics, physics, chemistry, economics, and biology.

This chapter presents Shannon’s information measures (entropy, conditional entropy, and mutual information) for discrete and continuous random variables, together with Kullback–Leibler distance, entropy rate, some relevant inequalities, and the information bottleneck method. Two main references on information theory are the books by Cover and Thomas [33] and Yeung [148].

1.1 ENTROPY

Let X be a discrete random variable with alphabet $\mathbb{X}$ and probability distribution {p(x)}, where p(x) = Pr{X = x} and x ∈ $\mathbb{X}$. The probability distribution {p(x)} will also be denoted by p(X) or simply p. For instance, a discrete random variable can be used to describe the toss of a fair coin, with alphabet $\mathbb{X}$ = {head, tail} and probability distribution p(X) = {1/2, 1/2}. The entropy H(X) of a discre...

Table of contents

• Cover
• Half Title
• Title Page
• Copyright Page
• Table of Contents
• Foreword
• Preface
• CHAPTER 1 ■ Basic Concepts of Information Theory
• CHAPTER 2 ■ Visualization and Information Theory
• CHAPTER 3 ■ Viewpoint Metrics and Applications
• CHAPTER 4 ■ Volume Visualization
• CHAPTER 5 ■ Flow Visualization
• CHAPTER 6 ■ Information Visualization
• Bibliography
• Index