Maximum Entropy

7 Key excerpts on "Maximum Entropy"

• 

  eBook - PDF

  Reliable Plan Selection By Intelligent Machines

  • John E Mclnroy, Joseph C Musto, George N Saridis(Authors)
  • 1996(Publication Date)
  • World Scientific

    (Publisher)

  3.2 Jaynes' Principle of Maximum Entropy In an attempt to generalize the principle used by Boltzmann and Shannon to describe the uncertainty of the performance of a system under a certain operating condition, Jaynes [26] formulated his Maximum Entropy Principle, to apply it in theoretical mechanics. In summary it claims that The uncertainty of an unspecified relation of the function of a system is expressed by an exponential density function of a known energy relation associated with the system. A modified version of the principle, as it applies to the control problem, will be applied in the sequel, using the calculus of variations [28]. The derivation represents a new formulation of the control problem, either for deterministic or stochastic systems and for optimal or non-optimal solutions. 26 CHAPTER 3. A REVIEW OF ENTROPY METHODS 3.3 Entropy in Intelligent Control Uncertainty has been associated with insufficient knowledge and lack of information in thermodynamics and information theory. The models used were probabilistic. As such, one may distinguish two categories: • Subjective (reducible) uncertainties • Objective (irreducible) uncertainties The first category refers to the lack of knowledge of the experimenter, while the latter refers to phenomena from which no further improvement of knowledge is attainable [29]. Intelligent control is a means of realization of Intelligent Machines. The problem of control design will be associated with subjective uncertainty. The designer is faced with the uncertainty of optimal design before solving the problem. He may assign an appropriate probability density function over the space of admissible controls, and reduce the design problem into the problem of finding the point (control law), that has maximum probability of attaining the optimal value. This approach will be pursued in the sequel with entropy, the measure of uncertainty.

  Sign up to read

  Learn more about book
• 

  eBook - PDF

  Image Recovery: Theory and Application

  • Henry Stark(Author)
  • 2013(Publication Date)
  • Academic Press

    (Publisher)

  CHAPTER 5 The Principle of Maximum Entropy in Image Recovery Xinhua Zhuang* Einar 0stevold Coordinated Science Laboratory Norwegian Defense Research University of Illinois at Establishment Champaign-Urbana N-2007 Kjeller, Norway Urbana, Illinois 61801 Robert M. Haralick Machine Vision International Ann Arbor, Michigan 48104 5.1 INTRODUCTION For many decades it has been recognized, or conjectured, that the notion of entropy defines a kind of measure on the space of probability distribu-tions, such that those of high entropy are in some sense favored over others, all other things being equal. The basis for this was stated first in a variety of intuitive forms: that distributions of higher entropy represent more disorder, that they are smoother, more probable, less predictable, that they assume less and hence are more natural according to Shan-non's interpretation of entropy as an information measure, etc. While each of these points of view doubtless reflects an element of truth, none seems explicit enough to lend itself to a quantitative demonstration of the kind we are accustomed to having in other fields of applied mathe-matics. Accordingly, many approaching this field are disconcerted by what they sense as a kind of vagueness, the underlying theory lacking a solid content. * Visiting Professor, People's Republic of China. 157 Copyright 1987 by Academic Press, Inc All rights of reproduction in any form reserved 158 Xinhua Zhuang et a. This has not prevented the fruitful exploitation of this property of entropy. The Maximum Entropy (ME) principle, briefly speaking, is: when we make inferences based on incomplete information, we should draw them from that probability distribution that has the Maximum Entropy allowed by the information we do have. Jaynes [1,2] has been a foremost proponent of Maximum Entropy prior distributions consistent with known information.

  Sign up to read

  Learn more about book
• 

  eBook - ePub

  The Handbook of Computational Linguistics and Natural Language Processing

  • Alexander Clark, Chris Fox, Shalom Lappin, Alexander Clark, Chris Fox, Shalom Lappin(Authors)
  • 2013(Publication Date)
  • Wiley-Blackwell

    (Publisher)

  Current Methods Passage contains an image

  5  Maximum Entropy Models

  ROBERT MALOUF

  1 Introduction

  Maximum Entropy (MaxEnt) models, variously known as log-linear, Gibbs, exponential, and multinomial logit models, provide a general-purpose machine learning technique for classification and prediction which has been successfully applied to fields as diverse as computer vision and econometrics. In natural language processing, recent years have seen MaxEnt techniques used for sentence boundary detection, part-of-speech tagging, parse selection and ambiguity resolution, machine translation, and stochastic attribute value grammars, to name just a few applications (Berger et al., 1996; Abney 1997; Ratnaparkhi 1998; Johnson et al., 1999; Foster 2000). Beyond these purely practical applications, statistical modeling techniques also offer a powerful set of tools for analyzing natural language data. A good statistical model can both clarify what the patterns are in a complex, possibly noisy, set of observations and at the same time shed light on the underlying processes that lead to those patterns (Breiman 2001b; McCullagh 2002).

  The fundamental problem for stochastic data analysis is model selection. How do we choose a model out of a given hypothesis space which best fits our observations? In all but the most trivial cases, our hypothesis space will provide an infinite range of possible models. This is a general problem: how do we pick a probability distribution given possibly incomplete information?

  More technically, suppose we have a random variable X , which can take on values x 1 , . . . ,

  xn

  Sign up to read

  Learn more about book
• 

  eBook - PDF

  Rational Descriptions, Decisions and Designs

  Pergamon Unified Engineering Series

  • Myron Tribus, Thomas F. Irvine, James P. Hartnett(Authors)
  • 2013(Publication Date)
  • Pergamon

    (Publisher)

  Chapter Five The Principle of Maximum Entropy IN PREVIOUS chapters we have considered the use of Bayes' equation to modify a given probability distribution. If the prior assignment of probability is p(A f |X) and some new data, D, become available, the probabilities are adjusted according to r ( A l D X ) - * ,| ) ( | , ) I in which the denominator has been extended according to eqn. III-3. Equation V-1 is mathematically correct and there has never been any serious objection to it. But since the term on the right depends upon the assignment of ( ,·| ), there has been much controversy over the question of how to assign prior probabilities or even if the equation should be used at all. From the point of view of the development in the previous chapters, the problem is simply one of deciding how to encode prior information. Alas, the problem is not that simple. It is indeed difficult to answer someone who asks, Tell me, what do you know? On the other hand there are often times when we can be fairly explicit about what we know in regard to a specific question. But this knowledge is often incomplete and must be encoded in a probability distribution before we can make use of it in our inferential procedures. This chapter will be concerned with a principle to be used in assigning prior probabilities. It is to be looked upon as a way of getting started, i.e., of encoding one's knowledge from the beginning. In the previous chapter the function called entropy, defined by S= -Eftlnp,. V-2 was shown to be a measure of the uncertainty of the knowledge about the answer to a well defined question. It was also shown to be a measure of how strong a hypothesis is compared to any possible competitor. (Actually, —5 measures the strength, S measures the weakness). 119 RATIONAL DESCRIPTIONS, DECISIONS AND DESIGNS In view of this meaning for entropy, we adopt as a logical principle, the following statement, originally put forward by E.

  Sign up to read

  Learn more about book
• 

  No longer available |Learn more

  Entropy Physics and its Applications

  • (Author)
  • 2014(Publication Date)
  • Learning Press

    (Publisher)

  ___________________________ WORLD TECHNOLOGIES ___________________________ Chapter- 6 Applications and Concepts of Entropy in Information Theory Entropy (information theory) In information theory, entropy is a measure of the uncertainty associated with a random variable. The term by itself in this context usually refers to the Shannon entropy , which quantifies, in the sense of an expected value, the information con-tained in a message, usually in units such as bits. Equivalently, the Shannon entropy is a measure of the average information content one is missing when one does not know the value of the random variable. The concept was introduced by Claude E. Shannon in his 1948 paper A Mathematical Theory of Communication. Shannon's entropy represents an absolute limit on the best possible lossless com-pression of any communication, under certain constraints: treating messages to be encoded as a sequence of independent and identically-distributed random variables, Shannon's source coding theorem shows that, in the limit, the average length of the shortest possible representation to encode the messages in a given alphabet is their entropy divided by the logarithm of the number of symbols in the target alphabet. A fair coin has an entropy of one bit. However, if the coin is not fair, then the uncer-tainty is lower (if asked to bet on the next outcome, we would bet preferentially on the most frequent result), and thus the Shannon entropy is lower. Mathematically, a coin flip is an example of a Bernoulli trial, and its entropy is given by the binary entropy function. A long string of repeating characters has an entropy rate of 0, since every character is predictable. The entropy rate of English text is between 1.0 and 1.5 bits per letter, or as low as 0.6 to 1.3 bits per letter, according to estimates by Shannon based on human experiments. Layman's terms Shannon entropy is a measure of disorder, or more precisely unpredictability.

  Sign up to read

  Learn more about book
• 

  eBook - ePub

  Cybernetical Intelligence

  Engineering Cybernetics with Machine Intelligence

  • Kelvin K. L. Wong(Author)
  • 2023(Publication Date)
  • Wiley-IEEE Press

    (Publisher)

  Overall, relative entropy and mutual information are both important concepts in information theory, providing a way to measure the difference between probability distributions and quantify the amount of information shared between random variables. The relationship between the two concepts highlights the fundamental connections between information theory and probability theory, and their applications in fields such as machine learning, data compression, and signal processing.

  13.3 Entropy in Performance Evaluation

  Entropy can be used as a metric to evaluate the performance of a classification model. It measures the degree of uncertainty or disorder in the system, and in the context of classification, it represents the uncertainty in the classification decision. Consider there is a classification model that assigns a label y to an input x. One also have a true label t for each input x. If one consider the set of all possible labels Y, the entropy of the classification model is defined as:

  (13.4)

  where py is the proportion of inputs that are classified as label y by the model, and log2 is the base‐2 logarithm.

  In this equation, one can see that the entropy H is a function of the distribution of the labels assigned by the model. If the model assigns the correct label to all inputs, the entropy will be 0, indicating that there is no uncertainty in the classification decision. On the other hand, if the model assigns different labels to different inputs with equal probability, the entropy will be maximal, indicating maximum uncertainty. To evaluate the performance of a classification model, one can use the concept of cross‐entropy. Cross‐entropy measures the difference between the predicted distribution and the true distribution of the labels. Assume to represent the true distribution of the labels as p′(y). The cross‐entropy H(p′, p) between the predicted distribution p(y) and the true distribution p′(y) is defined as:

  (13.5)

  In this equation, one can see that the cross‐entropy is a function of both the predicted distribution and the true distribution. If the predicted distribution is identical to the true distribution, the cross‐entropy will be 0, indicating that the model is perfectly accurate. On the other hand, if the predicted distribution is different from the true distribution, the cross‐entropy will be positive, indicating that the model is less accurate.

  Sign up to read

  Learn more about book
• 

  No longer available |Learn more

  Mathematical Modeling

  • J. N. Kapur(Author)
  • 2023(Publication Date)
  • Mercury Learning and Information

    (Publisher)

  n when the only knowledge available about the system is the value of the average energy of the system. According to the principle of maximum-entropy, to get the most unbiased estimates of probabilities, we maximize the entropy

  subject to Using Lagrange’s method, this gives

  where the Lagrange’s multiplier μ is determined by using Eqn. (83 ) so that

  The probability distribution in Eqn. (84 ) is known as the Maxwell-Boltzmann distribution. From Eqn. (85 )

  By using the Cauchy-Schwarz inequality , it is easily seen that the numerator of the RHS of Eqn. (86 ) ≤ 0 so that

  Thus μ is a monotonic decreasing function of the average energy and if we put

  then T is a monotonic increasing function of . We define T as the thermodynamic temperature of the system.

  Substituting from Eqn. (84 ) in Eqn. (82 ), we get the value S max of the Maximum Entropy as

  so that

  on making use of Eqns. (83 ) and (85 ). Again from Eqn. (83 )

  The first term on the right is due to the change in energies and is called the work effect and is denoted by –ΔW . The second term is due to changes in probabilities of various states and is called the heat effect and is denoted by ΔH , so that

  so that Eqn. (90 ) gives

  S max is defined on the thermodynamic entropy. Thus thermodynamic entropy is the maximum possible information-theoretic entropy of a system having a given average energy.

  Thus our model defines in a very natural manner temperature, work effect, heat effect, and thermodynamic entropy. From Eqn. (92 ) we get

  If ε 1 < ε 2 < … <

  εn

  , then when T → 0, μ → ∞ and from Eqn. (84 ) p 1 = 1 and all other probabilities tend to zero so that all the particles tend to be in the lowest energy state.

  In fact all the four laws of thermodynamics can be obtained by combining the concepts of entropy from information theory and the concept of energy from mechanics.

  10.4.2 Bose-Einstein, Fermi-Dirac, and Intermediate Statistics Distributions

  (a) Bose-Einstein Distribution

  In the last subsection, we assumed the knowledge of only the average energy of the system. Now we assume that we know in addition the expected number of particles in the whole system. Let

  pij

  be the probability of there being j particles in the i

  Sign up to read

  Learn more about book