Introduction to entropy: Difference between revisions

Revision as of 12:11, 3 March 2025

Revision as of 04:01, 23 March 2025

Line 46:

It turns out that if you flip a large number of coins, the macrostates at or near half heads and half tails accounts for almost all of the microstates. In other words, for a million coins, you can be fairly sure that about half will be heads and half tails. The macrostates around a 50–50 ratio of heads to tails will be the "equilibrium" macrostate. A real physical system in equilibrium has a huge number of possible microstates and almost all of them are the equilibrium macrostate, and that is the macrostate you will almost certainly see if you wait long enough. In the coin example, if you start out with a very unlikely macrostate (like all heads, for example with zero entropy) and begin flipping one coin at a time, the entropy of the macrostate will start increasing, just as thermodynamic entropy does, and after a while, the coins will most likely be at or near that 50–50 macrostate, which has the greatest information entropy – the equilibrium entropy.

The macrostate of a system is what we know about the system, for example the [[temperature]], [[pressure]], and [[volume (thermodynamics)|volume]] of a gas in a box. For each set of values of temperature, pressure, and volume there are many arrangements of molecules which result in those values. The number of arrangements of molecules which could result in the same values for temperature, pressure and volume is the number of microstates.

The concept of information entropy has been developed to describe any of several phenomena, depending on the field and the context in which it is being used. When it is applied to the problem of a large number of interacting particles, along with some other constraints, like the conservation of energy, and the assumption that all microstates are equally likely, the resultant theory of statistical mechanics is extremely successful in explaining the [[laws of thermodynamics]].

Line 54:

== Example of increasing entropy ==

{{Main article|Disgregation}}

Ice melting provides an example in which entropy increases in a small system, a thermodynamic system consisting of the surroundings (the warm room) and the entity of glass container, ice and water which has been allowed to reach [[thermodynamic equilibrium]] at the melting temperature of ice. In this system, some [[heat]] (''δQ'') from the warmer surroundings at {{Convert|298|K|C F|lk=on}} transfers to the cooler system of ice and water at its constant temperature (''T'') of {{Convert|273|K|C F}}, the melting temperature of ice. The entropy of the system, which is {{sfrac|δ''Q''|''T''}}, increases by {{sfrac|δ''Q''|273 K}}. The heat δ''Q'' for this process is the energy required to change water from the solid state to the liquid state, and is called the [[enthalpy of fusion]], i.e. Δ''H'' for ice fusion.

The entropy of the surrounding room decreases less than the entropy of the ice and water increases: the room temperature of {{val|298|u=K}} is larger than {{val|273|u=K}} and therefore the ratio, (entropy change), of {{sfrac|δ''Q''|298 K}} for the surroundings is smaller than the ratio (entropy change), of {{sfrac|δ''Q''|273 K}} for the ice and water system. This is always true in spontaneous events in a thermodynamic system and it shows the predictive importance of entropy: the final net entropy after such an event is always greater than was the initial entropy.

Line 63:

Originally, entropy was named to describe the "waste heat", or more accurately, energy loss, from heat engines and other mechanical devices which could never run with 100% efficiency in converting energy into work. Later, the term came to acquire several additional descriptions, as more was understood about the behavior of molecules on the microscopic level. In the late 19th century, the word "disorder" was used by [[Ludwig Boltzmann]] in developing [[Entropy (statistical views)|statistical views of entropy]] using [[probability theory]] to describe the increased molecular movement on the microscopic level. That was before quantum behavior came to be better understood by [[Werner Heisenberg]] and those who followed. Descriptions of thermodynamic (heat) entropy on the microscopic level are found in statistical thermodynamics and [[statistical mechanics]].

For most of the [[20th century]], textbooks tended to describe entropy as "disorder", following Boltzmann's early conceptualisation of the [[kinetic energy|"motional" (i.e. kinetic) energy]] of molecules. More recently, there has been a trend in chemistry and physics textbooks to describe [[Entropy (energy dispersal)|entropy as energy dispersal]].<ref name=Lambert>[http://franklambert.net/entropysite.com/ Entropy Sites — A Guide] Content selected by [[Frank L. Lambert]]</ref> Entropy can also involve the dispersal of particles, which are themselves energetic. Thus there are instances where both particles and energy disperse at different rates when substances are mixed together.

The mathematics developed in statistical thermodynamics were found to be applicable in other disciplines. In particular, information sciences developed the concept of [[information entropy]], which lacks the Boltzmann constant inherent in thermodynamic entropy.

Line 89:

* [[Entropy (order and disorder)|'''As a measure of disorder''']]: Traditionally, 20th century textbooks have introduced [[Entropy (order and disorder)|entropy as order and disorder]] so that it provides "a measurement of the disorder or randomness of a system". It has been argued that ambiguities in, and arbitrary interpretations of, the terms used (such as "disorder" and "chaos") contribute to widespread confusion and can hinder comprehension of entropy for most students. On the other hand, in a convenient though arbitrary interpretation, "disorder" may be sharply defined as the [[Entropy (information theory)|Shannon entropy]] of the probability distribution of microstates given a particular macrostate,<ref name="Callen1985">{{cite book|title=Thermodynamics and an Introduction to Thermostatistics|last=Callen|first=Herbert B.|date=1985|publisher=John Wiley & Sons|isbn=0-471-86256-8|edition=2nd|___location=New York|author-link=Herbert Callen}}</ref>{{rp|379}} in which case the [[Entropy in thermodynamics and information theory|connection of "disorder" to thermodynamic entropy]] is straightforward, but arbitrary and not immediately obvious to anyone unfamiliar with information theory.

* '''Missing information''': The idea that information entropy is a measure of how much one does not know about a system is quite useful.

: If, instead of using the natural logarithm to define information entropy, we instead use the base 2 logarithm, then the information entropy is roughly equal to the average number of (carefully chosen <ref name="questions">The minimum number of questions is achieved when each question either gives an answer with certainty, or cuts the remaining uncertainty in half. For example, it we had a probability function <math>P_i = (1/8,1/2,1/8,1/4)</math> associated with a variable <math>x=(x_1,x_2,x_3,x_4)</math>, then the optimum mode of questioning would be to first ask "is x equal to x₂?" If the answer is "yes", then ''x'' is certainly equal to x₂ after asking only one question, and the probability of this happening is ''P''₂ = 1/2. If the answer is "no", then the next question would be "Is ''x'' equal to ''x''₄? If the answer is yes, then ''x'' is certainly equal to ''x''₄ after asking two questions, and the probability of this happening is ''P''₄ = 1/4. If the answer is "no", we may finally ask "is ''x'' equal to ''x''₁?. If the answer is yes, then the ''x'' is certainly equal to ''x''₁ and if not, then ''x'' is certainly equal to ''x''₃, and the probability of requiring three questions is ''P''₁ + ''P''₃ = 1/4. The average number of binary questions asked is then ''Q'' = (1/2)(1)+(1/4)(2)+(1/4)(3) = 7/4. Calculating the Shannon information entropy:

: <math>Q=-\sum_{i=1}^4 P_i \log_2(P_i) = 7/4</math> Sh

which is in agreement with the step-by-step procedure. In most cases, it is not clear how to continually divide the remaining options in half with each question so the concept is strictly applicable only for special cases, and becomes more accurate as the number of possible outcomes increases. Nevertheless, the Shannon expression for ''Q'' is valid even in these cases.</ref>) yes/no questions that would have to be asked to get complete information about the system under study. In the introductory example of two flipped coins, the information entropy for the macrostate which contains one head and one tail, one would only need one question to determine its exact state, (e.g. is the first one heads?") and instead of expressing the entropy as ln(2) one could say, equivalently, that it is log₂(2) which equals the number of binary questions we would need to ask: One. When measuring entropy using the natural logarithm (ln), the unit of information entropy is called a "nat", but when it is measured using the base-2 logarithm, the unit of information entropy is called a "shannon" (alternatively, "bit"). This is just a difference in units, much like the difference between inches and centimeters. (1&nbsp;nat = log₂''e''&nbsp;shannons). Thermodynamic entropy is equal to the Boltzmann constant times the information entropy expressed in nats. The information entropy expressed with the unit [[shannon (unit)|shannon]] (Sh) is equal to the number of yes–no questions that need to be answered in order to determine the microstate from the macrostate.

: The concepts of "disorder" and "spreading" can be analyzed with this information entropy concept in mind. For example, if we take a new deck of cards out of the box, it is arranged in "perfect order" (spades, hearts, diamonds, clubs, each suit beginning with the ace and ending with the king), we may say that we then have an "ordered" deck with an information entropy of zero. If we thoroughly shuffle the deck, the information entropy will be about 225.6&nbsp;shannons: We will need to ask about 225.6 questions, on average, to determine the exact order of the shuffled deck. We can also say that the shuffled deck has become completely "disordered" or that the ordered cards have been "spread" throughout the deck. But information entropy does not say that the deck needs to be ordered in any particular way. If we take our shuffled deck and write down the names of the cards, in order, then the information entropy becomes zero. If we again shuffle the deck, the information entropy would again be about 225.6&nbsp;shannons, even if by some miracle it reshuffled to the same order as when it came out of the box, because even if it did, we would not know that. So the concept of "disorder" is useful if, by order, we mean maximal knowledge and by disorder we mean maximal lack of knowledge. The "spreading" concept is useful because it gives a feeling to what happens to the cards when they are shuffled. The probability of a card being in a particular place in an ordered deck is either 0 or 1, in a shuffled deck it is 1/52. The probability has "spread out" over the entire deck. Analogously, in a physical system, entropy is generally associated with a "spreading out" of mass or energy.

: The connection between thermodynamic entropy and information entropy is given by Boltzmann's equation, which says that {{nowrap|1=''S'' = ''k''_B ln ''W''}}. If we take the base-2 logarithm of ''W'', it will yield the average number of questions we must ask about the microstate of the physical system in order to determine its macrostate.<ref>In classical mechanics the velocities and positions are real numbers, and there is a [[Continuum (measurement)|continuum]] of an infinite number of microstates. This would mean that an infinite number of questions would have to be asked in order to determine a macrostate. In quantum mechanics, the microstates are "quantized" and there are a finite number of them for a given energy, so the number of questions is finite. Boltzmann developed his theory before the advent of quantum mechanics, and it is to his credit that he was nevertheless able to develop a theory that dealt with a theoretically infinite number of microstates.</ref>

== See also ==