Entropy: Three Discoveries, One Formula

August 2026

Source chart: @LensScientific on X, 2026-08-05. All three equations on it are correct, including the one detail that is easy to fudge — that Clausius’s heat transfer must be reversible, which the chart marks with a subscript rather than dropping.

The post’s caption says these three equations “belong to different branches of science” yet “describe remarkably similar ideas.” That is careful and correct, and it is also where it stops. This page is about the part it leaves out: why counting microscopic arrangements and measuring uncertainty about a message land on the same functional form. There is a real answer — better than “they happen to look alike,” though weaker than the word proof would suggest. Along the way it turns out that two of the three equations were not written by the men they are named after.

The three statements

Clausius (1865) — thermodynamic. Entropy defined by heat flow, with no mention of atoms or probability:

$$dS = \frac{\delta Q_{\text{rev}}}{T}$$

The subscript is load-bearing. The relation holds for a reversible transfer; more generally $dS \ge \delta Q/T_b$, where $T_b$ is the temperature at the boundary the heat crosses, with equality only for a reversible transfer.

Boltzmann (1877) — statistical. Entropy as a count of the microscopic arrangements consistent with what you can actually measure:

$$S = k_B \ln \Omega$$

A note on credit, since this page is partly about priority. Boltzmann’s 1877 paper established that entropy goes as the logarithm of a probability. He never wrote the equation in this form and never introduced the constant — Planck did, around 1900–01, and named it after him. The inscription on Boltzmann’s grave in Vienna was added after his death.

Shannon (1948) — informational. Entropy as the average uncertainty in a probability distribution:

$$H = -\sum_i p_i \log p_i$$

Why the same functional form? Shannon’s uniqueness theorem

The real answer is not that one formula resembles another. It is that almost nothing else satisfies three very ordinary requirements.

Shannon asked what properties a sensible measure of uncertainty must have, and required only three: it should vary continuously with the probabilities; for equally likely outcomes it should increase with the number of outcomes; and — the one doing the real work — if a choice is broken into two successive choices, the total should be the weighted sum of the individual ones. He then proved that these conditions pin the form down uniquely, up to a positive constant $K$, to $H = -K\sum p_i \log p_i$.

Anyone insisting on those three requirements ends up at this expression. That is a real constraint, not a coincidence of notation — and it is worth knowing that the third condition is exactly what other measures drop: Renyi and Tsallis entropies satisfy the first two and abandon the grouping rule, which is how they escape Shannon’s form.

Shannon himself declined to lean on this, and it would be dishonest to lean on it harder than he did. Immediately after the theorem he writes that it “and the assumptions required for its proof, are in no way necessary for the present theory,” that it is given “chiefly to lend a certain plausibility” to the definitions, and that the real justification “will reside in their implications.” So take it as a strong hint that the form is natural rather than a proof that it was inevitable — which is still a great deal more than coincidence.

The correspondence, worked out

With that in hand, the connection to Boltzmann is two lines. Take Shannon’s expression and set every one of the $\Omega$ microstates equally likely, $p_i = 1/\Omega$:

$$H = -\sum_{i=1}^{\Omega} \frac{1}{\Omega} \log \frac{1}{\Omega} = -\log \frac{1}{\Omega} = \log \Omega$$

Multiply by $k_B$, use natural logs, and you have $S = k_B \ln \Omega$ exactly. Shannon’s expression reduces to Boltzmann’s in the equal-probability (microcanonical) case.

The log base is a pure unit conversion — base 2 gives bits, natural log gives nats. So is $k_B$, strictly: since the 2019 SI redefinition it is exact by definition, and you could measure temperature in energy units and set it to 1 without losing anything. The substantive physics is not the constant but the identification it enables — that thermodynamic temperature is related to the microstate count by $1/T \equiv (\partial S/\partial E)_{V,N}$. That identification is the substantive step — the one that would make the two entropies the same quantity rather than the same algebra. The next section is where it gets made, and where its limits show.

And the general form predates Shannon. Gibbs introduced the ensemble entropy

$$S = -k_B \sum_i p_i \ln p_i$$

in Elementary Principles in Statistical Mechanics (1902) — Shannon’s expression with a constant in front, published 46 years earlier, and 14 years before Shannon was born. (Gibbs worked with a continuous phase-space density and what he called the index of probability, defining entropy as its average taken negatively. The discrete sum, and the constant in front, are both later renderings: Gibbs did not use $k_B$ — Planck had introduced it only a year or two earlier and it had not yet propagated into statistical mechanics — and worked instead in units where entropy came out dimensionless.) Shannon arrived at the same structure independently, from a completely different problem.

Closing the loop back to Clausius

That leaves the oldest of the three. The bridge is the definition of temperature: from $S = k_B\ln\Omega$, define

$$\frac{1}{T} \equiv \left(\frac{\partial S}{\partial E}\right)_{V,N}$$

Add a small amount of reversible heat $\delta Q_{\text{rev}}$ to a system at fixed volume and the energy rises by that amount, so $dS = \partial S/\partial E \cdot \delta Q_{\text{rev}} = \delta Q_{\text{rev}}/T$ — Clausius’s relation, recovered from the microstate count. Three honest caveats, because this sketch is weaker than it looks. It is a definition of statistical temperature that then turns out to reproduce the thermodynamic one, and making that identification airtight occupied physicists for decades. It only covers the fixed-volume case — reaching the general $dE = T,dS - p,dV$ needs pressure identified the same way, as $p = T(\partial S/\partial V)_E$. And it needs more than “fixed volume”: it needs no work of any kind done on the system, dissipative work included. Stir a fixed-volume fluid with a paddle and the energy rises with $\delta Q = 0$, so $dS = dE/T$ still holds but $dS = \delta Q/T$ does not. That is also where reversibility re-enters — it is what makes the general relation an equality rather than the inequality above, and the reason this sketch doesn’t seem to need it is that the distinction lives in $T_b$, a fact about the surroundings that a bare microstate count cannot see. This is the shape of the argument, not a proof of it.

What entropy is actually for: choosing the next question

The clearest place this site has found where entropy earns its keep is the puzzle David MacKay builds chapter 4 of his Cambridge information-theory course around — the classic balance-weighing problem — twelve balls, one of them a different weight, three weighings on a balance to find it and say whether it is heavy or light.

MacKay’s principle is to stop hunting for a clever trick and instead pick the weighing whose three outcomes are as close to equally likely as possible — which is the same thing as maximising the entropy of the result. (He sets the entropy calculation as an exercise rather than tabulating it; the table below is ours.) There are 24 equally likely hypotheses (12 balls × heavy-or-light). Weigh $n$ balls against $n$: the scale balances with probability $(24-4n)/24$, and tips each way with probability $2n/24$. Run it for every $n$:

The maximum is 4 against 4, at $\log_2 3 = 1.585$ bits — the ceiling for any three-outcome measurement, reached exactly when all three outcomes are equally likely.

And the puzzle is unforgiving about it, with no slack whatever. Three weighings can distinguish at most $3^3 = 27$ outcomes, or $\log_2 27 = 4.755$ bits, against the $\log_2 24 = 4.585$ bits this question actually contains. That 0.17-bit gap looks like headroom and is not: for this exact task — name the ball and say whether it is heavy or light, with no known-good reference ball — three weighings top out at exactly 12, and 13 is provably impossible, because the balance branch needs $k \ge 5$ and the tipping branch needs $k \le 4$. (Drop either requirement and 13 becomes reachable, which is why you will see that number quoted elsewhere.) The puzzle sits precisely at the ceiling.

Which is why the first move is forced. Choose 3-v-3 and if it balances, 6 balls remain — 12 live hypotheses — and two weighings can only separate 9. The puzzle becomes unsolvable on the first move. Only 4-v-4 leaves $24/3 = 8$ in every branch, inside the 9 that remain reachable.

That converts entropy from a quantity you calculate about a system into a way of choosing what to do next — take the action whose result you can least predict, because that is the action that tells you the most. One honest caveat: maximising outcome entropy equals maximising expected information gain only because the weighing’s result is a deterministic function of the answer. Once observations get noisy the two come apart. And determinism does not buy global optimality either — building an optimal decision tree is hard in general, and greedy information-gain selection is a very good heuristic rather than a theorem. For this puzzle it does hold — checked exhaustively, every maximum-entropy choice at every step leads to a solution — but that is a verified property of this problem, not something MacKay claims in general.

The same course also derives entropy as the average Shannon information content and makes the point that $H$ is the target a compressor should be aiming at — the floor on compression, in the precise sense that for a long message from an independent, identically distributed source you cannot do better than $H$ bits per symbol on average. It covers Shannon’s noisy-channel theorem too, quoting a capacity of about 0.53 for a channel that flips 10% of bits. That checks out: $1 - H_2(0.1) = 1 - 0.469 = 0.531$.

Why this page exists here

This site’s Formulas section carries the Friedmann equation, the QED anomalous magnetic moment and one-loop beta functions, and until now the site had nothing on entropy — the word appeared in passing in three files and was never explained anywhere.

No PBT connection is claimed. A medium with countable microstates is the kind of object a statistical entropy argument would apply to, and that is an observation rather than a derivation. Nothing in Push-Pressure Theory currently derives any of the relations above, and this page does not suggest otherwise.

Catalog status: Proven Systems

All three formulations are settled, independently verified science. The Clausius relation is classical thermodynamics; the Boltzmann and Gibbs forms are the foundation of statistical mechanics; Shannon’s theorems set the limits that every modern compression scheme and error-correcting code is measured against.

References

1. R. Clausius, Ueber verschiedene für die Anwendung bequeme Formen der Hauptgleichungen der mechanischen Wärmetheorie, Annalen der Physik und Chemie 125, 353–400 (1865) — the paper that coins Entropie. DOI
2. L. Boltzmann, Über die Beziehung zwischen dem zweiten Hauptsatze…und der Wahrscheinlichkeitsrechnung, Wiener Berichte 76, 373–435 (1877).
3. J. W. Gibbs, Elementary Principles in Statistical Mechanics (Scribner, 1902) — full text (PDF), Project Gutenberg — catalogued there as Elementary Principles of Statistical Mechanics. The ensemble entropy and the index of probability.
4. C. E. Shannon, A Mathematical Theory of Communication, Bell System Technical Journal 27, 379–423 and 623–656 (1948) — PDF. The uniqueness theorem is Theorem 2, with the proof in Appendix 2.
5. D. J. C. MacKay, Information Theory, Inference, and Learning Algorithms (Cambridge, 2003) — free full text. Entropy and the weighing puzzle are ch. 4 (exercise 4.1); the noisy-channel capacity of 0.53 is ch. 1, eq. 1.35. Lecture videos.
6. M. Tribus and E. C. McIrvine, Energy and Information, Scientific American 225, no. 3 (Sept 1971), 179–190 — the source of the von Neumann naming anecdote below.
7. Oral History: Claude E. Shannon, interviewed by Robert Price, 28 July 1982 — IEEE History Center. Shannon’s own denial of the von Neumann story.

A footnote on a good story: Shannon is often said to have taken the name “entropy” on von Neumann’s advice, on the grounds that the function already had that name in statistical mechanics and that “no one knows what entropy really is, so in a debate you will always have the advantage.” It reaches us through Tribus recounting a 1961 conversation, published a decade later — not a contemporaneous record. And the record cuts against it: in a 1982 IEEE oral history, asked directly, Shannon said “I’m quite sure that it didn’t happen between von Neumann and me.” His interviewer notes that the word already appears in Shannon’s classified 1945 cryptography report, before any plausible von Neumann conversation — though that detail is the interviewer’s, not Shannon’s. Widely repeated; treat it as a good story that its own protagonist denied.