140B Statistical Physics

Part 1. Statistical Ensembles

Information Theory

Probability Theory

Why Information Theory?

Both humans and artificial intelligence (AI) face a fundamental limitation: our computational power (the ability to process information) is finite.

not with a thirst for exhaustive knowledge,

but with a strategy of selective ignorance.

It teaches us to disregard as many details as possible, concentrating our limited computational power on the most crucial information.

This seemingly passive approach is our most powerful tool to understand many-body systems, allowing us to unearth the patterns and regularities hidden behind the world of uncertainty and randomness.

In some sense, thermodynamics is the dynamics of ignorance (i.e. lack of information, i.e. entropy).

Random Variable

The state of a spin (a classical bit): s=↑,↓.

The state of a classical particle: (x,p), x∈(ℝ^3) - position, p∈(ℝ^3) - momentum.

The state of a quantum harmonic oscillator: n∈ℕ - energy level.

The state of an image: x∈(ℝ^(W×H×3)) - RGB values of each pixel in a W×H image.

Probability

Probability assignment describes our current state of knowledge about the random variable. (It can be subjective.)

The probability assignment will be updated, if our knowledge is changed by observations (providing new evidence).

Prior probability before observation

({{x, head, tail}, {p(x), (1/2), (1/2)}})

Posterior probability after observing head up.

({{x, head, tail}, {p(x), 1, 0}})

Positivity:

Normalization:

Expectation Value

⟨f(x)⟩=𝔼[f(X)]=(∑_(x∈𝒳))f(x) p(x).

Double expectation

𝔼[𝔼[X]]=𝔼[X].

Linearity (for two random variables X and Y and a constant α)

𝔼[X+Y]=𝔼[X]+𝔼[Y],
𝔼[α X]=α 𝔼[X].

More generally, for multiple random variables (Subscript[X, i]) (i=1,2,…) linearly combined together,

𝔼[(∑_i)(Subscript[α, i])(Subscript[X, i])]=(∑_i)(Subscript[α, i])𝔼[(Subscript[X, i])].

Variance

⟨(x^2)⟩ is the second-moment of X

⟨(x^2)⟩=(∑_(x∈𝒳))(x^2) p(x).

Positivity: var x≥0 (because (((X-𝔼[X]))^2)\[VectorGreaterEqual]0).

Information and Entropy

Information

Sample collection (encoding scheme)

DynamicModule[{n=3,w=2,h=3,a=0.22,lt=0.8,op=0.2,cols,pts,graph,adjmat,focus,highlight},
cols=<|9->Red,10->Green,11->Blue|>;
pts=Range[2^n+n];
graph=Graph[pts,Flatten[Table[i\[DirectedEdge]#&/@DeleteCases[IntegerDigits[i-1,2,n](2^n+Range[n]),0],{i,2^n}]],
VertexCoordinates->Join[Table[{x,0},{x,2^n}],Table[{w x+(2^n+1-w(n+1))/2,-h},{x,n}]]];
adjmat=AdjacencyMatrix[graph];
focus[i_]:=Subgraph[graph,Flatten[First/@Most@ArrayRules[(#+adjmat.#+Transpose[adjmat].#)&@SparseArray[{i->1},Length@pts]]]];
highlight=graph;
Deploy@Dynamic@EventHandler[Graph[graph,VertexShapeFunction->Function[{p,name,bb},EventHandler[#,"MouseMoved":>(highlight=focus[name])]&@{Opacity[If[MemberQ[VertexList[highlight],name],1,op]],If[name<=2^n,{{Black,Circle[p,a]},{Black,Text[name,p,{-0.2,0.2}]}},{Translate[{{EdgeForm[None],Lighter[cols[name],lt],Polygon[{{a,a},{-a,a},{-a,-a},{a,-a}}]},{Black,Line[{{a,a},{-a,a},{-a,-a},{a,-a},{a,a}}]}},p]}]}],
EdgeShapeFunction->Function[{ps,e},
Block[{p1,p2,k1,k2},{p1,p2}=ps;
k1=(Position[#,e[[2]]][[1,1]]-(1+Length@#)/2)&@Flatten[First/@Most@ArrayRules[adjmat[[e[[1]]]]]];k2=(Position[#,e[[1]]][[1,1]]-(1+Length@#)/2)&@Flatten[First/@Most@ArrayRules[adjmat[[All,e[[2]]]]]];
{Opacity[If[MemberQ[EdgeList[highlight],e],1,op]],Arrowheads[{{0.04,0.25}}],cols[e[[2]]],Arrow[BSplineCurve[{p1+{0.1k1,-a},p1+{0.1 k1,-h/2},p2+{0.1k2,h/2},p2+{0.1k2,a}}]]}]],ImageSize->220],"MouseClicked":>(highlight=graph)]
]

Test result analysis (decoding scheme)

DynamicModule[{n=3,a=0.15,b=0.15,lt=0.8,op=0.2,pts,layers,cols,graph,adjmat,focus,highlight},
pts=Range[2^(n+1)-1];layers=First/@Rest@FoldList[TakeDrop[Last@#1,2^#2]&,{{},pts},Range[n,0,-1]];
cols=<|1->White,2->Blue,3->Green,4->Red|>;
graph=Graph[pts,Flatten[Table[#2\[DirectedEdge]i,{i,#1}]&@@@Thread@{Partition[#1,2],#2}&@@@Thread@{Most@layers,Rest@layers}]];
adjmat=AdjacencyMatrix[graph];
focus[i_]:=Subgraph[graph,Flatten[First/@Most@ArrayRules@Sum[Total[NestWhileList[#.mat&,SparseArray[{i->1},Length@pts],Total[#]>0&]],{mat,{adjmat,Transpose@adjmat}}]]];
highlight=graph;
Deploy@Dynamic@EventHandler[Graph[graph,VertexShapeFunction->Function[{p,name,bb},EventHandler[#,"MouseMoved":>(highlight=focus[name])]&@{Opacity[If[MemberQ[VertexList[highlight],name],1,op]],If[Position[layers,name][[1,1]]==1,{{Black,Circle[p,a]},{Black,Text[name,p,{-0.2,0.2}]}},{Translate[{{EdgeForm[None],Lighter[cols[Position[layers,name][[1,1]]],lt],Polygon[{{a,b},{-a,b},{-a,-b},{a,-b}}]},{Black,Line[{{a,b},{-a,b},{-a,-b},{a,-b},{a,b}}]}},p],{Black,Text["?",p,{-0.2,0.3}]}}]}],
EdgeShapeFunction->Function[{ps,e},Block[{f},
f=BSplineFunction[ps];
EventHandler[#,"MouseMoved":>(highlight=focus[e[[2]]])]&@
{Opacity[If[MemberQ[EdgeList[highlight],e],1,op]],Black,Arrow[{f[0.2],f[0.8]}],cols[Position[layers,e[[1]]][[1,1]]],Text[If[OddQ[e[[2]]],"-","+"],f[0.5],{-(-1)^e[[2]],-0.5}]}]],ImageSize->220],"MouseClicked":>(highlight=graph)]]

Identify a unique outcome out of (2^n) equally-likely possibilities,

Collapse the probability from p=((((1/2)))^n) to p=1 for the observed outcome,

Provide n bits of information:

p=((((1/2)))^n)=>n=-(Subscript[log, 2])p=-((log p)/(log 2)).

log 2=1bit is treated as an information unit.

({{x, a, b, c, d}, {p(x), (1/2), (1/4), (1/8), (1/8)}}).

Information gain can be different for different observation outcomes

I(a)=-log(1/2)=log 2=1 bit,
I(b)=-log(1/4)=2log 2=2 bit,
I(c)=-log(1/8)=3 log 2=3 bit,
I(d)=-log(1/8)=3 log 2=3 bit.

However, different outcome happens with different probability . What is average amount of information that we can obtain from observing X (regardless of its outcome)?

I(X)=I(a)p(a)+I(b)p(b)+I(c)p(c)+I(d)p(d)
=(1×(1/2)+2×(1/4)+3×(1/8)+3×(1/8))bit
=1.75 bit.

Entropy

Entropy is always non-negative (follows from 0<=p(x)<=1)

S(X)=0 means the value of X is known for certain (no randomness).

Large S(X) indicates large uncertainty in X.

Entropy can be changed by observation, as observation can remove/reduce uncertainty from a random variable.

A binary random variable X (with 𝒳={false,true})

p(false)=1-p, 
p(true)=p,

where 0<=p<=1. Entropy of X

S(X)=-p log p-(1-p)log (1-p).

Plot[-(p Log[p]+(1-p)Log[1-p])/Log[2],{p,0,1},PlotStyle->Black,FrameLabel->{"\!\( p\)","\!\(S(X)\) [bit]"},GridLines->{{0.5},{0,1}},ImageSize->180]

S(X)=0 when p=0 (X=false for sure) or p=1 (X=true for sure).

S(X) is maximized at p=1/2, where X is most uncertain. The maximum entropy of a binary random variable is 1 bit.

The Unit of Entropy

S(X)=-(Subscript[k, B])(∑_(x∈𝒳))p(x) log p(x).

In natural units (Planck units), (Subscript[k, B])=ℏ=c=G=1.

({{Field of physics, const., appeared in}, {Statistical mechanics, (Subscript[k, B]), E=(Subscript[k, B ])T}, {Quantum mechanics, ℏ, E=ℏ ω}, {Special relativity, c, E=c p}, {General relativity, G, E=-G ((M m)/r)}})

Fundamental constants are introduced to covert everything to energy.

The Epistemological Nature of Entropy

DynamicModule[{n=20,a=0.03,run=False,bar=True,col=False,xs,vs,cols,setcol,update},
SeedRandom[0];
xs=Join[#-{0.5,0}&/@RandomReal[{-0.5,0.5},{n/2,2}],#+{0.5,0}&/@RandomReal[{-0.5,0.5},{n/2,2}]];
vs=ReIm@Exp[I RandomReal[{0,2π},n]];
cols=Table[If[xs[[i,1]]<0,Red,Green],{i,n}];
update[dt_:0.03]:=If[run,Block[{mask,x0s},
Pause[dt];
x0s=xs;
xs=xs +vs dt;
Do[
If[bar,If[(xs[[i,1]]-a)(x0s[[i,1]]-a)<0,xs[[i,1]]=2a-xs[[i,1]];vs[[i,1]]=-vs[[i,1]]];
If[(xs[[i,1]]+a)(x0s[[i,1]]+a)<0,xs[[i,1]]=-2a-xs[[i,1]];vs[[i,1]]=-vs[[i,1]]];];
If[Abs[xs[[i,1]]]>1,xs[[i,1]]=Sign[xs[[i,1]]](2-Abs[xs[[i,1]]]);vs[[i,1]]=-vs[[i,1]]];
If[Abs[xs[[i,2]]]>0.5,xs[[i,2]]=Sign[xs[[i,2]]](1-Abs[xs[[i,2]]]);vs[[i,2]]=-vs[[i,2]]],{i,n}];]];
Deploy@Dynamic[update[];Graphics[{EventHandler[{EdgeForm[Black],White,Rectangle[{-1-a,-0.5-a},{1+a,0.5+a}]},"MouseClicked":>(run=!run)],
EventHandler[{If[col,#1,Gray],Disk[#2,a]},"MouseClicked":>(cols=Table[If[xs[[i,1]]<0,Red,Green],{i,n}];col=!col)]&@@@Thread[{cols,xs}],
EventHandler[{If[bar,Black,Directive[Opacity[0.2],Black,Dashed]],Line[{0,#}&/@{-0.5,0.5}]},"MouseClicked":>(Do[If[Abs[xs[[i,1]]]<=a,xs[[i,1]]=Sign[xs[[i,1]]](Abs[xs[[i,1]]]+a)],{i,n}];bar=!bar)]},ImageSize->180]]]

Ruben Verresen. Does entropy depend on observer? Physics StackExchange.

Remove, wait, and re-insert the partition - not much changed - there has been no increase of entropy:

Δ S=0.

However, it turns out we were color blind. The balls are red and green, and initially all red (green) balls are on the left (right). Mixing will increase the entropy by (1 bit for each ball)

Δ S=N log 2.

Both are correct.

Entropy is not a property of the physical system. It depends on what we can (or choose to) measure, and how much details we are able (or willing) to look into.

Information can be used to extract work.

Replace the partition with a semi-permeable membrane that only allows red balls to pass through (i.e., the membrane is not color-blind). It will experience an imbalanced pressure from the green side, known as the osmotic pressure, which can be used to do work.

To us, as color-blind observers, this seems to violate the second law of thermodynamics, that it is impossible to extract work from a maximal entropy system.

However, observing such a work done would force us to reconsider our limitations, and to acknowledge that there must be some underlying difference among the balls, even if we cannot directly perceive it!

Statistical Inference

Maximum Entropy Estimation

Entropy serves as a metric for our ignorance. By maximizing entropy, we are maximizing our ignorance in a controlled way.

We should be honest about our ignorance. MaxEnt insists that our probabilistic beliefs should be calibrated such that they do not pretend a level of knowledge that exceeds what our evidence or hypotheses justify.

E. T. Jaynes. Why do we Stand on Maximum Entropy. https://bayes.wustl.edu/etj/articles/stand.on.entropy.pdf

∀x∈𝒳:p(x)=(1/Ω),

S(X)=log Ω.

Prove that the equal probability distribution is maximal entropy.

S(X)=-(∑_(x∈𝒳))p(x)log p(x),

(∑_(x∈𝒳))p(x)=1.

ℒ=S(X)+λ ((∑_(x∈𝒳))p(x)-1).

((δℒ)/(δ p(x)))=- (1+log p(x))+λ=0.

p(x)=(e^(λ-1)).

(∑_(x∈𝒳))p(x)=Ω (e^(λ-1))=1
=> (e^(λ-1))=(1/Ω).

p(x)=(1/Ω).

Gibbs Distribution

⟨(Subscript[f, a])(x)⟩=(∑_(x∈𝒳))(Subscript[f, a])(x)p(x),

Use the Lagrangian multiplier method to show that Eq. () is the optimal solution of Eq. ().

S(X)=-(∑_(x∈𝒳))p(x)log p(x),

(∑_(x∈𝒳))p(x)=1,

(∑_(x∈𝒳))(Subscript[f, a])(x)p(x)=⟨(Subscript[f, a])(x)⟩.

ℒ=S(X)+(Subscript[λ, 0]) ((∑_(x∈𝒳))p(x)-1)+(∑_a)(Subscript[λ, a])(⟨(Subscript[f, a])(x)⟩-(∑_(x∈𝒳))(Subscript[f, a])(x)p(x)).

((δℒ)/(δ p(x)))=- (1+log p(x))+(Subscript[λ, 0])-(∑_a)(Subscript[λ, a])(Subscript[f, a])(x)=0.

p(x)=exp((Subscript[λ, 0])-(∑_a)(Subscript[λ, a])(Subscript[f, a])(x)-1)=(e^((Subscript[λ, 0])-1))exp(-(∑_a)(Subscript[λ, a])(Subscript[f, a])(x))

(∑_(x∈𝒳))p(x)=(e^((Subscript[λ, 0])-1))(∑_(x∈𝒳))exp(-(∑_a)(Subscript[λ, a])(Subscript[f, a])(x))=(e^((Subscript[λ, 0])-1))Z[λ]=1,

Z[λ]=(∑_(x∈𝒳))exp(-(∑_a)(Subscript[λ, a])(Subscript[f, a])(x)).

(e^((Subscript[λ, 0])-1))=(1/(Z[λ])),

p(x)=(1/(Z[λ]))exp(-(∑_a)(Subscript[λ, a])(Subscript[f, a])(x))

E. T. Jaynes. Information Theory and Statistical Mechanics. https://bayes.wustl.edu/etj/articles/theory.1.pdf

p(x) takes an exponential form, where the exponent is a linear combination of all constraint functions (Subscript[f, a])(x) by corresponding Lagrangian multipliers (Subscript[λ, a])

p(x)∝exp(-(∑_a)(Subscript[λ, a])(Subscript[f, a])(x)).

Z[λ] is called the partition function,

serving as the normalization factor to ensure (∑_(x∈𝒳))p(x)=1.

(Subscript[λ, a]) are Lagrangian multipliers, which should be determined by solving the corresponding constraint equation

-(∂/(∂(Subscript[λ, a])))log Z[λ]=⟨(Subscript[f, a])(x)⟩.

Derive Eq. ().

-(∂/(∂(Subscript[λ, a])))log Z[λ]=-(1/(Z[λ]))((∂Z[λ])/(∂(Subscript[λ, a])))
=-(1/(Z[λ]))(∑_(x∈𝒳))(∂/(∂(Subscript[λ, a])))exp(-(∑_a)(Subscript[λ, a])(Subscript[f, a])(x))
=(1/(Z[λ]))(∑_(x∈𝒳))(Subscript[f, a])(x)exp(-(∑_a)(Subscript[λ, a])(Subscript[f, a])(x))
=(∑_(x∈𝒳))(Subscript[f, a])(x)p(x).

(∑_(x∈𝒳))(Subscript[f, a])(x)p(x)=⟨(Subscript[f, a])(x)⟩,

-(∂/(∂(Subscript[λ, a])))log Z[λ]=⟨(Subscript[f, a])(x)⟩.

The entropy of the optimal distribution Eq. () is

S(X)=log Z[λ]+(∑_a)(Subscript[λ, a])⟨(Subscript[f, a])(x)⟩
=(1-(∑_a)(∂/(∂log (Subscript[λ, a]))))log Z[λ],

which has been maximal under the constraints of ⟨(Subscript[f, a])(x)⟩.

Derive Eq. ().

S(X)=-(∑_(x∈𝒳))p(x)log p(x)
=-(∑_(x∈𝒳))p(x)(-log Z[λ]-(∑_a)(Subscript[λ, a])(Subscript[f, a])(x))
=log Z[λ](∑_(x∈𝒳))p(x)+(∑_a)(Subscript[λ, a])(∑_(x∈𝒳))(Subscript[f, a])(x)p(x)
=log Z[λ]+(∑_a)(Subscript[λ, a])⟨(Subscript[f, a])(x)⟩.

Statistical Ensembles

Microcanonical Ensemble

Probability Distribution

Block[{ar=0.618,d=0.15,lt=0.5,a=0.06,n=10},
SeedRandom[4];Graphics[{{EdgeForm[Black],{HatchFilling[],Rectangle[{-ar-d,-1-d},{ar+d,1+d}]},{Lighter[Red,lt],Rectangle[{-ar,-1},{ar,1}]}},{Disk[#,a]&/@Thread[{RandomReal[{-ar+a,ar-a},n],RandomReal[{-1+a,1-a},n]}]},
Text[Style["\!\(E,N\)",Background->Lighter[Red,lt]],{ar,-1},{1.1,-1}]},PlotRange->{{-1.5,1.5},{-1.5,1.5}},ImageSize->140]]

All accessible microstates x∈(Subscript[𝒳, E,N]) of the system have the same energy E and particle number N. The ensemble of all such systems belongs to the same macrostate labeled by (E,N).

Without further constraints, the MaxEnt estimation asserts that the probability distribution is uniform, i.e., every microstate is equally likely

where

Ω(E,N) is the number of microstates in (Subscript[𝒳, E,N]) (the set of microstates with energy E and particle number N),

In microcanonical ensemble, Ω(E,N) serves the role of the partition function.

Thermodynamics

The entropy function enables us to define temperature T and chemical potential  μ through its derivatives, [accept these definitions as historical conventions for now]

(Subscript[(((∂S)/(∂E))), N])=(1/T),
(Subscript[(((∂S)/(∂N))), E])=-(μ/T).

In differential form,

dS=(1/T)dE-(μ/T)dN,

which can be written in a more familiar form as the thermodynamic identity for energy

This is also known as the first law of thermodynamics, which states that the energy change dE is a sum of heat TdS and work  μdN.

The conjugate pair (μ,N) can be replace by other pairs in other thermodynamic problems, but the underlying principles are the same. [See the section Conjugate Pairs of Variables for more examples.]

Equilibrium Condition

The probability to observe them in microstates ((Subscript[x, 1]),(Subscript[x, 2])) is

(Subscript[p, 0])((Subscript[x, 1]),(Subscript[x, 2]))=p((Subscript[x, 1]))p((Subscript[x, 2])).

Imaging bringing the two systems in contact for some time t. They jointly evolve into new microstates:

Block[{r=0.45,h=0.5,d=0.6,box},
box[lt_:0,p_:""]:={{Lighter[Lighter[Red,lt],0.5],Rectangle[{0,0},{r,h}],Lighter[Darker[Orange,lt],0.5],Rectangle[{r,0},{1,h}]},{Line[{{0,0},{0,h},{1,h},{1,0},{0,0}}],Line[{{r,0},{r,h}}],
Text["\!\(x\_1"<>p<>"\)",{r/2,h/2},{0,0}],
Text["\!\(x\_2"<>p<>"\)",{(r+1)/2,h/2},{0,0}]}};
Graphics[{Translate[box[0.2,""],{-d,0}],Text["\!\(→\&t\)",{0.5,h/2},{0,0}],Translate[box[0,"\%′"],{d,0}]},ImageSize->260]]

Assuming the time evolution is deterministic, the probability to observe ((Subscript[x, 1]^′),(Subscript[x, 2]^′)) after the contact is

(Subscript[p, t])((Subscript[x, 1]^′),(Subscript[x, 2]^′))=(Subscript[p, 0])((Subscript[x, 1]),(Subscript[x, 2]))=p((Subscript[x, 1]))p((Subscript[x, 2])).

If the two systems were in equilibrium, their joint probability distribution should be time independent, meaning

(Subscript[p, t])((Subscript[x, 1]^′),(Subscript[x, 2]^′))=(Subscript[p, 0])((Subscript[x, 1]^′),(Subscript[x, 2]^′))=p((Subscript[x, 1]^′))p((Subscript[x, 2]^′)).

p(x)=(1/(Ω(E,N))).

Block[{r=0.45,h=0.5,d=0.6,box},
box[lt_:0,p_:""]:={{Lighter[Lighter[Red,lt],0.5],Rectangle[{0,0},{r,h}],Lighter[Darker[Orange,lt],0.5],Rectangle[{r,0},{1,h}]},{Line[{{0,0},{0,h},{1,h},{1,0},{0,0}}],Line[{{r,0},{r,h}}],
Text["\!\(x\_1"<>p<>"\)",{r/2,h},{0,1}],
Text["\!\(x\_2"<>p<>"\)",{(r+1)/2,h},{0,1}],Text["\!\(E\_1"<>p<>",N\_1"<>p<>"\)",{r/2,0},{0,-1}],
Text["\!\(E\_2"<>p<>",N\_2"<>p<>"\)",{(r+1)/2,0},{0,-1}]}};
Graphics[{Translate[box[0.2,""],{-d,0}],Text["\!\(→\&t\)",{0.5,h/2},{0,0}],Translate[box[0,"\%′"],{d,0}]},ImageSize->260]]

(1/(Ω((Subscript[E, 1]),(Subscript[N, 1]))))(1/(Ω((Subscript[E, 2]),(Subscript[N, 2]))))=(1/(Ω((Subscript[E, 1]^′),(Subscript[N, 1]^′))))(1/(Ω((Subscript[E, 2]^′),(Subscript[N, 2]^′)))),

S((Subscript[E, 1]),(Subscript[N, 1]))+S((Subscript[E, 2]),(Subscript[N, 2]))=S((Subscript[E, 1]^′),(Subscript[N, 1]^′))+S((Subscript[E, 2]^′),(Subscript[N, 2]^′)),

(Subscript[E, 1])+(Subscript[E, 2])=(Subscript[E, 1]^′)+(Subscript[E, 2]^′),
(Subscript[N, 1])+(Subscript[N, 2])=(Subscript[N, 1]^′)+(Subscript[N, 2]^′).

d(Subscript[E, 1])+d(Subscript[E, 2])=0,
d(Subscript[N, 1])+d(Subscript[N, 2])=0,
dS((Subscript[E, 1]),(Subscript[N, 1]))+dS((Subscript[E, 2]),(Subscript[N, 2]))
=(Subscript[(((∂S)/(∂(Subscript[E, 1])))), (Subscript[N, 1])])d(Subscript[E, 1])+(Subscript[(((∂S)/(∂(Subscript[N, 1])))), (Subscript[E, 1])])d(Subscript[N, 1])+(Subscript[(((∂S)/(∂(Subscript[E, 2])))), (Subscript[N, 2])])d(Subscript[E, 2])+(Subscript[(((∂S)/(∂(Subscript[N, 2])))), (Subscript[E, 2])])d(Subscript[N, 2])
=0.

Derive the following two equilibrium conditions from Eq. ().

d(Subscript[E, 1])+d(Subscript[E, 2])=0=>d(Subscript[E, 2])=-d(Subscript[E, 1]),
d(Subscript[N, 1])+d(Subscript[N, 2])=0=>d(Subscript[N, 2])=-d(Subscript[N, 1]),

(Subscript[(((∂S)/(∂(Subscript[E, 1])))), (Subscript[N, 1])])d(Subscript[E, 1])+(Subscript[(((∂S)/(∂(Subscript[N, 1])))), (Subscript[E, 1])])d(Subscript[N, 1])-(Subscript[(((∂S)/(∂(Subscript[E, 2])))), (Subscript[N, 2])])d(Subscript[E, 1])-(Subscript[(((∂S)/(∂(Subscript[N, 2])))), (Subscript[E, 2])])d(Subscript[N, 1])
=((Subscript[(((∂S)/(∂(Subscript[E, 1])))), (Subscript[N, 1])])-(Subscript[(((∂S)/(∂(Subscript[E, 2])))), (Subscript[N, 2])]))d(Subscript[E, 1])+((Subscript[(((∂S)/(∂(Subscript[N, 1])))), (Subscript[E, 1])])-(Subscript[(((∂S)/(∂(Subscript[N, 2])))), (Subscript[E, 2])]))d(Subscript[N, 1])
=0.

(Subscript[(((∂S)/(∂(Subscript[E, 1])))), (Subscript[N, 1])])-(Subscript[(((∂S)/(∂(Subscript[E, 2])))), (Subscript[N, 2])])=0,
(Subscript[(((∂S)/(∂(Subscript[N, 1])))), (Subscript[E, 1])])-(Subscript[(((∂S)/(∂(Subscript[N, 2])))), (Subscript[E, 2])])=0.

The thermal equilibrium condition

(1/(Subscript[T, 1]))=(Subscript[(((∂S)/(∂(Subscript[E, 1])))), (Subscript[N, 1])])=(Subscript[(((∂S)/(∂(Subscript[E, 2])))), (Subscript[N, 2])])=(1/(Subscript[T, 2])),

or equivalently

which defines what should be called temperature.

The chemical equilibrium condition

-((Subscript[μ, 1])/(Subscript[T, 1]))=(Subscript[(((∂S)/(∂(Subscript[N, 1])))), (Subscript[E, 1])])=(Subscript[(((∂S)/(∂(Subscript[N, 2])))), (Subscript[E, 2])])=-((Subscript[μ, 2])/(Subscript[T, 2])),

or equivalently (assuming (Subscript[T, 1])=(Subscript[T, 2]) is already achieved)

which defines what should be called chemical potential.

Canonical Ensemble

Probability Distribution

Block[{ar=0.618,d=0.1,lt=0.5,a=0.06,n=10},
SeedRandom[4];Graphics[{{EdgeForm[Red],{Lighter[Red,lt],Rectangle[{-ar-d,-1-d},{ar+d,1+d}]},{Lighter[Red,lt],Rectangle[{-ar,-1},{ar,1}]}},{Disk[#,a]&/@Thread[{RandomReal[{-ar+a,ar-a},n],RandomReal[{-1+a,1-a},n]}]},
Text[Style["\!\(T,N\)",Background->Lighter[Red,lt]],{ar,-1},{1.1,-1}]},PlotRange->{{-1.5,1.5},{-1.5,1.5}},Background->Lighter[Red,lt],ImageSize->140]]

Microstates x∈(Subscript[𝒳, N]) of the system still have the same particle number N, but can take all different energies.

An energy function E(x) is therefore introduced to evaluate the energy associated with each state x.

One can still talk about the energy expectation value for the ensemble

E=⟨E(x)⟩=(∑_(x∈(Subscript[𝒳, N])))E(x)p(x).

Given temperature T, the ensemble is expected to have an energy expectation value ⟨E(x)⟩ that matches the macroscopic energy E.

Given the energy constraint ⟨E(x)⟩=E, the MaxEnt estimation asserts that the probability distribution should be [recall Eq. ()]

where

The partition function Z(β,N) is given by

Originally introduced as a Lagrangian multiplier,  β is found to be related to the temperature T by (assuming (Subscript[k, B])=1)

so  β is also called the inverse temperature.

Lagrangian Multipliers

E=-(Subscript[(((∂log Z)/(∂β))), N]).

S=log Z+β E.

dS=β dE+(Subscript[(((∂log Z)/(∂N))), β])dN.

Derive Eq. () using Eq. ().

dS=d(log Z)+d(β E)
=(Subscript[(((∂log Z)/(∂β))), N])dβ+(Subscript[(((∂log Z)/(∂N))), β])dN+β dE+Edβ
(Overscript[=, Eq. ()])-Edβ+(Subscript[(((∂log Z)/(∂N))), β])dN+β dE+Edβ
=β dE+(Subscript[(((∂log Z)/(∂N))), β])dN.

dS=(1/T)dE-(μ/T)dN,

β=(1/T),
 μ=-T (Subscript[(((∂log Z)/(∂N))), β])=-T (Subscript[(((∂log Z)/(∂N))), T]).

Thermodynamics

The free energy enables us to compute:

Entropy

S=-(Subscript[(((∂F)/(∂T))), N]),

Derive Eq. ().

S=(1-(∂/(∂log β)))log Z
=(1+(∂/(∂log T)))log Z.

β=1/T
=>log β=-log T
=>d(log β)=-d(log T)
=>(∂/(∂log β))=-(∂/(∂ log T)).

S=log Z+((∂log Z)/(∂log T))
=log Z+T(∂/(∂T))log Z
=(∂/(∂T))(T log Z)
=-((∂F)/(∂T)).

Chemical potential

μ=(Subscript[(((∂F)/(∂N))), T]).

Derive Eq. ().

μ=-T (Subscript[(((∂log Z)/(∂N))), T]).

μ=(Subscript[(((∂(-T log Z))/(∂N))), T])=(Subscript[(((∂F)/(∂N))), T]).

In differential form,

This is the thermodynamic identity for free energy.

Derive Eq. () from Eq. ().

S=log Z+β E 
=>log Z=S-β E=S-(E/T).

F=-T log Z=-T(S-(E/T))=E-T S.

E=F-(Subscript[T(((∂F)/(∂T))), N]).

Grand Canonical Ensemble

Probability Distribution

Block[{ar=0.618,d=0.15,lt=0.5,a=0.06,n=10},
SeedRandom[4];Graphics[{{EdgeForm[Directive[Red,Dashed]],Lighter[Red,lt],Rectangle[{-ar,-1},{ar,1}]},{Disk[#,a]&/@Thread[{RandomReal[{-ar+a,ar-a},n],RandomReal[{-1+a,1-a},n]}]},
{Disk[#,a]&/@Cases[RandomReal[{-1.5,1.5},{50,2}],{x_,y_}/;Abs[x]>ar+a||Abs[y]>1+a]},
Text[Style["\!\(T,μ\)",Background->Lighter[Red,lt]],{ar,-1},{1.1,-1}]},PlotRange->{{-1.5,1.5},{-1.5,1.5}},Background->Lighter[Red,lt],ImageSize->140]]

Microstates x∈𝒳 can take all possible states of different energies and particle numbers.

To evaluate the energy and particle number associated with a particular microstate x, the energy function E(x) and the particle number function N(x) are introduced respectively.

The energy E and particle number N of the macrostate refers to their expectation values in grand canonical ensemble

E=⟨E(x)⟩=(∑_(x∈𝒳))E(x)p(x),
N=⟨N(x)⟩=(∑_(x∈𝒳))N(x)p(x).

Given the energy constraint ⟨E(x)⟩=E and the particle number constraint ⟨N(x)⟩=N, the MaxEnt estimation asserts that the probability distribution should be [recall Eq. ()]

where

The (grand) partition function ℨ(β,μ) is given by

The Lagrangian multipliers are parameterized by the inverse temperature  β=1/T and the chemical potential  μ.

Lagrangian Multipliers

ℨ(β,-γ/β)=(∑_(x∈𝒳))(e^(-β E(x)-γ N(x))),

E=-(Subscript[(((∂log ℨ)/(∂β))), γ]),
N=-(Subscript[(((∂log ℨ)/(∂γ))), β]),

S=log ℨ+β E+γ N.

dS=βdE+γdN.

Derive Eq. () using Eq. ().

dS=d(log ℨ)+d(β E)+d(γ N)
=(Subscript[(((∂log ℨ)/(∂β))), γ])dβ+(Subscript[(((∂log ℨ)/(∂γ))), β])dγ+βdE+Edβ+γdN+Ndγ
(Overscript[=, Eq. ()])-Edβ-Ndγ+βdE+Edβ+γdN+Ndγ
=βdE+γdN.

dS=(1/T)dE-(μ/T)dN,

β=(1/T),
γ=-(μ/T),

Thermodynamics

Some textbook defines a similar concept, call the “grand potential”: Φ(T,μ)=-𝔉(T,μ).

The (grand) free energy enables us to compute:

Entropy

S=-(Subscript[(((∂𝔉)/(∂T))), μ]),

Derive Eq. ().

ℨ(β,-γ/β)=(∑_(x∈𝒳))(e^(-β E(x)-γ N(x))),

S=log ℨ-(Subscript[(((∂log ℨ)/(∂log β))), γ])-(Subscript[(((∂log ℨ)/(∂log γ))), β]).

({{{, ({{ β=1/T,}, {γ=-μ/T,}})}})
=>({{{, ({{log β=-log T,}, {log γ=log(-μ)-log T,}})}})
=>({{{, ({{d(log β)=-d(log T),}, {d(log γ)=d(log(-μ))-d(log T).}})}})

({{{, ({{d(log β)=-d(log T),}, {d(log γ)=-d(log T).}})}})

(Subscript[(((∂log ℨ)/(∂log T))), μ])=-(Subscript[(((∂log ℨ)/(∂log β))), γ])-(Subscript[(((∂log ℨ)/(∂log γ))), β]),

S=log ℨ+(Subscript[(((∂log ℨ)/(∂log T))), μ])
=log ℨ+(Subscript[T(((∂log ℨ)/(∂T))), μ])
=(Subscript[(((∂(T log ℨ))/(∂T))), μ])
=-(Subscript[(((∂𝔉)/(∂T))), μ]).

Particle number

N=-(Subscript[(((∂𝔉)/(∂μ))), T]),

Derive Eq. ().

N=-(Subscript[(((∂log ℨ)/(∂γ))), β]).

N=-(Subscript[(((∂log ℨ)/(-β∂μ))), β])
=(1/β)(Subscript[(((∂log ℨ)/(∂μ))), β]).

N=(Subscript[T(((∂log ℨ)/(∂μ))), T])
=-(Subscript[(((∂(-T log ℨ))/(∂μ))), T])
=-(Subscript[(((∂𝔉)/(∂μ))), T]).

In differential form,

This is the thermodynamic identity for (grand) free energy.

𝔉=E-T S-μ N.

Derive Eq. () from Eq. ().

S=log ℨ+β E+γ N
=>log ℨ=S-β E-γ N=S-(E/T)+((μ N)/T).

𝔉=-T log ℨ=-T(S-(E/T)+((μ N)/T))
=E-T S-μ N.

E=𝔉-(Subscript[T(((∂𝔉)/(∂T))), μ])-(Subscript[μ(((∂𝔉)/(∂μ))), T]).

Summary

Statistical Ensembles

({{Ensemble, Microcanonical,  , Canonical,  , Grand Canonical}, {Microstates, (Subscript[𝒳, E,N]), ⊆, (Subscript[𝒳, N]), ⊆, 𝒳}, {Macrostate, (E,N),  , (T,N),  , (T,μ)}, {Probability, p=(1/Ω),  , p=(1/Z)(e^(-β E)),  , p=(1/ℨ)(e^(-β (E-μ N)))}, {Partition function, Ω=∑1,  , Z=∑(e^(-β E)),  , ℨ=∑(e^(-β (E-μ N)))}, {Free energy, none,  , F=-T log Z,  , 𝔉=-T log ℨ}, {Entropy, S=log Ω,  , S=-(Subscript[(((∂F)/(∂T))), N]),  , S=-(Subscript[(((∂𝔉)/(∂T))), μ])}, {Energy, E,  , E=F+T S,  , E=𝔉+T S+μ N}, {Particle number, N,  , N,  , N=-(Subscript[(((∂𝔉)/(∂μ))), T])}, {Temperature, T=(Subscript[(((∂E)/(∂S))), N]),  , T,  , T}, {Chemical potential, μ= (Subscript[(((∂E)/(∂N))), S]),  ,  μ=(Subscript[(((∂F)/(∂N))), T]),  , μ}, {Thermodynamics, dE=TdS+μ dN,  , dF=-S dT+μdN,  , d𝔉=-SdT-Ndμ}})

Conjugate Pair of Variables

Intensive variable (like T,  μ) to be balanced at equilibrium,

Extensive variable (like E, N) to be transferred in order to reach the equilibrium,

({{Process/System, Conjugate pair of variables, , Contribution}, {, Intensive variable, Extensive variable, to energy (dE)}, {Heat transfer, Temperature (T), Entropy (S), +TdS}, {Particle transfer, Chemical potential (μ), Particle number (N), +μdN}, {Charge transfer, Electric potential (ℰ), Charge (Q), +ℰdQ}, {Gas, Liquid, Pressure (P), Volume (V), -PdV}, {Membrane, Surface tension (γ), Area (A), -γdA}, {Elastic material, Stree tensor (σ), Strain tensor (ϵ), +Tr σdϵ}, {Dielectric material, Electric field (D), Electric polarization (P), -(1/ε)D·dP}, {Magnetic material, Magnetic field (B), Magnetization (M), -B·dM}, {Rotating galaxy, Angular velocity (ω), Angular momentum (L), +ωdL}})

Gas (-PdV),

Heat transfer (+TdS),

Particle transfer (+μdN).

The thermodynamics is described by

or, in terms of free energies,

dF=-PdV-S dT+μdN,
d𝔉=-PdV-SdT-Ndμ.

This enables us to write done thermodynamic identities, such as