Two Infrared Sensors

An object at true temperature θ, measured by two infrared sensors that differ only in their noise.

One temperature

Drag θ: each curve is one sensor's voltage distribution p(v; θ) at that temperature, and the ticks under it are 20 readings from that sensor. Pin θ₀ to keep the distribution at one temperature on screen, dashed, while θ moves.

True temperature θ42.0 °C

Sensor A

noise σ 0.8 mV

Fisher information ℐ(θ)
Relative entropy D(p_θ₀ ‖ p_θ)

Sensor B

noise σ 3.0 mV

Fisher information ℐ(θ)
Relative entropy D(p_θ₀ ‖ p_θ)

Two nearby temperatures

Each sensor takes 20 readings with the cup at θ₁ and 20 more at a nearby temperature θ₂. The noisier the sensor, the more the two sets of readings look alike. Both plots span the same number of noise widths σ, so a curve has the same shape in each.

Lower temperature θ₁40.0 °C Gap θ₂ − θ₁2.0 °C

θ₂ = 42.0 °C

Sensor A

noise σ = 0.8 mV

Shift of the mean g(θ₂ − θ₁)
Relative entropy D(p_θ₁ ‖ p_θ₂)

Sensor B

noise σ = 3.0 mV

Shift of the mean g(θ₂ − θ₁)
Relative entropy D(p_θ₁ ‖ p_θ₂)

Fisher information

The relative entropy between the distributions at two temperatures measures how easily readings at one can be told apart from readings at the other. For a sensor with gain g and noise σ, raising the temperature by δ gives

D (p_{θ} ‖ p_{θ + δ}) = \frac{g^{2} δ^{2}}{2 σ^{2}} .

The relative entropy grows with the square of δ, faster for the less noisy sensor. Its curvature at δ = 0 is the Fisher information.

Definition

ℐ (θ) = \lim_{δ \to 0} \frac{2}{δ^{2}} D (p_{θ} ‖ p_{θ + δ})

equivalently

D (p_{θ} ‖ p_{θ + δ}) = \frac{1}{2} ℐ (θ) δ^{2} + o (δ^{2})

For both sensors the relative entropy is exactly quadratic, so ℐ(θ) = g²/σ² at every temperature: for sensor A and for sensor B. Sensor A's Fisher information is times sensor B's, and so is the relative entropy produced by any change in temperature.

Relative entropies of independent readings add, so 20 readings carry 20 times the Fisher information of one: for sensor A and for sensor B.

Relative entropy between the distributions at θ and θ + δ for each sensor. The dashed line marks the gap θ₂ − θ₁ set in the section above.

Both sensors have gain g = 1 mV per °C and read zero when the object is at their own temperature, 25 °C. A reading is v = g(θ − 25) + ε with ε ~ N(0, σ²), so p(v; θ) = N(v; g(θ − 25), σ²). Its Fisher information is ℐ(θ) = g²/σ², and D(p_θ₀ ‖ p_θ) = ℐ(θ)(θ − θ₀)²/2 exactly.