Step 03Question 2.3 · model selection

Choosing ρ by simulation

A penalty that is too small keeps the noise; one that is too large erases the signal. Because we can generate as many datasets as we like, ρ can be picked by brute force: fit the lasso again and again on fresh noise and keep the value with the lowest average error.

For each of 21 values $\rho \in \{0, 0.05, \dots, 1\}$ and each of 10 realisations, new noise is drawn, X is rebuilt and standardised, the lasso is fitted, and the reconstruction error is recorded:

\begin{gathered} \mathrm{MSE}(\rho) = \frac{1}{NV}\,\big\lVert X - D_{\mathrm{LR}} A_{\mathrm{LR}} \big\rVert_F^2 \\ D_{\mathrm{LR}} = X A_{\mathrm{LR}}^\top \end{gathered}

That is 210 lasso fits on 240 × 441 matrices. They run in a background Web Worker, so the page stays responsive. The random stream continues from the seeded Γ_t and Γ_s exactly as it did in the notebook, which is why the live curve lands on the one I plotted in 2021.

Using the 2021 parameters. Results match the report.

Realisations10

Each realisation redraws Γt and Γs for every one of the 21 values of ρ, as the notebook did, and fits the lasso to each new X.

Spatial maps used to rebuild X

The notebook re-read the maps from a row-major CSV here. See the fidelity notes on step 2.

Figure 3.1

Mean MSE against the lasso penalty ρ

Mean MSE (live)
Min to max over realisations
2021 report
Lowest mean MSE
ρ chosen in 2021

Computing the first realisation…

Mean of ‖X − D_LRA_LR‖²_F / NV over 10 realisations at each ρ (log scale), with the shaded band spanning the smallest to largest realisation. The wide grey line is the curve from the 2021 notebook. With the default settings the two coincide exactly.

Lowest mean MSE: –; computing
ρ used in 2021: 0.625; midway between 0.60 and 0.65
MSE at ρ = 1: –; computing

Added in 2026 · uncertainty

How sure is ρ = 0.625?

Ten realisations give one mean curve and one minimum, with no sense of how much either would move on a different ten. Below, the same procedure runs for as many realisations as you like. The mean curve gets a 95% interval (percentile bootstrap from 30 realisations, Student-t below) and the minimum gets a bootstrap distribution. With the as-submitted design the first ten realisations are exactly the 2021 ones, so Figure 3.1 stays the faithful reference.

The paired design reuses one noisy dataset for every ρ within a realisation (common random numbers). Each realisation is then one dataset with its own error curve, so it has its own best ρ, and two values of ρ are compared on the same data. That is what makes a direct statement about the cost of ρ = 0.625 possible. In the as-submitted design every ρ sees different noise, so a single realisation's lowest point is not a dataset's best ρ. Why I chose ρ the way I did, and what I would do now, is in decision record DR-001.

Realisations50

Noise seed 30034. With the as-submitted design, realisations 1 to 10 are exactly the 2021 ones.

How noise is shared across ρ

Fresh noise for every pair of realisation and ρ, as the notebook did, on the 2021 grid of 21 values.

Figure 3.2

Mean MSE with a 95% bootstrap interval

Mean MSE
95% interval
Lowest mean MSE
ρ chosen in 2021

Running the Monte-Carlo…

Computing…

Figure 3.3

Where the best ρ lands

Bootstrap argmin of the mean

Running the Monte-Carlo…

ρ with the lowest mean MSE: –; computing
Bootstrap resamples with the argmin in the 2021 bracket 0.60 to 0.65: –; computing
What ρ = 0.625 costs: Paired only; 0.625 is not on the 2021 grid, and with fresh noise for every ρ two values of ρ are never compared on the same data. Switch to the paired design to measure what ρ = 0.625 costs directly.

Optional. Needs your own Anthropic or OpenAI key, and sends only the instructions and figure summary shown below.

What the extra realisations say

With the 2021 settings (seed 30034) and 50 realisations of the as-submitted procedure, the lowest mean MSE moves from ρ = 0.60 to ρ = 0.55, and no bootstrap resample puts the minimum of the mean curve at 0.65. The bottom of the curve sits between 0.55 and 0.60, so my midpoint of 0.60 and 0.65 lands on the steeper side.

The paired design answers the question for one dataset at a time. Across 50 datasets, each reused for every ρ, the best ρ is 0.575 for 38, 0.60 for 7 and 0.55 for 5. That puts 7 of 50 in my bracket (Wilson 95% CI 7% to 26%) and none at 0.625 itself. Against ρ = 0.60, fixed in advance as the lowest point of the 2021 curve, ρ = 0.625 gives a higher MSE on all 50 datasets, by 0.060 on average (95% CI 0.057 to 0.062), about 10% more. Against 0.575 the gap is 0.089, but 0.575 was picked as the best ρ on these same datasets, so that larger figure is biased upwards (DR-004). The lasso still beats ridge at ρ = 0.625 (see step 06), so the conclusion of Question 2.4 survives, but the reasoning behind the number does not.

Next · Step 04

Principal components

See why regressing on the PCs of the TCs loses their shape.