Cycle 3

17 findings. Ordered by priority: invariants, calibration, degeneracy, plausibility, saturation, sensitivity, framework.

F-cycle-003-01: The backcast ends trade and investment 1.3 points too low against the 2026 ratings.

Layer: parameter. Class: calibration_error. Source: rules

Evidence: checks.parquet backcast_bias:trade_and_investment: trade_and_investment: simulated -1.33 points of change on average against -0.04 observed (bias -1.29, error 2.77 against 1.76 for persistence, direction right 40 per cent) (calibration nations; reconstructed 1996 starting positions) [backcast_comparison].

Hypothesis: Capability in this domain decays too fast for what nations actually spent on it, or costs too much to build: the model cannot reproduce the gains history shows.

Proposed change: Multiply the trade_and_investment class's decay rates by 0.75 in data/sim/seed/params_v0_1.yaml

Scope: parameter

Expected effect: |bias| in trade_and_investment under 1 point on the calibration nations, without worsening the hold-out.

Verification: Rerun the backcast; accept if backcast_bias:trade_and_investment passes and backcast_skill_holdout does not fall.

Risk: Changes every nation's trajectory in this domain, not only the backcast's; the reconstruction itself may be what is wrong.

F-cycle-003-02: The backcast ends financial strength 1.2 points too low against the 2026 ratings.

Layer: parameter. Class: calibration_error. Source: rules

Evidence: checks.parquet backcast_bias:financial_strength: financial_strength: simulated -1.27 points of change on average against -0.04 observed (bias -1.22, error 2.80 against 1.79 for persistence, direction right 40 per cent) (calibration nations; reconstructed 1996 starting positions) [backcast_comparison].

Hypothesis: Capability in this domain decays too fast for what nations actually spent on it, or costs too much to build: the model cannot reproduce the gains history shows.

Proposed change: Multiply the financial_strength class's decay rates by 0.75 in data/sim/seed/params_v0_1.yaml

Scope: parameter

Expected effect: |bias| in financial_strength under 1 point on the calibration nations, without worsening the hold-out.

Verification: Rerun the backcast; accept if backcast_bias:financial_strength passes and backcast_skill_holdout does not fall.

Risk: Changes every nation's trajectory in this domain, not only the backcast's; the reconstruction itself may be what is wrong.

F-cycle-003-03: The backcast ends information and influence 1.2 points too low against the 2026 ratings.

Layer: parameter. Class: calibration_error. Source: rules

Evidence: checks.parquet backcast_bias:information_and_influence: information_and_influence: simulated -1.25 points of change on average against -0.03 observed (bias -1.22, error 2.33 against 1.93 for persistence, direction right 52 per cent) (calibration nations; reconstructed 1996 starting positions) [backcast_comparison].

Hypothesis: Capability in this domain decays too fast for what nations actually spent on it, or costs too much to build: the model cannot reproduce the gains history shows.

Proposed change: Multiply the information_and_influence class's decay rates by 0.75 in data/sim/seed/params_v0_1.yaml

Scope: parameter

Expected effect: |bias| in information_and_influence under 1 point on the calibration nations, without worsening the hold-out.

Verification: Rerun the backcast; accept if backcast_bias:information_and_influence passes and backcast_skill_holdout does not fall.

Risk: Changes every nation's trajectory in this domain, not only the backcast's; the reconstruction itself may be what is wrong.

F-cycle-003-04: The backcast ends production and innovation 1.1 points too low against the 2026 ratings.

Layer: parameter. Class: calibration_error. Source: rules

Evidence: checks.parquet backcast_bias:production_and_innovation: production_and_innovation: simulated -1.16 points of change on average against -0.04 observed (bias -1.12, error 2.39 against 1.50 for persistence, direction right 44 per cent) (calibration nations; reconstructed 1996 starting positions) [backcast_comparison].

Hypothesis: Capability in this domain decays too fast for what nations actually spent on it, or costs too much to build: the model cannot reproduce the gains history shows.

Proposed change: Multiply the production_and_innovation class's decay rates by 0.75 in data/sim/seed/params_v0_1.yaml

Scope: parameter

Expected effect: |bias| in production_and_innovation under 1 point on the calibration nations, without worsening the hold-out.

Verification: Rerun the backcast; accept if backcast_bias:production_and_innovation passes and backcast_skill_holdout does not fall.

Risk: Changes every nation's trajectory in this domain, not only the backcast's; the reconstruction itself may be what is wrong.

F-cycle-003-05: The backcast does worse than assuming nothing changed since 1996.

Layer: mechanic. Class: calibration_error. Source: rules

Evidence: checks.parquet backcast_skill_calibration: 48 calibration nations: simulated 2026 groups miss the ratings by 2.29 points on average, against 1.69 for assuming nothing changed (skill -0.35); bias -0.51; the change in each domain has the right sign 40 per cent of the time; index error 1.51 against 1.25 [backcast_comparison].

Hypothesis: The model moves capability by the wrong amounts or in the wrong places: compare simulated with observed change by domain.

Proposed change: Treat the domain biases first; then test whether the credit calibration (income proportional to the cost of the starting position) lets low-rated nations catch up as history shows.

Scope: structural

Expected effect: backcast_skill_calibration above 0.

Verification: Rerun the backcast; accept if skill is positive on the calibration nations and not worse on the hold-out.

Risk: Over-fitting to reconstructed starting positions.

F-cycle-003-06: The same move is repeated in most rounds in 4 experiments.

Layer: harness. Class: degenerate_strategy. Source: rules

Evidence: checks.parquet same_action: e4-bgd 67% of rounds; e4-phl 77% of rounds; e4-per 77% of rounds; e4-kaz 70% of rounds [decision_sequence].

Hypothesis: The mirror-peer policy measures its gaps against current scores and ignores levels still in delivery, so it keeps funding a gap it has already paid to close; near its horizon greedy can only value fast (bought) moves, so it repeats them.

Proposed change: Count levels still in delivery when mirror_peer measures a gap and when greedy measures the room left below 20 (pipeline/sim/policies).

Scope: harness

Expected effect: same_action passes for the mirror-peer experiments.

Verification: Rerun e3-*-mirror-peer; accept if same_action passes for all five nations.

Risk: Changes the baselines' behaviour, so cycle-on-cycle comparison of those experiments breaks once.

F-cycle-003-07: Groups move more than 3 points in a year without any shock.

Layer: parameter. Class: implausible_dynamics. Source: rules

Evidence: 3 experiments: e4-isr move_without_shock 4.45; e4-rus move_without_shock 4.11; e4-jpn move_without_shock 4.03 (checks.parquet) [decision_sequence]. Of the 218 group-years that moved more than 3 points without a shock, investment arriving averaged 3.72 points and propagation +0.000: the moves are investment, not the graph.

Hypothesis: A move is a share of income, and income scales with the cost of holding the whole position, so the largest chunk buys several levels of a weak group in one year.

Proposed change: Lower chunk_large from 0.03 to 0.018 in data/sim/seed/params_v0_1.yaml (model).

Scope: parameter

Expected effect: move_without_shock at or below 3 points in the E3 experiments that fail it.

Verification: Rerun e4-isr, e4-jpn, e4-rus; accept if the share of failing experiments falls by half.

Risk: Slower growth for every nation and policy; less of the headroom is spent, so carried credits rise.

F-cycle-003-08: Domains gain more than two tiers within five years, including under the hold policy, which makes no moves.

Layer: methodology. Class: methodology_question. Source: rules

Evidence: 27 experiments: e4-rus 11 tiers; e4-ltu 8 tiers; e4-idn 7 tiers; e4-grc 7 tiers; 0 of them hold experiments (checks.parquet) [tier_transitions].

Hypothesis: Below Tier 1 the Pareto layers of a domain are close together (a tolerance of a quarter of a point across 197 nations), so a small move crosses several layers. The heuristic's two tiers may suit coarse tiers, not Pareto layers.

Proposed change: Decide how the model should read tier movement: against the Index's published tier bands, with a wider tolerance, or not at all.

Scope: methodology

Expected effect: A tier measure that moves only when capability moves materially.

Verification: Rerun the cycle batch with the chosen measure; accept if tier_jump passes for the hold experiments.

Risk: Changing the tiering here would make results incomparable with the Index's own tiers.

F-cycle-003-09: Groups are driven to 20 and held there in many runs, across nations and policies.

Layer: parameter. Class: saturation. Source: rules

Evidence: 11 experiments above 25 per cent of runs: e4-sgp 100%; e4-irl 100%; e4-twn 100%; e4-isr 100%; e4-fin 100%. Groups most often driven to 20: Health System Capability (120), National Resilience & Crisis Management (120), Semiconductors & Advanced Computing (90), Artificial Intelligence (90), Energy Technologies (90) [trajectory_by_domain].

Hypothesis: Costs above the frontier (17) do not steepen enough to stop a well-funded nation buying the last levels.

Proposed change: Raise frontier_premium from 1 to 1.5 (model).

Scope: parameter

Expected effect: The share of runs ending with a group newly at 20 falls below 25 per cent in the greedy E3 experiments.

Verification: Rerun the E3 greedy experiments; accept if saturation_high passes in at least half of those that failed.

Risk: Frontier nations gain less; the frontier parameter is already the most sensitive.

F-cycle-003-10: step_time in the batch.

Layer: harness. Class: sensitivity. Source: rules

Evidence: checks.parquet: median step 1695 us against 1026 us in cycle-002.

Hypothesis: Slower code or a busier machine.

Proposed change: Find the change and either record it with a version bump or revert it.

Scope: harness

Expected effect: step_time passes.

Verification: Rerun the batch.

Risk: None.

F-cycle-003-AI01: Four domain classes end the backcast 1.1-1.3 points below observed.

Layer: parameter. Class: calibration_error. Source: ai:deepseek-v4-pro

Evidence: backcast BY DOMAIN table: financial_strength bias -1.22, information_and_influence -1.22, production_and_innovation -1.12, trade_and_investment -1.29.

Hypothesis: Their decay rates are too high relative to historical net change.

Proposed change: Multiply the built and bought decay rates for these four classes by 0.75 in data/sim/seed/params_v0_1.yaml.

Scope: parameter

Expected effect: Reduces simulated domain decline and biases toward zero for non-hold-out calibration nations.

Verification: Rerun backcast on non-hold-out calibration nations and compare domain MAE; require bias <0.5 in absolute value.

Risk: May over-correct saturated high-rated nations if frontier saturation is not fixed first; high-rated groups could rise faster.

F-cycle-003-AI02: The backcast correlation is -0.09 and the model cannot reproduce observed catch-up growth.

Layer: mechanic. Class: calibration_error. Source: ai:deepseek-v4-pro

Evidence: backcast comparison: observed 30-year changes -5.6 to +6.5, simulated -2.8 to +0.7; five fastest risers observed +3.5, model -0.8.

Hypothesis: Income is calibrated to the cost of the current starting position, so low-rated nations remain credit-poor.

Proposed change: Add a catch-up term to the income calculation: an additive bonus scaled by distance below the frontier, gated by governance/human capital or development thresholds.

Scope: structural

Expected effect: Low-start nations receive more credits and can fund moves; backcast correlation should become positive.

Verification: Backcast the five fastest historical risers; require simulated average change at least +2 for them.

Risk: May produce runaway growth for some low-capability nations if not gated; test on holdout CHL/POL only after tuning on non-holdouts.

F-cycle-003-AI03: One-year moves above 3 points occur without shocks in several experiments.

Layer: parameter. Class: implausible_dynamics. Source: ai:deepseek-v4-pro

Evidence: move_without_shock warnings: e4-isr 4.45, e4-rus 4.11; 100% of runs have one above 3.

Hypothesis: chunk_large at 0.03 of last income is large enough to deliver >3 levels in a cheap group in one year.

Proposed change: Reduce chunk_large from 0.03 to 0.018 in params_v0_1.yaml.

Scope: parameter

Expected effect: Maximum one-year build/buy step falls to roughly 2 points, eliminating the warning.

Verification: Rerun the move_without_shock check on ISR and RUS; largest expected below 3.

Risk: Slows all policy movement; may interact with delivery lags and reduce responsiveness.

F-cycle-003-AI04: Groups are driven to 20 and held there in many high-rated runs.

Layer: parameter. Class: saturation. Source: ai:deepseek-v4-pro

Evidence: saturation_high warnings: e4-sgp 100% runs median 4 groups; e4-fin median 3; e4-isr median 2; e4-twn and e4-irl 100% runs.

Hypothesis: The frontier premium is too low to slow investment above the frontier, so scores max out.

Proposed change: Raise frontier_premium from 1.0 to 1.5 in params_v0_1.yaml.

Scope: parameter

Expected effect: Increases marginal cost above frontier, reducing the number of groups at 20.

Verification: Rerun saturation_high check on affected nations; median new saturating groups below 1.

Risk: May over-penalise legitimate frontier leaders; if high starts decline, pair with the catch-up fix.

F-cycle-003-AI05: Domain tier gains of more than two tiers in five years are widespread, including under a hold policy with no moves.

Layer: methodology. Class: methodology_question. Source: ai:deepseek-v4-pro

Evidence: tier_jump warnings in e4-est, e4-chl, e4-pol, e4-chn, e4-ind, e4-vnm, e4-mys, e4-idn, e4-bra, e4-rus, e4-zaf, e4-ken, e4-tha, e4-sau; e4-rus largest gain 11 tiers, e4-idn 7 tiers.

Hypothesis: Published tier bands are too sensitive to small continuous score changes, so tier rank is a noisy objective.

Proposed change: Use the capability index and rank as primary objectives; relegate Pareto tier rank to diagnostic, or widen tolerance to at least 0.5 and report only horizon tier rank.

Scope: methodology

Expected effect: Fewer spurious tier_jump warnings and more stable ranking objectives.

Verification: Compare tier_rank_p50 with index_p50 and rank_p50; check if tier changes remain when tolerance is widened.

Risk: If stakeholders require tier thresholds, this removes a meaningful metric; document the trade-off.

F-cycle-003-AI06: The most repeated move appears in a median 67% of BGD rounds and 77% of PHL rounds.

Layer: harness. Class: degenerate_strategy. Source: ai:deepseek-v4-pro

Evidence: same_action warnings for e4-bgd and e4-phl.

Hypothesis: The policy measure of group gap or remaining room does not subtract credits already in delivery, so it repeats the same move.

Proposed change: Count pipeline credits (future deliveries) as committed before mirror_peer or greedy chooses the next move; for scripted policies, add round-robin or random perturbation.

Scope: harness

Expected effect: Reduces repeated moves and diversifies portfolio.

Verification: Rerun same_action checks; median repetition below 50%.

Risk: May reduce full funding of priority groups and slow high-ranked objectives.

F-cycle-003-AI07: The model currently produces broad declines for most developing nations but saturates high-rated groups; this is coherent with a maintenance model, not a development model.

Layer: framework. Class: framework_question. Source: ai:deepseek-v4-pro

Evidence: 37/50 median index values below start; backcast correlation -0.09; saturation_high warnings.

Hypothesis: The framework may be missing a positive development feedback: capabilities attract income and investment beyond current E, enabling catch-up.

Proposed change: Treat the income function as a framework question: decide whether the simulation is meant to reproduce historical development or to stress-test maintenance under shocks; if the former, add catch-up and diminishing returns.

Scope: framework

Expected effect: Aligns model architecture with the intended use and improves backcast fit.

Verification: Backcast correlation and MAE against history; policy experiments should show plausible divergence.

Risk: Broadens scope and may delay cycle; requires revisiting model version.

What the batch taught (research notes, not defects)

Backcast: across 48 calibration nations, the correlation between observed and simulated change in the index from 1996 to 2026 is -0.09; the observed changes range from -5.6 to +6.5 points, the simulated from -2.8 to +0.7 [backcast_comparison].

Backcast: closest to history SAU (error 0.02), SGP (error 0.04), LKA (error 0.08), CZE (error 0.09); furthest CHN (observed +6.5, simulated -2.2), IND (observed +3.7, simulated -1.6), TUR (observed +1.3, simulated -2.6), VEN (observed -5.6, simulated -1.8) [backcast_comparison].

Backcast: the five fastest risers in history (CHN, IND, KOR, ARE, VNM) gained +3.5 index points on average; the model gives them -0.8. It has no mechanism for catch-up growth: income is calibrated to the cost of the starting position, so a nation that starts low stays poor in credits [backcast_comparison].

Across the batch, the graph's busiest edges are Economic Output & Productivity to Fiscal Capacity & Public Finance (-0.28 points a run); Transport Infrastructure to Trade Facilitation & Logistics (+0.21 points a run); Energy Infrastructure to Advanced Manufacturing (+0.16 points a run) [top_edges].

This batch exposes a model that is structurally mean-reverting and often too pessimistic for low-rated nations. Across the 50 scripted runs, 37 of 50 median index values at the horizon are below their 2026 start, while only 13 are above; the p5-p95 intervals are narrow, typically under 0.2 index points, so the scripted policy and event stream produce little distributional spread. High-rated nations also saturate: e4-sgp has 100% of runs ending with a median of four groups at 20 that did not start there, with similar warnings for IRL, TWN, ISR and FIN. Two experiments show implausibly large one-year moves with no shock: ISR 4.45 and RUS 4.11 points. The backcast against history is the most concerning: correlation between observed and simulated 30-year change is -0.09; observed changes range from -5.6 to +6.5, simulated only -2.8 to +0.7. The five fastest historical risers gained 3.5 index points on average; the model gives them -0.8. Domain-level backcast biases are systematic: trade_and_investment -1.29, financial_strength -1.22, information_and_influence -1.22, production_and_innovation -1.12. These are distributional patterns, not single-run artefacts. The model lacks catch-up growth and simultaneously saturates high-rated groups because frontier costs are too cheap. The next cycle should prioritise the income/catch-up mechanism and frontier saturation before trusting rank or tier movements.