Define the question.
Protect the test.
A specification for evaluating AI forecasts against future clinical disclosures. This page describes the intended protocol; it is not evidence of completed prospective model predictions.
Inspect the registry referencesA judgment becomes a record.
A result becomes a test.
The intended prospective workflow. Current research tooling and the forecast protocol are identified separately.
Define the question
One trial, one specified endpoint and a rule for what will count as an answer.
ProtocolDate the evidence
Keep a source-linked snapshot of what was available before the result.
Research records availableSubmit independently
Models make their first judgment without seeing the others. Abstention stays visible.
Planned forecast workflowPreserve the forecast
Retain the original submission; a revision becomes a separate version.
Planned forecast workflowResolve against the source
Use the predefined rule and official result. Unresolved questions keep waiting.
Planned adjudicationEvaluate over time
Compare probability scores over prospective outcomes, including the misses.
Scoring explainer available
A question needs an answer rule.
Specify the trial, population, primary endpoint, comparison and acceptable official source before any forecast is accepted. Multiple primary endpoints, protocol amendments and ambiguous disclosures require explicit rules or a decision not to forecast.
A revision cannot replace the original.
The intended record retains a source snapshot and the original submission. Revisions should receive their own identifiers and timestamps. A content hash checks that a file matches a fingerprint; it does not independently establish when the record existed or prove a scientific claim.
Independent means independent.
Each model’s first forecast must be produced without access to the other submissions. A sequence of roles using one model is useful for research drafting, but is not an independent cross-model forecasting benchmark.
Waiting is a valid state.
Questions without an official result remain unresolved. Withdrawals, changed endpoints and disclosures that do not answer the frozen question need an explicit void or unresolved decision, with the reason retained. They must not be silently scored as failures.
Results need a review trail.
A future adjudication should link to the official disclosure, apply the frozen criteria and allow review of disagreements. Accuracy and calibration claims require a sufficiently informative prospective record, with abstentions, exclusions and uncertainty disclosed.
What can be inspected today.
The research workspace persists data, tasks and uploaded files. Public-source collectors perform deterministic retrieval, numeric checks and descriptive summaries. A separate model research-drafting workflow requires authorized server configuration. Dedicated forecasting, independent time attestation and prospective outcome adjudication remain planned capabilities.
Open authorized research workspaceConfidence needs
a consequence.
Try a binary Brier score. Choose two probabilities, reveal an illustrative outcome and see how the score changes.
Illustrative inputs · not model forecasts
LOSS = (PROBABILITY − OUTCOME)²Forecast A receives the lower loss for this outcome.
Scores are displayed to four decimal places. A single result illustrates the scoring rule. Predictive ability requires many resolved, prospective questions; unresolved cases do not count as failures.