Evaluation Protocol
Reference hardware for current leaderboard runs and the scoring methodology used to rank models on holdout cases.
Reference Environment
Most benchmarks have completed in roughly 2–42 minutes of training wall time. Training and inference times on the leaderboard are informational only and not used for ranking.
How we score results
Rankings reward models that are accurate on both engineering scalars and simulation fields. A high combined score requires strong performance on holdout cases the model did not train on, such as unseen angles of attack or hole diameters.
Holdout inference
Each submission is evaluated on hidden test inputs. Ground truth is never exposed during inference. Predictions are compared to simulation reference values after the run completes.
Global scalar accuracy
For decision-driving quantities (Cl, Cd, peak von Mises stress, stress concentration factor, etc.), each holdout case gets an accuracy score: max(0, 1 − |predicted − truth| / |truth|). We average these across all global metrics and holdout cases for the submission.
Nodal field R²
Mesh field outputs (pressure, velocity components, stress tensors, etc.) are scored with R² on nodal values. Negative R² from poor fits is clamped to zero before averaging.
Combined score (geometric mean)
The headline score for a benchmark run is the geometric mean of mean global accuracy and mean field R². This penalizes models that excel on scalars but miss fields, or vice versa.
Domain and leaderboard rollups
Domain columns (Structural, Fluid dynamics, etc.) average combined scores across benchmarks in that domain. Overall leaderboard rank uses the cross-benchmark average combined score.
What is not ranked
Training time, inference time, and hardware core type are recorded for context but do not affect rank. See submission scores on the leaderboard for per-run breakdowns.
Submission scoresFor each model on a benchmark:
combined = √(global_avg × field_r²_avg)
Both inputs are on a 0–1 scale. Example: 0.99 global accuracy and 0.97 mean field R² → combined ≈ 0.98.