Submission Guide

Overview

Submissions use a BYOM (bring your own model) archive. Train on the public split, wrap your inference entry point in a small Python plugin, zip it with a manifest, and upload at Submit. The site launches an eval job on Rescale, runs your model on holdout inputs, and stores results when the job finishes.

Archive layout

One archive per submission. Required paths:

my-submission/
  submission.json       # Plugin path + training provenance (labels optional)
  plugin/
    __init__.py         # Your inference class (see sample below)
    model/              # Weights, checkpoints (optional path)
    ...                 # Any helper modules you need
  requirements.txt      # Optional pip deps for your plugin only

Supported formats: .tar.gz, .tgz, or .zip.

Size limit & what to pack

Archives are capped at 1 GB (compressed). Larger uploads are rejected at the launch step before the eval job starts. This is a temporary ceiling while we validate real submissions end-to-end; expect the cap to tighten once a fuller set of submissions is on record.

Rule of thumb: pack only what your plugin actually needs to run inference on a holdout case — nothing more. For reference, every model currently on the benchmark pages fits under 250 MB when trimmed to inference essentials. An analytical model with no weights (see the plate-with-hole smoke fixture) fits in a few KB.

Include

• plugin/__init__.py — your RescaleCustomModel subclass and any helper modules it imports.
• Whatever weights or artifacts your plugin loads at inference time — typically the trained checkpoint you want scored plus normalization / feature statistics. If you have multiple checkpoints (epoch snapshots, an AutoML sweep, a “best” and a “latest”), pick the one you want evaluated; you don't need to ship the others.
• Any config your plugin reads to rebuild the model (Hydra config.yaml or equivalent).
• requirements.txt at the archive root listing pip deps your plugin imports.
• A vendored SDK under plugin/vendor/ if your inference code depends on a library that isn't on PyPI.

Leave out

• Extra checkpoints your plugin won't load (epoch snapshots, AutoML trial sweeps).
• Training data, cached preprocessed inputs, and any reference-run prediction outputs (e.g. evaluation/, cached_data/, artifacts/).
• Training logs, TensorBoard event files, wandb runs, Hydra run history.
• Holdout meshes or ground truth (we provide inputs at eval time).
• Scoring or eval pipeline code (the judge runs separately on our side).

The eval runner treats your plugin as a black box. It doesn't care what framework you use (torch, onnx, jax, a REST call, an analytical formula) or what your directory layout inside plugin/ looks like — as long as plugin/__init__.py exposes exactly one RescaleCustomModel subclass and its three methods honour the contract below.

submission.json

Declares the plugin path and training provenance. On the website, the Submit wizard is authoritative for which benchmark to evaluate and how the run is labeled (model name, org, etc.) — values in this file do not need to match the wizard.

{
  "schemaVersion": 1,
  "benchmarkId": "plate-with-hole",
  "modelType": "byom",
  "pluginDir": "plugin",
  "displayName": "my-model-v1",
  "architecture": "Custom GNN",
  "architectureSummary": "Message-passing GNN on surface nodes; strong on smooth fields, weaker near stress concentrations.",
  "trainingTimeSeconds": 7200,
  "trainingHardwareSummary": "1 node · 1× NVIDIA A10 GPU",
  "trainingHardwareDetails": "32 GB RAM · 4 vCPU · 2nd gen AMD EPYC @ 2.8 GHz\nRescale grossular-1",
  "hyperparameters": { "learning_rate": 0.0001, "epochs": 500, "batch_size": 1 }
}

• pluginDir — folder with your inference class (default plugin).
• Optional benchmarkId / displayName — packaging notes only when you use Submit. The wizard dropdown and form fields win. Useful if you run the eval runner locally without the site.
• Optional architectureSummary — strengths, tradeoffs, and benchmark fit. Shown in the submission registry and on benchmark-page ℹ️ tooltips (same field as the wizard's “Architecture at a glance”). Legacy key notes still works.
• Optional trainingTimeSeconds — wall-clock training time in seconds (self-reported).
• Recommended training hardware — use trainingHardwareSummary for the short inline label (e.g. 1 node · 1× NVIDIA A10) and trainingHardwareDetails for extra lines in the hardware ℹ️ tooltip (RAM, CPU, core type). Legacy trainingHardwareDescription still works: first line = summary, following lines = details.
• Optional hyperparameters — JSON object of training knobs (learning rate, epochs, batch size, …) stored for dossier context.
• Rescale AI Physics exports use the matching fields in versionMetadata.yaml instead of this manifest.

plugin/ contract

Put your inference entry point in plugin/__init__.py. Define exactly one Python class with three methods: what inputs you need, what outputs you produce, and how to run one holdout case. At eval time we load that class, pass each holdout mesh and its parameters, and collect your predictions.

The base class (RescaleCustomModel) is already in the eval environment. If you trained with Rescale AI Physics, the same hooks exist under rescale_ai.models.custom. You do not copy our scoring code. You only implement inference.

Sample plugin/__init__.py (plate with hole)

from pathlib import Path

import numpy as np
from aiphysics_eval.byom_contract import (
    CustomInferenceResult,
    OutputSpec,
    ParameterSpec,
    RescaleCustomModel,
)


class MyPlateModel(RescaleCustomModel):
    requires_mesh = True

    def __init__(self, artifacts_dir: Path) -> None:
        super().__init__(artifacts_dir)
        # Load checkpoints from self.artifacts_dir / "model" / ...

    def describe_parameters(self) -> list[ParameterSpec]:
        return [ParameterSpec(name="hole_diameter", default=10.0)]

    def describe_outputs(self) -> OutputSpec:
        return OutputSpec(
            point_fields=["S_Mises"],
            global_outputs=["max_von_mises_mpa", "stress_concentration_factor"],
        )

    def run_inference(self, mesh, parameters: dict[str, float]) -> CustomInferenceResult:
        hole_mm = float(parameters["hole_diameter"])

        # Your model forward pass here (torch, onnx, etc.)
        peak_stress = ...
        kt = ...
        nodal_stress = ...  # length = mesh.n_points

        mesh["S_Mises_pred"] = np.asarray(nodal_stress, dtype=np.float64)
        return CustomInferenceResult(
            mesh=mesh,
            global_outputs={
                "max_von_mises_mpa": float(peak_stress),
                "stress_concentration_factor": float(kt),
            },
        )

• describe_parameters() names the design inputs for this benchmark (hole diameter here). Must match the benchmark spec.
• describe_outputs() lists globals and nodal fields you return. Nodal arrays usually use a _pred suffix on the mesh (for example S_Mises_pred).
• run_inference() receives the holdout mesh and one parameter dict. Return updated mesh fields plus global scalars.
• Only one model class per plugin/__init__.py. Other benchmarks use different parameter and field names. Check each benchmark page.

After upload

The success screen links to a track page for your job. When scoring finishes, metrics appear on the benchmark submissions table. Expand a row for per-case global accuracy, field R², relative L1, RMSE, and related breakdowns. See the evaluation protocol for what each metric means.