Data card

Summary

• 2,409 unique models and 4,818 paired language tasks.
• 2,000 Core models stratified into four equal difficulty tiers.
• 409 disjoint Long-Horizon Challenge models.
• 400-model dev split and 1,600-model test split.
• English abstract and detailed descriptions for every model.

Source and transformation

Records originate from Onshape parametric modeling histories. The conversion pipeline normalizes sketches, profiles, feature dependencies, solid operations, and refinement operations into executable macro actions. Candidates with unsupported features or failed isolated replay are excluded. Accepted records must produce a valid B-Rep in CadQuery/OpenCASCADE.

Identifiers

Public model IDs are anonymous stable IDs of the form onshape-xxxxxxxxxxxx. Private source-number mappings and local paths are not included. Abstract and detailed task IDs share the same model ID.

Annotation status

This preview contains the initial v2.2 language annotations. A replacement annotation pass uses seven rendered views, deterministic STEP geometry summaries, a vision extraction model, a separate writer, and a vision-based verification gate. Only records that pass the gate enter the refreshed release.

Known limitations

• The public repository cannot transfer rights in underlying third-party CAD designs.
• Test and Challenge ground truth is withheld, so official geometric scoring requires the private evaluator or a future submission service.
• Language descriptions may allow multiple geometrically valid answers.
• Current operations cover sketch, extrude, revolve, fillet, chamfer, and shell; the benchmark is not a complete representation of all commercial CAD tools.
• Exact topology is not a reliable semantic-equivalence criterion and is not the primary metric.

Leakage prevention

Only the task instruction, initial state, text level, and description may be shown to an evaluated agent. Dev oracle programs are included for engineering and reproducibility but must be removed from model context.