Evaluation metrics

Primary metric

shape_success evaluates the exterior surface after translation, rotation, and uniform-scale alignment. It combines a normalized Chamfer-style distance, surface agreement, and shape descriptors. This is the primary score because language descriptions can permit multiple valid dimensions and construction histories while still specifying the same recognizable part.

Validity and strict geometry

• execution_success: a non-empty, solid-containing, kernel-valid final B-Rep.
• dimensioned_geometry_success: scale-preserving aligned geometry.
• gt_frame_geometry_success: pose- and location-sensitive comparison in the reference coordinate frame.
• exact_topology_success: exact solid/face/edge counts plus volume tolerance; retained as a legacy diagnostic.
• exact_reconstruction_success: strict geometry, topology, dimensions, and alignment; audit-only because equivalent CAD can have different topology.

Program and interaction diagnostics

• command precision/recall/F1;
• longest-common-subsequence F1;
• sketch curve-type F1;
• successful calls, protocol errors, tool errors, and recovery attempts;
• decision horizon and compiled micro-actions;
• token use, wall time, termination reason, and explicit finalization.

Reporting contract

Report Core macro averages and each of T1–T4. Report Challenge separately. Break down abstract and detailed prompts. Every rate must state its numerator and eligible denominator; missing legacy fields must not be counted as failure. Record the complete geometry-threshold dictionary and environment version.