Resolves YES if, by Jan 1, 2028:
a publicly documented benchmark exists that evaluates physical (non-simulated) household tasks;
tasks have explicit success/failure criteria;
results are intended for comparison across different robots or systems (not a single-vendor demo).
Resolves NO otherwise.
1,000@VladMoor I think HomeRobot already satisfies the written criteria: section 3.2 evaluates physical object-moving tasks in a three-room apartment, Table 4 compares heuristic and RL systems, and Appendix C.2 defines success as completing all four stages within 300 steps. HomeRobot: Open-Vocabulary Mobile Manipulation
This predates the market; the description says a benchmark exists by the deadline, without requiring creation afterward. Does that qualify for YES now? If not, which criterion is missing? A newer cross-model example is PhAIL DROID, with physical household tabletop tasks and published success/failure scoring: https://phail.ai/benchmark/droid . I hold M50 YES, but do not assume these interpretations are settled.
