Multi-agent Planning using Visual Language Models

ECAI 2024

Examine a CD by a lamp (trial_T20190907_114323_767231)

Ground truth Single Agent with table Multi Agent with table Single Agent with image Multi Agent with image
STEP 1: Turn right and walk to the green stand then take a left and walk to the trash can that is on the right in the corner of the room
STEP 2: Pick up the CD that is in the trash can
STEP 3: Turn around and cross the room the green stand then turn right and head to the desk.
STEP 4: Turn on the lamp that is sitting on the desk
Turn around and walk to the GarbageCan at position (2.257, -0.008, 2.603).
Take the CD from the GarbageCan.
Turn around and walk to the DeskLamp at position (-0.669, 0.923, -0.062).
Examine the CD by the DeskLamp.
Turn around and walk to the desk., Take the CD from the right side of the desk., Turn around and walk to the desk lamp., Hold the CD under the desk lamp to examine it. 1. Walk to the desk located in the center of the room.
2. Pick up the CD from the desk.
3. Turn to the left to face the lamp on the bedside table.
4. Walk to the bedside table next to the bed.
5. Examine the CD by the lamp on the bedside table.
1. Walk to the desk located next to the bed. 2. Locate the lamp on the desk. 3. Locate the CD on the desk. 4. Pick up the CD from the desk. 5. Turn on the lamp if it is off. 6. Hold the CD under the lamp. 7. Examine the CD by rotating it slowly under the lamp light to check for any imperfections or damages.
Architecture