Multi-agent Planning using Visual Language Models

ECAI 2024

Examine a CD with a lamp (trial_T20190907_114323_767231)

Ground truth Single Agent with table Multi Agent with table Single Agent with image Multi Agent with image
STEP 1: turn right, walk across room towards dresser, turn left, walk across room, turn right, turn left, go to the mirror
STEP 2: pick up the CD that is in the trash below the mirror
STEP 3: turn around, walk across the room, turn right, walk across the room to the dresser
STEP 4: turn the lamp on
Turn around and walk to the GarbageCan at position (2.257, -0.008, 2.603).
Take the CD from the GarbageCan.
Turn around and walk to the DeskLamp at position (-0.669, 0.923, -0.062).
Examine the CD under the DeskLamp.
Turn and walk to the desk where the CD is located.,
Take the CD from the desk.,
Turn and walk to the desk where the DeskLamp is located.,
Turn on the DeskLamp.,
Hold the CD under the DeskLamp to examine it.
1. Walk to the desk located in the center of the room.
2. Pick up the CD that is on the desk.
3. Turn to face the lamp on the same desk.
4. Examine the CD with the lamp.
1. Walk to the door in the room.
2. From the door, walk to the bed in the room.
3. From the bed, walk to the desk near the bed.
4. Pick up the CD on the desk.
5. Examine the CD under the lamp on the desk.
Architecture