Multi-agent Planning using Visual Language Models

ECAI 2024

Examine a disc in the light of a lamp (trial_T20190907_114323_767231)

Ground truth Single Agent with table Multi Agent with table Single Agent with image Multi Agent with image
STEP 1: Turn right and begin walking across the room, then hang a left and walk over to the small bin below the mirror to the right of the door.
STEP 2: Pick up the disc that is in the small bin in the corner of the room.
STEP 3: Turn right and walk towards the bed, then hang a left and walk over to the lamp on the wooden desk.
STEP 4: Turn on the lamp on the wooden desk.
Turn around and walk to the GarbageCan at position (2.257, -0.008, 2.603).
Take the CD from the GarbageCan.
Turn around and walk to the DeskLamp at position (-0.669, 0.923, -0.062).
Examine the CD in the light of the DeskLamp.
Turn around and walk to the desk.
Take the CD from the desk.
Turn around and walk to the desk lamp.
Turn on the desk lamp.
Hold the CD under the desk lamp to examine it.
1. Walk to the desk located on the left side of the room.
2. Pick up the disc from the desk.
3. Turn to face the lamp on the bedside table next to the bed.
4. Walk to the bedside table with the lamp.
5. Hold the disc under the light of the lamp to examine it.
1. Obtain the disc from its current location.
2. Walk to the desk lamp located near the chest of drawers on the right side of the room.
3. If the lamp is off, turn on the desk lamp by toggling its switch.
4. Hold the disc with one hand.
5. Position the disc under the light of the desk lamp.
6. Rotate the disc slowly to examine it under the lamp's light from different angles.
Architecture