Multi-agent Planning using Visual Language Models

ECAI 2024

Grab the disc from the trash bin, turn on the lamp on the dresser (trial_T20190907_114323_767231)

Ground truth Single Agent with table Multi Agent with table Single Agent with image Multi Agent with image
STEP 1: turn right, turn left, walk to the corner with a trash bin
STEP 2: grab the disc from the trash bin
STEP 3: turn around, walk to the dresser
STEP 4: turn on the lamp on the dresser
Turn around and walk to the garbage can at position (2.257, -0.008, 2.603).,
Take the CD from the garbage can.,
Turn around and walk to the desk lamp at position (-0.669, 0.923, -0.062).,
Turn on the desk lamp.
Walk to the GarbageCan, ID 27.,
Bend down and grab the CD from the GarbageCan, ID 27.,
Stand up and turn around to face the dresser.,
Walk to the dresser with the DeskLamp, ID 22.,
Turn on the DeskLamp on the dresser, ID 22.
1. Walk to the trash bin located near the bottom right corner of the bed.
2. Reach down to grab the disc from the trash bin.
3. Turn around to face the dresser with the lamp on it, located to the left of the bed.
4. Walk to the dresser with the lamp.
5. Reach out to turn on the lamp on the dresser.
1. Walk to the trash bin near the desk.
2. Look into the trash bin to locate the disc.
3. Reach into the trash bin and grab the disc.
4. Walk to the dresser near the bed.
5. Locate the switch on the lamp on the dresser.
6. Flip the switch to turn on the lamp.
Architecture