Multi-agent Planning using Visual Language Models

ECAI 2024

Carry a CD while turning on a desk lamp (trial_T20190907_114323_767231)

Ground truth Single Agent with table Multi Agent with table Single Agent with image Multi Agent with image
STEP 1: Move to the trash bin against the wall underneath the mirror
STEP 2: Pick up the CD that is sitting in the trash bin
STEP 3: Carry the CD to the right side of the wood desk where the desk lamp is sitting
STEP 4: Turn on the desk lamp
Turn around and walk to the GarbageCan at position (2.257, -0.008, 2.603).
Pick up the CD from the GarbageCan.
Turn around and walk to the DeskLamp at position (-0.669, 0.923, -0.062).
Turn on the DeskLamp.
Walk to the location of the CD at position (2.225, 0.049, 2.634).,
Pick up the CD from receptacle 27.,
Turn around and walk to the desk lamp at position (-0.669, 0.923, -0.062).,
Turn on the desk lamp.
1. Walk to the desk on the left side of the room.
2. Pick up the CD located on the left side of the desk.
3. Turn to face the desk lamp in the center of the desk.
4. Move to the desk lamp.
5. Turn on the desk lamp.
1. Walk to the desk located in the center of the room, in front of the window and next to the bed.
2. Pick up the CD located on the desk.
3. Turn on the desk lamp located on the right side of the desk.
Architecture