Multi-agent Planning using Visual Language Models

ECAI 2024

Put a cold pot on a table. (trial_T20190909_004531_429065)

Ground truth Single Agent with table Multi Agent with table Single Agent with image Multi Agent with image
STEP 1: Take a step forward, turn left, go forward and face right side of the oven.
STEP 2: Take the pot from the right side of the stove top.
STEP 3: Turn right, go forward a step, turn right, go forward, turn left at the fridge, go forward to the fridge.
STEP 4: Chill the pot on the top shelf of the fridge. Take the pot from the fridge.
STEP 5: Turn left, go forward, turn left at the square white table, go to the table.
STEP 6: Put the pot on the table.
Turn around and walk to the stove burner.,
Take the pot from the left stove burner.,
Turn around and walk to the dining table.,
Place the pot on the dining table.
Turn and walk to the stove burner at position (-1.552, 0.930, 0.785).,
Take the pot from the stove burner.,
Turn around and walk to the dining table at position (-0.350, -0.025, 0.857).,
Place the pot on the dining table.
1. Walk to the pot located on the stove to the left side of the kitchen.
2. Pick up the pot from the stove.
3. Turn right and walk to the table located in the center of the kitchen.
4. Place the pot on the table.
1. Walk to the stove where the pot is located.
2. Pick up the cold pot from the stove.
3. Turn towards the table, which is in front of the stove.
4. Walk to the table with the cold pot.
5. Place the cold pot on the table.
Architecture