Multi-agent Planning using Visual Language Models

ECAI 2024

Place a washed sponge in a tub (trial_T20190906_234735_610018)

Ground truth Single Agent with table Multi Agent with table Single Agent with image Multi Agent with image
STEP 1: Walk forward to the tub.
STEP 2: Pick up the sponge in the tub.
STEP 3: Turn around and veer left to the sink.
STEP 4: Wash the sponge in the sink.
STEP 5: Turn around a veer right to the tub.
STEP 6: Put the sponge in the tub to the left of the red rag.
END
Turn around and walk to the countertop.,
Take the dish sponge from the countertop.,
Turn around and walk to the sink.,
Wash the sponge in the sink.,
Turn around and walk to the bathtub.,
Place the washed sponge in the bathtub.
Turn and walk to the CounterTop at position (-1.892, 0.704, -0.528).,
Take the DishSponge from the CounterTop.,
Turn around and walk to the Sink at position (-1.919, 0.441, 2.032).,
Turn on the Faucet at position (-1.919, 1.064, 2.182) and wash the DishSponge.,
Turn off the Faucet.,
Turn and walk to the Bathtub at position (-3.706, 0.342, 0.636).,
Place the washed DishSponge in the Bathtub.
1. Walk to the sponge located on the edge of the bathtub.
2. Pick up the sponge.
3. Turn to face the tub.
4. Place the sponge in the tub.
1. Identify the location of the sponge (sponge is on the countertop).
2. Walk to the countertop (adjacent to the tub).
3. Pick up the sponge from the countertop.
4. If the sponge is not already washed, walk to the sink (next to the countertop).
5. Wash the sponge under the sink.
6. Walk back to the tub (next to the sink).
7. Place the washed sponge in the tub.
Architecture