| Ground truth | Single Agent with table | Multi Agent with table | Single Agent with image | Multi Agent with image |
|---|---|---|---|---|
| STEP 1: Turn left and go through the kitchen to the front of the counter top on the left of the sink
STEP 2: Pick up the red tomato on the left side of the counter. STEP 3: Turn around and go to the front of the refrigerator. STEP 4: Chill the tomato in the refrigerator and take it out. STEP 5: Turn around and go left to the front of the microwave above the stove. STEP 6: Place the chilled tomato in the microwave and close it. |
Turn around and walk to the fridge., Take a tomato from the fridge., Turn around and walk to the microwave., Place the tomato in the microwave. | Turn around and walk to the fridge., Open the fridge door., Take a tomato from the fridge., Close the fridge door., Turn around and walk to the microwave., Open the microwave door., Place the chilled tomato in the microwave., Close the microwave door. | 1. Walk to the refrigerator located on the left side of the kitchen. 2. Open the refrigerator door. 3. Reach for the chilled tomato inside the refrigerator. 4. Close the refrigerator door. 5. Turn right and walk to the microwave situated on the countertop to the right of the sink. 6. Open the microwave door. 7. Place the chilled tomato inside the microwave. 8. Close the microwave door. 9. Operate the microwave to start heating the tomato. | 1. Walk to the countertop in front of the window. 2. Locate the tomato on the countertop, positioned to the left of the watermelon and to the right of the apple. 3. Pick up the tomato from the countertop. 4. Turn to face the microwave, which is next to the sink on the countertop. 5. Open the microwave door. 6. Place the tomato inside the microwave. 7. Close the microwave door. 8. Set the microwave to the desired settings for warming a chilled tomato. 9. Start the microwave. |