Semantic Audio-driven Understanding for Dynamic Humanoid Whole Body Control

RoboCup Symposium 2026

J. Marcelo1, M. Brienza1, E. Bugli1, L. Comito1, D. Nardi1, D. D. Bloisi2, and V. Suriani1
1 Dept. of Computer, Control, and Management Engineering, Sapienza University of Rome, Rome (Italy)
2 Dept. of International Humanities and Social Sciences, International University of Rome, Rome (Italy)
Paper arXiv Code

Supplementary materials: github.com/Lab-RoCoCo-Sapienza/semantic-WBC

Video

Semantic audio-driven whole-body control (simulation and Unitree G1).

Abstract

Recent advances in humanoid robotics and reinforcement learning have enabled the acquisition of highly expressive whole-body motion policies. However, most robotic performances remain based on pre-scripted sequences or externally triggered behaviors, limiting autonomy and responsiveness to dynamic environments. In this work, we introduce a novel multi-modal orchestration framework for semantic audio-driven humanoid control, enabling robots to autonomously select and execute appropriate motion skills in real time. The system processes continuous audio streams and routes them into music or speech branches. Music input is handled via audio fingerprinting and semantic embeddings to retrieve track identity and temporal alignment, allowing dynamic mapping between musical segments and motion policies. Speech input is grounded into a discrete library of imitation-learned skills, enabling direct human-robot interaction. Both modalities share a unified interface that schedules skill execution over a reinforcement learning control pipeline. We validate the approach in simulation and on a Unitree G1 humanoid, showing robust sim-to-real transfer and consistent audio-conditioned policy selection. Supplementary materials are available at the following site: https://github.com/Lab-RoCoCo-Sapienza/semantic-WBC

Real World Validation

Building on the simulation results, the same framework is deployed on the physical Unitree G1 humanoid to verify that the control logic transfers to a real execution setting. The real-world demonstration confirms that chunk-level retrieval is sufficiently stable to drive consistent policy selection in a live scenario.

Simulation
Real robot (Unitree G1)

Riferimenti (BibTeX)

Blocco BibTeX per citare il lavoro da repository o articoli; aggiorna chiave, titolo, autori, venue, anno, DOI/arXiv.

@inproceedings{marcelo2026semantic,
  title={{Semantic Audio-driven Understanding for Dynamic Humanoid Whole Body Control}},
  author={Marcelo, J. and Brienza, M. and Bugli, E. and Comito, L. and Nardi, D. and Bloisi, D. D. and Suriani, V.},
  booktitle={{RCS}},
  year={2026},
  url={https://github.com/Lab-RoCoCo-Sapienza/semantic-WBC}
}