Conference Proceeding

Hacking Nonverbal Communication between Pedestrians and Vehicles in Virtual Reality

  • Henri Schmidt (Massachusetts Institute of Technology)
  • Jack Terwilliger (Massachusetts Institute of Technology)
  • Dina AlAdawy (Massachusetts Institute of Technology)
  • Lex Fridman (Massachusetts Institute of Technology)


We use an immersive virtual reality environment to explore the intricate social cues that underlie non-verbal communication involved in a pedestrian’s crossing decision. We “hack” non-verbal communication between pedestrian and vehicle by engineering a set of 15 vehicle trajectories, some of which follow social conventions and some that break them. By subverting social expectations of vehicle behavior we show that pedestrians may use vehicle kinematics to infer social intentions and not merely as the state of a moving object. We investigate human behavior in this virtual world by conducting a study of 22 subjects, with each subject experiencing and responding to each of the trajectories by moving their body, legs, arms, and head in both the physical and the virtual world. Both quantitative and qualitative responses are collected and analyzed, showing that, in fact, social cues can be engineered through vehicle trajectory manipulation. In addition, we demonstrate that immersive virtual worlds which allow the pedestrian to move around freely, provide a powerful way to understand both the mechanisms of human perception and the social signaling involved in pedestrian-vehicle interaction.

How to Cite:

Schmidt, H. & Terwilliger, J. & AlAdawy, D. & Fridman, L., (2019) “Hacking Nonverbal Communication between Pedestrians and Vehicles in Virtual Reality”, Driving Assessment Conference 10(2019), 78-84. doi:

Rights: Copyright © 2019 the author(s)

Download pdf
View PDF



Published on
25 Jun 2019
Peer Reviewed