Privacy Leakage Study and Protection for Virtual Reality Devices

Advisor: Dr. Yingying (Jennifer) Chen

Mentors: Changming Li^GR, Honglu Li^GR, Tianfang Zhang^GR

Team: Dirk Catpo Risco^GR, Brandon (Jinu) Son^UG, Brody^HS, Emily Yao^HS

Final Poster with Results

Project Overview

Augmented reality/virtual reality (AR/VR) is used for many applications and have been used for many purposes ranging from communicating and tourism, all the way to healthcare. Accessing the built-in motion sensors does not require user permissions, as most VR applications need to access this information in order to function. However, this introduces the possibility of privacy vulnerabilities: zero-permission motion sensors can be used in order to infer live speech, which is a problem when that speech may include sensitive information.

Project Goal

The purpose of this project is to extract motion data from AR/VR devices inertial measurement unit (IMU), and then input this data to a large language model (LLM) to predict what the user is doing

Weekly Updates

Week 1

Progress

• Read research paper [1] regarding an eavesdropping attack called Face-Mic

Next Week Goals

• We plan to meet with our mentors and get more information on the duties and expectations of our project

Week 2

Progress

• Read research paper [2] regarding LLMs comprehending the physical world
• Build a connection between research paper and also privacy concerns of AR/VR devices

Next Week Goals

• Get familiar with AR/VR device:
  • Meta Quest
  • How to use device
  • Configure settings on host computer
• Extract motion data from IMU
  • Connecting motion sensor application program interface (API) to access data
  • Data processing method

Week 3

Progress

• Set up host computer and android studio environment
• Started extracting data from the inertial measurement unit (IMU)
• Recorded and ran trials of varying head motions

Next Week Goals

• Run more tests to collect more data
• Design more motions for data collection *Different head motions
  • Rotational
  • Linear
• Combinations of head motions
  • Looking left and then right
  • Looking up and then down

Week 4

Progress

• Placeholder

Next Week Goals

• Placeholder

Week 5

Progress

• Placeholder

Next Week Goals

• Placeholder

Week 6

Progress

• Placeholder

Next Week Goals

• Placeholder

Week 7

Progress

• Placeholder

Next Week Goals

• Placeholder

Links to Presentations

Week 1 Week 2 Week 3 Week 4 Week 5 Week 6 Week 7 Week 8 Final Presentation

References

[1] Shi, C., Xu, X., Zhang, T., Walker, P., Wu, Y., Liu, J., Saxena, N., Chen, Y. and Yu, J., 2021, October. Face-Mic: inferring live speech and speaker identity via subtle facial dynamics captured by AR/VR motion sensors. In Proceedings of the 27th Annual International Conference on Mobile Computing and Networking (pp. 478-490).

[2] Xu, H., Han, L., Yang, Q., Li, M. and Srivastava, M., 2024, February. Penetrative ai: Making llms comprehend the physical world. In Proceedings of the 25th International Workshop on Mobile Computing Systems and Applications (pp. 1-7).