Privacy Leakage Study and Protection for Virtual Reality Devices

Advisor: ​Dr. Yingying (Jennifer) Chen

Mentors: ​Changming Li^GR, ​Honglu Li^GR, and ​Tianfang Zhang^GR

Team: ​Dirk Catpo Risco^GR, ​Brandon (Jinu) Son^UG, Brody^HS, and Emily Yao^HS

​Final Poser With Results

Project Overview

Augmented reality/virtual reality (AR/VR) is used for many applications and have been used for many purposes ranging from communicating and tourism, all the way to healthcare. Accessing the built-in motion sensors does not require user permissions, as most VR applications need to access this information in order to function. However, this introduces the possibility of privacy vulnerabilities: zero-permission motion sensors can be used in order to infer live speech, which is a problem when that speech may include sensitive information.

Project Goal

The purpose of this project is to extract motion data from AR/VR devices inertial measurement unit (IMU), and then input this data to a large language model (LLM) to predict what the user is doing

Weekly Updates

Week 1

Progress

• Read research paper [1] regarding an eavesdropping attack called Face-Mic

Next Week Goals

• We plan to meet with our mentors and get more information on the duties and expectations of our project

Week 2

Progress

• Read research paper [2] regarding LLMs comprehending the physical world
• Build a connection between research paper and also privacy concerns of AR/VR devices

Next Week Goals

• Get familiar with AR/VR device:
  • Meta Quest
  • How to use device
  • Configure settings on host computer
• Extract motion data from IMU
  • Connecting motion sensor application program interface (API) to access data
  • Data processing method

Week 3

Progress

• Placeholder

Next Week Goals

• Placeholder

Week 4

Progress

• Placeholder

Next Week Goals

• Placeholder

Week 5

Progress

• Placeholder

Next Week Goals

• Placeholder

Week 6

Progress

• Placeholder

Next Week Goals

• Placeholder

Week 7

Progress

• Placeholder

Next Week Goals

• Placeholder

Links to Presentations

Week 1   Week 2   Week 3   Week 4   Week 5   Week 6   Week 7   Final Presentation  

References

[1] ​Shi, C., Xu, X., Zhang, T., Walker, P., Wu, Y., Liu, J., Saxena, N., Chen, Y. and Yu, J., 2021, October. Face-Mic: inferring live speech and speaker identity via subtle facial dynamics captured by AR/VR motion sensors. In Proceedings of the 27th Annual International Conference on Mobile Computing and Networking (pp. 478-490).

[2] ​Xu, H., Han, L., Yang, Q., Li, M. and Srivastava, M., 2024, February. Penetrative ai: Making llms comprehend the physical world. In Proceedings of the 25th International Workshop on Mobile Computing Systems and Applications (pp. 1-7).