Augmented Reality Mural

WINLAB Summer Internship 2023

Group Members: Alexander Clarke, Ana Obradovic, Arya Shetty, Nicholas Clarke, Pavan datta Vedire Reddy

Project Objective

The objective of this project is to create a collaborative augmented reality application which will allow remote users to collaboratively create a virtual mural. This project will be created using the Unity game engine and deployed to AR platforms such as the Microsoft Hololens.

In addition to creating the software that will manage the generation and rendering of the mural, another important component of this project is designing an infrastructure that will allow for very low latency collaboration. If time allows, one of the end goals of this project is to deploy it using an "edge cloud" architecture using the OpenAR Cloud project.

Students should get started with this project by familiarizing themselves with the Unity game engine and Hololens development tools.

Week 1: Presentation Link: ​https://docs.google.com/presentation/d/1FjPUL7XrPDCdE972nUbW7S74IX70BCDgDVR_eASt9H0/edit?usp=sharing

• Started to familiarize ourselves with the Unity Software and Hololens.
• We started with connecting the Hololens to the Unity app through remote connection.
• Once we were able to see the project scene through the Hololens we tested limits of the environment

Week 2: Presentation Link: ​https://docs.google.com/presentation/d/1LQVxsd-889PpTrCxg7lbWw3VhyYNj-niy3aS0FdTTnw/edit?usp=sharing

• Set up essential MRTK (mixed reality toolkit) packages in Unity
• Downloaded and deployed app on Hololens
• Moved and resized objects
• Used eye tracking feature for selection
• Used voice commands to switch color of objects
• Added template menu
• Wrote code for drawing application using mouse
• Worked on iterations of line drawings (2D & 3D)

Week 3: Presentation Link: ​https://docs.google.com/presentation/d/1sy6CcWGTzTh1RKmW2iR24s1odf7g0WaoQa8a8A7ZQSw/edit?usp=sharing

• Resize and move rendered lines
• Highlight selected line objects
• Selection button in menu
• Rendering smooth curves with control points using splines
• Ability to erase rendered lines
• Learned about the client-server architecture
• Learned about communication protocols —> Transmission Control Protocol (TCP) & User Datagram Protocol (UDP)
• Coded a simple TCP Chat Room with multiple clients

Week 4: Presentation Link: ​https://docs.google.com/presentation/d/16PWZwUThJP8fDfp_JMGwXbcaIh0d8_6qrnV-ixCmmJA/edit?usp=sharing

• Message Queuing telemetry transport —> Setup broker, Connected one unity client to broker, Implements Publisher-subscriber method
• Singleton pattern for save system
• Serialize game objects and scenes
• Save and load structures
• Merged features like smooth line drawing, erasing and editing
• Tool selection and drawing resizing
• Brush settings and color selection menu

Week 5: Presentation Link: ​https://docs.google.com/presentation/d/1w9NXW46-_z1c2cesLaWOiW27Q5gIE-tJYjH5PGD_khY/edit?usp=sharing

• More work with MQTT, setting up unity client
• More work on save system, and transferring game object files to JSON format

Week 6: Presentation Link: ​https://docs.google.com/presentation/d/1QC8dDQiOfP1Gx0GNxpk489A0xPRhpyjq8akN6ApKZss/edit?usp=sharing

• Used MQTT to send JSON files from different devices
• Implemented saving lines by position change
• Fixed UI and optimized brushes
• Set up Node-Red to manage MQTT topics

Week 7: Presentation Link: ​https://docs.google.com/presentation/d/1I73XR4envKSiZ2wyW71QvwwqoPtDomBs0DI1XrCHSLs/edit?usp=sharing

• Two player collaboration
• File Server
• Allow for multiple users to draw incrementally
• New menu flow and shape tools

Week 8: Presentation Link: ​https://docs.google.com/presentation/d/1zA52CP9krTh-wP0SNyQAoHZ3nT4O-MO8huG1WbuqBug/edit?usp=sharing

• Fixed bugs to reduce lag
• Allowed for multiple users to draw simultaneously
• Mesh, shape, and drawing functionality refined
• Applied save/load features to different textures

Week 9: Presentation Link: ​https://docs.google.com/presentation/d/1yg9vJt0DxlAo2eEjRVaes9K5XTRtt0YS3PNbwkqtp7c/edit?usp=sharing

• Fixed more bugs
• Deployed working single player app on hololens
• Put broker on a linux VM hosted by orbit nodes
• Created presentation and poster for final week

Week 10: Presentation Link: ​https://docs.google.com/presentation/d/1yg9vJt0DxlAo2eEjRVaes9K5XTRtt0YS3PNbwkqtp7c/edit?usp=sharing

• ​https://youtu.be/V-7BNvZp4og
• Created demo video
• Final app testing
• Commenting code for next year