wiki:Other/Summer/2023/Inference

Version 92 (modified by tm897, 10 months ago) ( diff )

—

Resilient Edge-Cloud Autonomous Learning with Timely inferences

Project Advisor and Mentors: Professor Anand Sarwate, Professor Waheed Bajwa, Nitya Sathyavageeswaran, Vishakha Ramani, Yu Wu, Aliasghar Mohammadsalehi, Muhammad Zulqarnain

Team: Shreya Venugopaal, Haider Abdelrahman, Tanushree Mehta, Lakshya Gour, Yunhyuk Chang

Objective: The purpose of this project is to use ORBIT in order to design experiments which will analyze the training and prediction of various ML models across multiple edge devices. Additionally, students will develop a latency profiling framework for MEC-assisted machine learning using cutting edge techniques such as splitting model computing and early exit. They will then analyze these models for latency and accuracy tradeoff analysis, along with measuring network delays.

Overview

As machine learning models become increasingly advanced and complex, running these models on less powerful devices is becoming increasingly difficult, especially when accounting for latency. However, it is also not efficient to run everything using only the cloud as it creates too much traffic. A viable solution to this problem is edge computing, where we use the edge (the networks in between user devices and the cloud) for computation.

  1. Trained a small and large Neural Networks (DENSENET & Mobilenet V2) on the CIFAR10 dataset
  2. Performed PCA and SVM on NNs to familiarize ourselves with PyTorch
  3. Loaded the MNIST database (image) onto an orbit node
  4. Connected 2 nodes with client-server architecture and extracted data for time and accuracy measurements
  5. Compared performances of both neural networks on the CIFAR10 dataset
    • Established connection between two nodes
    • Communicated test data between nodes to compare accuracy and delay between our NN models
  6. Worked with professors/mentors and read papers to understand the concepts of early exit, split computing, accuracy/latency tradeoff, and distributed deep neural networks over the edge cloud

7.Split the NN ResNet 18 using split computing onto two different devices and ran an inference across a network

  1. Used Network Time Protocol (NTP) and sent data in "packages" (chunks) to collect latency and delay data

9.Explored different research questions with the data collected:

  1. Limited CPU power in terminal to imitate mobile devices
  2. Implemented different threshold values based on confidence for sending the data to the edge and server for inference
    • Generated graphs for threshold vs latency, accuracy vs latency, (LINK TO GRAPHS)
  3. Retrained neural network to achieve 88% accuracy and collected new graphs (LINK)
  4. Introduced a delay in the inference as well as data transfer to simulate a queue

Week 1

Summary

  • Understood the goal of the project and broke down its objectives
  • Got familiar with Linux by practicing some simple Linux commands.
  • Review some basic Python and other coding materials
  • Understanding the basis of Machine Learning models and algorithms

Next Steps

  • Create and train a “small” and “large” Neural Network
  • Attempt to simulate the difference between their performances at inference

Week 2

Summary

  • Performed basics of pattern recognition and Machine Learning (PPA - Patterns, Predictions, Actions) using Pytorch
  • Created a node image with Pytorch
  • Created small Machine Learning models
  • Loaded the Modified National Institute of Standards and Technology (MNIST) database onto the orbit node

Next Steps

  • Create and train a “small” and “large” Neural Network
  • Attempt to simulate the difference between their performances at inference

Week 3

Summary

  • Debugged issues with work we had done in previous weeks
  • Connected 2 nodes with client-server architecture
  • Extracted data for time and Accuracy measurements
  • Added logging to code and stored logs in a table format (csv)
  • Ready to start testing!

Next Steps

  • Compute the delay time for each step to a higher precision
  • Plot and identify trends in data (accuracy & latency)
  • Begin to implement Split Computing - 10 layer MLP, ResNet
  • Think about other implementations - Early Exiting, model compression, data compression, … Mixture?

Week 4

Summary

Next Steps

  • Divide data into “chunks” for faster and more efficient transmission
  • Design experiments to test different architectures and implementations of Early Exiting and Split Computing
  • Track and add Age of Information Metrics

Week 5

  • ntp

Week 6

Summary

  • Figured out how to properly split a NN using split computing
  • First Experimented with image inferencing on the same device
  • Split the Neural Network (ResNet 18) onto two different devices
  • Ran an inference with the split NN across a network

Next Steps

  • Take a step back
  • Ask: what questions do we want to answer?
  • As you vary the threshold for asking for help, how does the average latency change(over the dataset)?

Open ended

Transfer to Precision time protocol(PTP)

Week 7

Summary

  • Explored different research questions with the data collected
  • Limited CPU power in terminal to imitate mobile devices
  • Implemented different threshold values based on confidence for sending the data to the edge and server for inference
  • Generated graphs for threshold vs latency

Next Steps

  • Use PTP timing to get more precise time measurements
  • After analyzing latency, move on to accuracy tradeoffs using different thresholds

Links to Presentations

Week 1   Week 2   Week 3   Week 4   Week 5 - Cumulative   Week 6   Week 7  

Note: See TracWiki for help on using the wiki.