Distributed Data Infrastructure

---

Project Advisor: Professor Alexei Kotelnikov

Team Members: Keshav Subramaniyam, Jason Zhiyuan Zhang, Samyak Agarwal, Anna Kotelnikov, Gyana Deepika Dasara

---

Objective Of This Project

The objective of the project "Distributed Data Infrastructure focuses in general on developing a Ceph file system as well as stress testing the system's robustness and resilience. In order for us to be able to accomplish this goal, we are learning multiple concepts such as the Linux operating systems, file systems, networking, as well as Ansible.

Week 1

Summary

• Understood the goal of the project.
• Installed packages and wired a cluster of 8 servers.
• Got familiar with Linux by practicing some simple Linux commands.

Idea Of Our Progress This Week

Our main goal for this week was to spend time understanding the goal of the project and installing the necessary tools that would be used throughout the project, such as SSH, Orbit, Linux Packages, and eight cluster servers. We also got access to our Wireguard where we would be getting access to containers. We also installed and wired a cluster of 8 servers and issued some commands from our LXC containers in our terminal where we worked with Linux and understood some of the basic commands that would be useful for some more advanced work in the project. The final thing that we did was to install Debian and Ubuntu packages, removing Ubuntu packages with APT, and also searched and queried Ubuntu packages with APT.

Project Resources

Linux Exercises: ​https://linuxcourse.rutgers.edu/Winlab_Internship_2023/html/linux_commands.html Linux Package Installation: ​https://linuxcourse.rutgers.edu/Winlab_Internship_2023/html/debian_packaging.html

Week 2

Summary

• Installed Debian Linux in Hard Drives
• Solidified our understanding with networking as well as the Network File Systems (NFS).
• Worked with start and stop services.

Week 3

Summary

• Went over Shell Scripting and Python Scripting
• Worked With Distributed File Systems
• Performance Testing for ZFS and also MDRAID

Week 4

Summary

• Focus on Ceph installation
• Did some troubleshooting relating to OSDs

Presentation Links

• Week 1: ​https://docs.google.com/presentation/d/1-KpBGlFWiV0LEJFXrU6Nk_VxNchvado48oIbh6KHOdk/edit#slide=id.p
• Week 2: ​https://docs.google.com/presentation/d/1Pujv2HFNYLHuhLFh_0tSRZg4yv6gs8nNNyNNcMXD8s8/edit#slide=id.p
• Week 3: ​https://docs.google.com/presentation/d/1uBwrVGlGCIyD_Ex1miklX3PFOZ2JsUqF-Jm7kR1JiSY/edit#slide=id.p
• Week 4: ​https://docs.google.com/presentation/d/1VeIosZZ-GPm8SnB294C6rbX2b-hYHRSArvzhm4MPiY4/edit#slide=id.p