{{{#!comment See (https://www.orbit-lab.org/wiki/WikiFormatting) for wiki formatting }}} [[TOC(Other/Summer/2023/SelfDriving/*, depth=1, heading=Self Driving Vehicular Project)]] = Self Driving Vehicular Project = **WINLAB Summer Internship 2023** **Group Members:** Adarsh Narayanan, Brandon Cheng, Christopher Sawicki, Joshua Menezes, Ruben Alias, Tommy Chu, Aleicia Zhu, Ranvith Adulla, Arya Chhabra, Suhani Sengupta == Project Objective == This project has several objectives involving the WINLAB miniature intersection. ||= RASCAL \\ (Robotic Autonomous Scale Car for Adaptive Learning) =||= SCAMP \\ (Self-guided Computer Assisted Mecanum Pathfinder) =|| || Continue work on the miniature autonomous car hardware project developed last summer (see the project page here: [https://www.orbit-lab.org/wiki/Other/Summer/2022/Hardware]). \\\\ The goal for this summer is to finish prototyping the hardware and develop drivers to control the car with ROS. \\ If time allows, once the ROS integration is complete, we will be able to develop and test machine learning models to control the car. || Develop a remote-controllable car which can be used to mimic the path taken by a car in the real intersection. \\\\ For this platform, mecanum wheels will be used to allow for real-time corrections to the path.\\ The platform will be developed using Robot Operating System (ROS). \\ Students will need to prototype hardware and develop software to extract the path from video of the real intersection, translate that into a path for the miniature intersection, and produce a set of commands for the car. || *RASCAL pseudonyms: `UpCar`, `BARTHOLOMEW` *SCAMP pseudonyms: `MIMIC`, `CORNELIUS` === Week 1 === [https://docs.google.com/presentation/d/1xI1j6bT0DHHZn49lkX46HIa0S8HK6EFORhHdp0s7ubw/edit?usp=sharing Week 1 Slides] ||= RASCAL =||= MIMIC =|| || Worked on getting RASCAL driving with a controller \\ Was able to parallel park || Started designing MIMIC hardware setup || === Week 2 === [https://docs.google.com/presentation/d/1iGiPmPWDiUlBCvQfNxJomSGOvTsIeXy3BlOZD4x3UdU/edit?usp=drive_link Week 2 Slides]: ||= RASCAL =||= MIMIC =|| || Switched from PyCharm to Vim due to storage issues \\ Started implementing ROS || Started wiring with IMU, Jetson Nano, Teensy, four brushless \\ motors, and two motor drivers || === Week 3 === [https://docs.google.com/presentation/d/1P8vwjlIrdJqn0JxDZvIFFVKU4G3govuB-HsCFOkXvmc/edit?usp=drive_link Week 3 Slides]: ||= RASCAL =||= MIMIC =|| || Started setting up Rosbags to collect data to \\ eventually train an ML model || Set up ROS and serial control || === Week 4 === [https://docs.google.com/presentation/d/1e9mE7dmLPfX8NrHO0R2h8viQOAZ2b9qCVxmgUfG48ig/edit?usp=drive_link Week 4 Slides]: ||= RASCAL =||= MIMIC =|| || Created Flutter app to clean data from camera \\ Set up scripts for converting .bag files to .csv || Set up controls with PS3 controller to prevent motor stall \\ Created web server to track and simulate car movement || === Week 5 === [https://docs.google.com/presentation/d/1YZyKdFTEkPWX_MtU9aXOE59pvN-EeOz8yq32NEeIPb8/edit?usp=drive_link Week 5 Slides]: ||= RASCAL =||= MIMIC =|| || Worked on creating imitation learning model to drive autonomously || Created an electrical schematic \\ Started working on PID for motor control || === Week 6 === [https://docs.google.com/presentation/d/1ZR2ZkWQStdnRA0vMXVnJyFXjbEZQFBcmicK7ssUWeV8/edit?usp=sharing Week 6 Slides]: ||= RASCAL =||= MIMIC =|| || Started designing and testing for hardware issues with v2 (BARTHOLOMEWTWO) \\ Set up IMU for better odometry and position tracking || Moves precise distances and angles for path following || === Week 7 === [https://docs.google.com/presentation/d/1Um9QVwXIs2ysFlWj7XCzzLwgOrnQ015TrQP_p2XK10I/edit?usp=sharing Week 7 Slides] ||= RASCAL =||= MIMIC =|| || Finished designing BARTHOLOMEWTWO and successfully ran the imitation learning model || Able to follow a path drawn on the web server || === Week 8 === [https://docs.google.com/presentation/d/1a7_UKj993JpUqb-6AmBOYRsbLaYymeJoxLLY6LJCFNw/edit?usp=sharing Week 8 Slides] ||= RASCAL =||= MIMIC =|| || Tested pure pursuit and normalized ML inputs || Implement full omni-drive motion for path following || === Week 9 === [https://docs.google.com/presentation/d/1YZyKdFTEkPWX_MtU9aXOE59pvN-EeOz8yq32NEeIPb8/edit#slide=id.g25642a0be74_0_374 Week 9 Slides] === Week 10 === [https://docs.google.com/presentation/d/1_JRd1HWG_AiUundypvOmt649NiOR2pexfJPSa3DTZS0/edit#slide=id.g25fc006d2f0_0_6 Week 10 Slides] == Project Setup Instructions == Set up SSH key for Gitlab - If you clone the repository with HTTPS, then you don't need to do this. However, then you will need to input your username and password every time you want to fetch or push from gitlab. To get around this, you can generate an SSH key pair and upload the public key to gitlab. - navigate to your .ssh directory by doing "cd ~/.ssh" - run the command "ssh-keygen". - It will prompt you to name the file. For example, "gitlab". - It will then prompt for a passphrase for the key. Leave it empty if you don't want to put in a password every time you use it, or put one for security if you prefer. - Two files called "gitlab" and "gitlab.pub" will be created in the ~/.ssh directory now. We need to upload the public key to gitlab. - Navigate to your .ssh directory and use the command "cat gitlab.pub" to print the contents of the file into the terminal. Copy all of the file, including the "sha-rsa" at the beginning and the username at the end. - Login to gitlab (the winlab specific gitlab! https://gitlab.orbit-lab.org) and click on your profile picture on the top right. - Go to SSH keys and add the public key in the text box - Now go to ~/.ssh and add a file called "config" by doing "touch config". Edit this file (with nano or vim) to look like this: Host gitlab.orbit-lab.org PreferredAuthentications publickey IdentityFile ~/.ssh/gitlab - make sure the file path goes to the private key that you made, if you named it something other than gitlab. - You should now be able to access orbit gitlab repositories through ssh. - NOTE: If you have previously cloned a repository using HTTPS instead of SSH, then you will need to update the remote URL. Follow this guide: https://docs.github.com/en/get-started/getting-started-with-git/managing-remote-repositories Project instructions: - Clone the desired repository. You can click "clone" on the repository page for the desired repository on Gitlab (https://gitlab.orbit-lab.org/self-driving-car-2023). For example, "git clone git@gitlab.orbit-lab.org:self-driving-car-2023/upcar.git" - - Make sure you set up ROS in your .bashrc file. You need to have "source /opt/ros/noetic/setup.bash" in it - You need to add the setup file for the catkin workspace of the project in your .bashrc file. To do this, first take note of the absolute path to the project directory. You will need to add "source path_to_catkin_ws/catkin_ws/devel/setup.bash" (replace path_to_catkin_ws with the actual absolute path on your system.) - One you are done changing the setup files, logout and login. - Install the joy package by following this tutorial: http://wiki.ros.org/joy/Tutorials/ConfiguringALinuxJoystick - The user needs to have input permission for ROS to read data from joystick inputs. To set this, do "sudo usermod -aG input username". -aG specifies adding an additional group (input) to a user (username) - Changing groups requires the user to log out and in for effects to take place - Install the realsense package (Add Link!) ROS setup instructions: - Follow these instructions to set up ROS Noetic: http://wiki.ros.org/noetic/Installation/Ubuntu - Note: any time you edit a startup file, like .bashrc, you need to logout and login (on VM you might need to power it off and on) for the effects to take place. - ROS Noetic is used for Ubuntu 20.04 (on RASCAL's Intel UpBoard), ROS Melodic is used for Ubuntu 18.04 (on SCAMP's and new RASCAL's Jetson Nano) Using ROS: - To run a ROS node: use "rosrun package nodename". For example, "rosrun joy joy_node" to run the joystick package. - roscore must be running first by typing "roscore" in the terminal. - To run multiple processes at once, you can use Tmux (terminal multiplexer). More advanced uses can be found online, but the basic usage is as follows: - use "sudo apt install tmux" to install it. - use "tmux" to launch. hit ctrl+b and then % to split vertically, and ctrl+b and then " to split horizontally. - ctrl+b and then arrow key lets you switch between panes - Mouse can be enabled if you use a tmux configuration file. Follow this: https://unix.stackexchange.com/questions/516800/how-do-i-enable-tmux-mouse-support - To run a ROS launch file, use "roslaunch package launchfile.launch". For example, "roslaunch rascal sim.launch". - on the first launch, or any time you change build files (CMakeLists, package.xml) you will need to navigate to the catkin workspace directory and run "catkin_make". - To create a new package: http://wiki.ros.org/ROS/Tutorials/CreatingPackage - To create a new python file that can be run through rosrun: Create the file somewhere within the package's src directory. Make sure the shebang is in the beginning ("#!/usr/bin/env python"). Then add the file to the package CMakeLists.txt in catkin_install_python. - To be able to import python modules from within the same package: make sure catkin_python_setup() is uncommented in the package's CMakeLists.txt. Then reference the directory structure visible in the rascal package. Make sure to include the "setup.py" file in the package's directory. VM setup instructions: - Download virtualbox https://www.virtualbox.org/wiki/Downloads - Download ubuntu https://releases.ubuntu.com/focal/ - Follow a guide to install ubuntu on Virtualbox: https://www.wikihow.com/Install-Ubuntu-on-VirtualBox - If you are on mac, MAKE SURE you start Virtualbox by typing "sudo virtualbox" in the commandline, instead of starting it normally. This is the only workaround I could find to get controller input to work in virtualbox! - If you are using Ubuntu 22.04, then there is a bug when using it with VirtualBox where you can't open the terminal by default. You need to update the locale of the machine for some reason. Follow this link to get into terminal mode: https://superuser.com/questions/100693/how-to-switch-to-non-graphical-view-in-ubuntu - Then follow this to update the locale: https://askubuntu.com/questions/1435918/terminal-not-opening-on-ubuntu-22-04-on-virtual-box-7-0-0 - Restart the VM (power it off and on in VirtualBox) and now you should be able to open the terminal in Graphical mode - You will need to have Sudo access on your account in Ubuntu. Follow this StackOverflow: https://askubuntu.com/questions/124166/how-do-i-add-myself-into-the-sudoers-group - Run "sudo apt update" to update packages before doing things. - If you want to be able to ssh into the VM through your local machine, you need to set up port forwarding on the VM. Follow this guide: https://help.skytap.com/connect-to-a-linux-vm-with-ssh.html To run the simulation on a VM: - Switch to the simulation branch by using "git checkout simulation" (in the upcar repository) - Joystick needs to be configured in virtualbox. If you are using mac, the VirtualBox must be run as sudo. - Follow these instructions: https://askubuntu.com/questions/25596/how-to-set-up-usb-for-virtualbox - The joystick needs to be set. navigate to /dev/input. if you do "ls -a" in here, you should see js0 and js1 which are added by virtualbox by default, and then js2 if your controller is configured correctly. Run "sudo jstest /dev/input/js2" to test if it is working. - You need to open a port to be able to view the web display on the host machine of the VM. - Open virtualbox and go to the settings for the VM. - Go to network -> Adapter 1 -> make sure it is on NAT - click advanced -> port forwarding -> add a new port forwarding rule - Fill the following fields (you may need to scroll right): name: flask, Host port: 5001, Guest port: 5000 - note: you can use any unused port instead of 5001. 5000 is used for the guest port as it is the default for flask. - you may need to restart the VM. - run simlaunch.sh - To view the website, go to 127.0.0.1:5001 on your web browser Other Notes: - For cars built on the Jetson Nano (ie SCAMP and new RASCAL), the use of ROS Melodic requires that any python files need to be marked as executable. To do this, run "sudo chmod +x path/to/file.py". This adds execute permission (+x) to the file. - If you run out of space on ubuntu vm: https://technology.amis.nl/platform/virtualization-and-oracle-vm/ubuntu-vm-virtualbox-increase-size-disk-make-smaller-exports-distribution/ ==== Instructions for MIMIC (aka SCAMP) ==== Clone repository into /opt ===== Teensy 4.0 Setup: ===== 1. copy the links for the downloadable tar files for arduino-cli and teensy_loader_cli -> https://github.com/arduino/arduino-cli/releases -> https://github.com/PaulStoffregen/teensy_loader_cli/releases 2. Use **wget** to download the tar files and extract them with **tar -xvf {filename}** - Follow the instructions on the teensy_loader_cli github (Run `sudo apt-get install libusb-dev` for Ubuntu, and `make` to compile to an executable) 3. Move the two executable files to "/bin" to add it to path (makes it so that the executable can be run from any directory) -> This allows the **./teensy-tool.sh** file in mimic/teensy to compile and upload to the teensy 4. Refer to https://arduino.github.io/arduino-cli/0.33/getting-started/#adding-3rd-party-cores to add Teensy support to arduino-cli -> You may have to create a file called arduino-cli.yaml in ~/.arduino15 with information about additional urls, format is provided in the link above -> https://www.pjrc.com/teensy/td_download.html -> https://www.pjrc.com/teensy/package_teensy_index.json 5. Follow the instructions to create the 00-teensy.rules file: -> https://www.pjrc.com/teensy/00-teensy.rules 6. Use **arduino-cli board list** command to find the port of the Teensy -> Modify the sketch.yaml file in the ./teensy directory to makes sure default_port and default_fqbn matches what is listed from **arduino-cli board list** 7. Upload teensy code by running **./teensy-tool -bu** in mimic/teensy/ to build and upload the code -> additionally, you can add the -m flag to open the monitor as well ===== ROS Melodic Setup: ===== Uses ROS Melodic for Ubuntu 18.04 on the Jetson Nano 1. Refer to http://wiki.ros.org/melodic/Installation/Ubuntu for install 2. Add the following lines to ~/.bashrc - source/opt/ros/melodic/setup.bash - source/opt/mimic/devel/setup.bash 3. cd into mimic repo and run **catkin_make** to create ROS environment 4. Run ROS nodes by running **./roslaunch.sh** in mimic/ -> check ports for teensy for serial communication -> check joystick connection 5. Making new nodes/msg -> List out .py files and .msg files as needed in the CMakeLists.txt file located in the package folder (ie test_pkg) -> Run **catkin_make** to refresh environment messages/nodes ===== Integration with Web Display: ===== 1. Use **ip a** to find the ip of the Jetson Nano 2. Open a web brower and go to {ip}:5000 to view ===== Troubleshooting ===== 1. "SerialException: Device reports readiness to read but returned no data" ROS will not be able to communicate with serial if there is another service using the port, i.e. Serial Monitor - Use lsof to check with programs are using the port `lsof /dev/ttyACM0` - Kill the program that is using the port `kill {PID}` 2. "socket.error: [Errno 48] Address already in use" Flask server was not properly stopped before starting it again - Close the connection `sudo fuser -k 5000/tcp` - [https://stackoverflow.com/questions/34457981/trying-to-run-flask-app-gives-address-already-in-use More Info]