= Self Driving Vehicular Project = **Team:** Christopher Lee, Michelle Gutwein, Thierry Antoine, Varun Chirravuri, and Anjali Kapilavai [[BR]] **Advisors:** Ivan Seskar and Jennifer Shane[[BR]] **Project Description & Goals:**[[BR]] Build and train miniature autonomous cars to drive in a miniature city.[[BR]] **Technologies:** ROS (Robot Operating System), Pytorch ---- == Week 1 Read through the [https://docs.google.com/document/d/1KWIyvGMnB2nboosv068qp-X2BbNm9y4tUnWLzKbz6bI/edit?tab=t.0#heading=h.gz0otq9rwrdy manual] provided to us. [[BR]] \\ [https://docs.google.com/presentation/d/1ZJatipqqiaRQ06gbw_t5Jbho-dZOFxXZS6fXx_FxIIc/edit?slide=id.p#slide=id.p Week 1 slides]// == Week 2 Learned what ROS is and how to use it[[BR]] Also got to know the car, RASCAL[[BR]] \\ [https://docs.google.com/presentation/d/1VsER6V9A0wsgOowDd7vA26k3LNe6V8tMvsMXYWiOBVo/edit?slide=id.p#slide=id.p Week 2 slides]// == Week 3 Read through all of the code in the [https://gitlab.orbit-lab.org/self-driving-car-2023/upcar/-/tree/main/catkin_ws/src/rascal/src?ref_type=heads Gitlab] [[BR]] Learned how to record and edit data on the web display[[BR]] \\ [https://docs.google.com/presentation/d/1NL-Xl_2XD7dQo6XJMo8_wsapmv6N-sxQl58-IZ-0_L0/edit?slide=id.p#slide=id.p Week 3 slides]// == Week 4 Learned how to train a model[[BR]] Trained and tested the model on simple paths[[BR]] \\ [https://docs.google.com/presentation/d/10KDlyMXHnCqYHTe5A6nhevLICf3jup91HIhf3qXCt6U/edit?slide=id.g369586f805a_0_10#slide=id.g369586f805a_0_10 Week 4 slides]// == Week 5 Trained model on making right and left turns in the city environment[[BR]] Started training on turns at the intersection[[BR]] Created a new model called gps which will allow users to give the car commands on the direction to turn at an intersection before running the car[[BR]] \\ [https://docs.google.com/presentation/d/1_Utb2f8qbcjIbTonJ2XXRmijsZU6iXQUkLB3nwDeOGs/edit?slide=id.p#slide=id.p Week 5 slides]// == Week 6 Collected new data on making wider right and left turns in the city environment[[BR]] Continued training on turns at the intersection[[BR]] Updated the documentation to be more clear [[BR]] \\ [https://docs.google.com/presentation/d/1HPqGgHc9MtS-kRgKqUCfaMNVU0m56_TEW2BSjwRlzpk/edit?slide=id.p#slide=id.p Week 6 slides]// == Week 7 Collected lots of data on left turns at the intersection[[BR]] Figured out how to free up space on the car since it was full[[BR]] Connected the car to intersection wifi to be able to connect though orbit rather then Zerotier[[BR]] \\ [https://docs.google.com/presentation/d/1cWWVyynt3JQVEMevvJNloXy6zKUr8TI4Zk4XfLalfAs/edit?slide=id.p#slide=id.p Week 7 slides]// == Week 8 Tested the car to see if it would stop at intersections. It didn't. [[BR]] Found out that in addition to automatically labeling lookahead point angles, we also needed to manually label intersection frames from the data [[BR]] Now the car stops at most intersections [[BR]] Recorded more data at and around the main intersection to fine tune this [[BR]] Started work on creating a mapping system of the intersection area by measuring the length and width of the lanes [[BR]] \\ [https://docs.google.com/presentation/d/1qi_ZzB_SwP99b-R7k28fyWV2QZ1rjOJGeZy_W1YOxq4/edit?slide=id.p#slide=id.p Week 8 slides]// == Connecting to Rascal Through Zerotier: [[BR]] 1. Sign into Zerotier under the account **rascalstonesdc-at-gmail.com** password: **si2022sdc** [[BR]] 2. Connect to the network: rascalstonesdc's 1st network **(ID: d3ecf5726d1a9fcd)** [[BR]] 3. Now you can open a terminal and connect to the IP address of the rascal using the command: **ssh rascal@10.244.114.238 [[BR]]** 4. In a separate terminal, enter **ssh rascaltrain@10.244.176.240** to connect to the rascal server [[BR]] 5. If you get an error message that says: **access denied public key** then connect through ORBIT (instructions below) to add your public key to the /.ssh file titled **authorized_keys** [[BR]] 6. Now you should be able to connect to the server through Zerotier [[BR]] Through ORBIT: [[BR]] 1. Make a reservation on your orbit account under the **intersection** bed [[BR]] 2. Open a new terminal and type **ssh @console.intersection.orbit-lab.org** [[BR]] 3. You should see the 'welcome to ORBIT-LAB' sign [[BR]] 4. Enter **ssh rascaltrain@srv1** for the rascal server [[BR]] == Collecting Data To record data:[[BR]] 1. In the car's terminal, run the command: **roslaunch rascal pure_pursuit.launch** to launch all the nodes [[BR]] 2. In a web page type **10.244.114.238:5000** to open the web display where you can record data [[BR]] 3. Press **bagrecorder/enable** on the web display to start recording [[BR]] 4. Use the joystick or keys on the keyboard (wasd) to move the car around[[BR]] 5. Press **bagrecorder** again to stop recording [[BR]] 6. You can record multiple segments in a loaded session [[BR]] 7. When finished, press ctrl+c in the terminal to save the recordings [[BR]] 8. Go to **/opt/upcar/dataProcessing** and type **python bag2AllData.py** in the terminal to access all recorded sessions [[BR]] 9. Select the number that you want to download (should be the most recent one)[[BR]] 10. Open a new terminal and connect to the rascal server [[BR]] 11. Open **/upcar/dataProcessing** on the server and type **./copyFromCar.sh** which will get the files from the car and bring them over to the server [[BR]] 12. It will ask you to select a host. Choose **1) rascal@172.24.114.238** which is the car's IP address [[BR]] 13. If it asks for a password it's always **si2022sdc** [[BR]] 14. It will ask you which sessions you want to copy over (enter the indices of the sessions you want to copy) [[BR]] == Editing and labeling the Data 1. In the server's terminal, type the command: **roslaunch rascal sim.launch** which will launch the web dislpay for the server where you can edit the desired session [[BR]] 2. In a web page type **10.244.176.240:5000** to open the web display where you can edit data for training (it's best to use the server's web display instead of the car's) [[BR]] 3. In the web display you can click **replayer/pause-play** to run through all of the data segments [[BR]] === Deleting bad segments of data: [[BR]] When portions of the recorded data are messy or irrelevant for training, they can be excluded to improve model performance [[BR]] 1. To jump between different segments that the web display automatically divides the data into, click **replayer/nextSeg** or **replayer/prevSeg** 2. When you get to a segment you want to delete, click the desired starting point on the orange path and hit **replayer/editMode** [[BR]] 3. Select the endpoint of the segment you want to delete by clicking the orange path at that point [[BR]] 4. Select **replayer/editMode** again to highlight the part that's going to be deleted [[BR]] - You can use the **replayer/jumpStart** and **replayer/jumpEnd** buttons to go to the start or end of a segment [[BR]] - You can use the **replayer/forward** and **replayer/backward** to go back and forth between frames in a given segment to help get exact data highlighted [[BR]] 5. Click **replayer/deleteSelection** to delete the highlighted segment [[BR]] === Labeling intersection data: [[BR]] For intersection data you MUST label the data manually at each segment where an intersection occurs [[BR]] 1. Highlight the frames of data that has multiple valid paths (left, right, and straight) the same way you would for deleting segments [[BR]] 2. Click **replayer/setLvalid, replayer/setRvalid, replayer/setSvalid** to indicate which paths are possible at that intersection. This will be used later when training.[[BR]] IMPORTANT: Once you are finished editing you MUST save the data by running the **data_loader/smoothen** command == Training Rascal We used the fisheye and discrete_waypoint model to train on [[BR]] 1. If you go to **/sessions** in the server terminal and type **ls** your chosen files should be listed there [[BR]] 2. Go back to **/upcar** and move your data to /rascaltraining/data: **mv dataProcessing/sessions/ rascaltraining/data [[BR]] 3. If you go into **/rascaltraining/data** you should see your file listed [[BR]] 4. Go back to **/rascaltraining** and type the command: **python trainModel.py -m fisheye --use_labels 1.0 -s labels** when using fisheye. Type the command: **python trainModel.py -m discrete_waypoint --center_width 30 --num_bins 3 --use_labels 1.0 -s labels** when using discrete_waypoint[[BR]] 5. Now your model is being trained [[BR]] 6. To stop training press enter [[BR]] 7. Remember to stop training when the validation loss doesn't seem to be decreasing by a significant amount anymore to avoid overfitting [[BR]] 8. To check what number model you just created, go to **/savedModels** and type **ls** (yours will be the most recent number) 9. To push your newly made model to gitlab, type **git add -f ** [[BR]] 10. Next type the command **git commit -m ""** [[BR]] 11. Next type the command **git push origin main** [[BR]] 12. Finally, back in the car's terminal go to **/savedModels** and type **git pull origin main** [[BR]] You're now ready to start testing! [[BR]] == Testing Rascal To test rascal: [[BR]] 1. Go to /upcar in the car's terminal [[BR]] 2. Enter **./launch_with_model.sh pure_pursuit** [[BR]] 3. Open up the web display [[BR]] 4. Run these commands in the web display: [[BR]] a. sim_camera/set_enable (set to true) [[BR]] b. ml/imgDisp (set to true) [[BR]] c. ml/0/enable (click enable) [[BR]] Watch the car move on it's own! [[BR]] Make sure it doesn't crash!