= Self Driving Vehicular Project = **Team:** Christopher Lee, Michelle Gutwein, Thierry Antoine, Varun Chirravuri, and Anjali Kapilavai [[BR]] **Advisors:** Ivan Seskar and Jennifer Shane[[BR]] **Project Description & Goals:**[[BR]] Build and train miniature autonomous cars to drive in a miniature city.[[BR]] **Technologies:** ROS (Robot Operating System), Pytorch ---- == Week 1 Read through the [https://docs.google.com/document/d/1KWIyvGMnB2nboosv068qp-X2BbNm9y4tUnWLzKbz6bI/edit?tab=t.0#heading=h.gz0otq9rwrdy manual] provided to us. [[BR]] \\ [https://docs.google.com/presentation/d/1ZJatipqqiaRQ06gbw_t5Jbho-dZOFxXZS6fXx_FxIIc/edit?slide=id.p#slide=id.p Week 1 slides]// == Week 2 Learned what ROS is and how to use it[[BR]] Also got to know the car, RASCAL[[BR]] \\ [https://docs.google.com/presentation/d/1VsER6V9A0wsgOowDd7vA26k3LNe6V8tMvsMXYWiOBVo/edit?slide=id.p#slide=id.p Week 2 slides]// == Week 3 Read through all of the code in the [https://gitlab.orbit-lab.org/self-driving-car-2023/upcar/-/tree/main/catkin_ws/src/rascal/src?ref_type=heads Gitlab] [[BR]] Learned how to record and edit data on the web display[[BR]] \\ [https://docs.google.com/presentation/d/1NL-Xl_2XD7dQo6XJMo8_wsapmv6N-sxQl58-IZ-0_L0/edit?slide=id.p#slide=id.p Week 3 slides]// == Week 4 Learned how to train a model[[BR]] Trained and tested the model on simple paths[[BR]] \\ [https://docs.google.com/presentation/d/10KDlyMXHnCqYHTe5A6nhevLICf3jup91HIhf3qXCt6U/edit?slide=id.g369586f805a_0_10#slide=id.g369586f805a_0_10 Week 4 slides]// == Week 5 Trained model on making right and left turns in the city environment[[BR]] Started training on turns at the intersection[[BR]] Created a new model called gps which will allow users to give the car commands on the direction to turn at an intersection before running the car[[BR]] \\ [https://docs.google.com/presentation/d/1_Utb2f8qbcjIbTonJ2XXRmijsZU6iXQUkLB3nwDeOGs/edit?slide=id.p#slide=id.p Week 5 slides]// == Week 6 Collected new data on making wider right and left turns in the city environment[[BR]] Continued training on turns at the intersection[[BR]] Updated the documentation to be more clear [[BR]] \\ [https://docs.google.com/presentation/d/1HPqGgHc9MtS-kRgKqUCfaMNVU0m56_TEW2BSjwRlzpk/edit?slide=id.p#slide=id.p Week 6 slides]// == Week 7 Collected lots of data on left turns at the intersection[[BR]] Figured out how to free up space on the car since it was full[[BR]] Connected the car to intersection wifi to be able to connect though orbit rather then Zerotier[[BR]] \\ [https://docs.google.com/presentation/d/1cWWVyynt3JQVEMevvJNloXy6zKUr8TI4Zk4XfLalfAs/edit?slide=id.p#slide=id.p Week 7 slides]// == Week 8 Tested the car to see if it would stop at intersections. It didn't. [[BR]] Found out that in addition to automatically labeling lookahead point angles, we also needed to manually label intersection frames from the data [[BR]] Now the car stops at most intersections [[BR]] Recorded more data at and around the main intersection to fine tune this [[BR]] Started work on creating a mapping system of the intersection area by measuring the length and width of the lanes [[BR]] \\ [https://docs.google.com/presentation/d/1qi_ZzB_SwP99b-R7k28fyWV2QZ1rjOJGeZy_W1YOxq4/edit?slide=id.p#slide=id.p Week 8 slides]// == Week 9 Collected data on the entire city at all intersections [[BR]] Labeled data correctly by highlighting frames with the intersection in them [[BR]] Tested the car with this new data - it still wasn't stopping at all the intersections and seemed to have trouble making some of the left and right turns [[BR]] Printed out Aruco markers to put on the ceiling and made a virtual map of the city for the GPS system [[BR]] \\ [https://docs.google.com/presentation/d/1ffeXI1N9efWs9p-BQeueuo7iaU-4ubH-8KVGldLL8aw/edit?slide=id.p#slide=id.p Week 9 slides]// == Week 10 Added an additional label for turns [[BR]] Put up Aruco Markers [[BR]] Fixed bugs in the training code to allow for our additional label [[BR]] \\ [https://docs.google.com/presentation/d/1P2Wru9RDuzSBghoWyTv7puq-xD8T0JuzRMwTQgDo5iA/edit?slide=id.p#slide=id.p Final presentation slides]// == Connecting to Rascal Through Zerotier: [[BR]] 1. Sign into Zerotier under the account **rascalstonesdc-at-gmail.com** password: **si2022sdc** [[BR]] 2. Connect to the network: rascalstonesdc's 1st network **(ID: d3ecf5726d1a9fcd)** [[BR]] 3. Now you can open a terminal and connect to the IP address of the rascal using the command: **ssh rascal@10.244.114.238 [[BR]]** 4. In a separate terminal, enter **ssh rascaltrain@10.244.176.240** to connect to the rascal server [[BR]] 5. If you get an error message that says: **access denied public key** then connect through ORBIT (instructions below) to add your public key to the /.ssh file titled **authorized_keys** [[BR]] 6. Now you should be able to connect to the server through Zerotier [[BR]] Through ORBIT: [[BR]] 1. Make a reservation on your orbit account under the **intersection** bed [[BR]] 2. Open a new terminal and type **ssh @console.intersection.orbit-lab.org** [[BR]] 3. You should see the 'welcome to ORBIT-LAB' sign [[BR]] 4. Enter **ssh rascaltrain@srv1** for the rascal server [[BR]] == Collecting Data To record data:[[BR]] 1. In the car's terminal, run the command: **roslaunch rascal pure_pursuit.launch** to launch all the nodes [[BR]] 2. In a web page type **10.244.114.238:5000** to open the web display where you can record data [[BR]] 3. Make sure **wideFOV** is set to true for the fisheye camera [[BR]] 4. Press **bagrecorder/enable** on the web display to start recording [[BR]] 5. Use the joystick or keys on the keyboard (wasd) to move the car around[[BR]] 6. Press **bagrecorder** again to stop recording [[BR]] 7. When finished, press ctrl+c in the terminal to save the recorded session [[BR]] 8. You can record multiple segments within a single session, but note that it's better to keep sessions short— record a few segments, save the session, and then start a new one. This helps prevent data loss in case the car malfunctions or the Wi-Fi connection drops. [[BR]] 9. Go to **/opt/upcar/dataProcessing** and type **python bag2AllData.py** in the terminal to access all recorded sessions [[BR]] 10. Select the number that you want to download (should be the most recent one)[[BR]] 11. Open a new terminal and connect to the rascal server [[BR]] 12. Open **/upcar/dataProcessing** on the server and type **./copyFromCar.sh** which will get the files from the car and bring them over to the server [[BR]] 13. It will ask you to select a host. Choose **1) rascal@172.24.114.238** which is the car's IP address [[BR]] 14. If it asks for a password it's always **si2022sdc** [[BR]] 15. It will ask you which sessions you want to copy over (enter the indices of the sessions you want to copy) [[BR]] 16. In **/sessions** change the file name to something relevant to keep track of your data (using **mv**) == Editing and labeling the Data 1. In the server's terminal, type the command: **roslaunch rascal sim.launch** which will launch the web dislpay for the server where you can edit the desired session [[BR]] 2. In a web page type **10.244.176.240:5000** to open the web display where you can edit data for training (it's best to use the server's web display instead of the car's) [[BR]] 3. In the web display you can click **replayer/pause-play** to run through all of the data segments [[BR]] 4. Use [https://docs.google.com/document/d/1KWIyvGMnB2nboosv068qp-X2BbNm9y4tUnWLzKbz6bI/edit?tab=t.0#heading=h.2t8tt1e6yq4r Key Binds] to make editing easier and faster [[BR]] === Deleting bad segments of data: [[BR]] When portions of the recorded data are messy or irrelevant for training, they can be excluded to improve model performance [[BR]] 1. To jump between different segments that the web display automatically divides the data into, click **replayer/nextSeg** or **replayer/prevSeg** 2. When you get to a segment you want to delete, click the desired starting point on the orange path and hit **replayer/editMode** [[BR]] 3. Select the endpoint of the segment you want to delete by clicking the orange path at that point [[BR]] 4. Select **replayer/editMode** again to highlight the part that's going to be deleted [[BR]] - You can use the **replayer/jumpStart** and **replayer/jumpEnd** buttons to go to the start or end of a segment [[BR]] - You can use the **replayer/forward** and **replayer/backward** to go back and forth between frames in a given segment to help get exact data highlighted [[BR]] 5. Click **replayer/deleteSelection** to delete the highlighted segment [[BR]] === Labeling intersection data: [[BR]] For intersection data you MUST label the data manually at each segment where an intersection occurs [[BR]] 1. Highlight the frames of data that has multiple valid paths (left, right, and straight) the same way you would for deleting segments [[BR]] 2. Click **replayer/setLvalid, replayer/setRvalid, replayer/setSvalid** to indicate which paths are possible at that intersection. This will be used later when training.[[BR]] === Labeling turns: In addition to the intersection labeling, you also need to label the turns themselves that take place immediately after the intersection, so that the car knows to complete the full turn before reenabling stopping mode for the next intersection. [[BR]] 1. Highlight the frames that indicate a full turn after an intersection occurs [[BR]] 2. Use **replayer/setLturn** to highlight left turns and **replayer/setRturn** to highlight right turns the same way you would highlight the intersection. [[BR]] IMPORTANT: Once you are finished editing you MUST save the data by running the **data_loader/smoothen** command == Training Rascal We used the fisheye and discrete_waypoint model to train on [[BR]] 1. If you go to **/sessions** in the server terminal and type **ls** your chosen files should be listed there (it will have the term **labeled** or **smooth** in the name) [[BR]] 2. Go back to **/upcar** and copy your data to /rascaltraining/data: **cp dataProcessing/sessions/ rascaltraining/data [[BR]] 3. If you go into **/rascaltraining/data** you should see your file listed [[BR]] 4. Go back to **/rascaltraining** and type the command: **python trainModel.py -m fisheye --use_labels 1.0 -s labels** when using fisheye. Type the command: **python trainModel.py -m discrete_waypoint --center_width 30 --num_bins 3 --use_labels 1.0 -s labels** when using discrete_waypoint[[BR]] 5. Now your model is being trained [[BR]] 6. To stop training press enter [[BR]] 7. Remember to stop training when the validation loss doesn't seem to be decreasing by a significant amount anymore to avoid overfitting [[BR]] 8. To check what number model you just created, go to **/savedModels** and type **ls** (yours will be the most recent number) 9. To push your newly made model to gitlab, type **git add -f ** [[BR]] 10. Next type the command **git commit -m ""** [[BR]] 11. Next type the command **git push origin main** [[BR]] 12. Finally, back in the car's terminal go to **/savedModels** and type **git pull origin main** [[BR]] You're now ready to start testing! [[BR]] == Testing Rascal To test rascal: [[BR]] 1. Go to /upcar in the car's terminal [[BR]] 2. Enter **./launch_with_model.sh pure_pursuit** [[BR]] 3. Open up the web display [[BR]] 4. Run these commands in the web display: [[BR]] a. wideFOV (set to true) [[BR]] b. ml/imgDisp (set to true) [[BR]] c. ml/0/enable (click enable) [[BR]] Watch the car move on it's own! [[BR]] Make sure it doesn't crash!