AUTO PATROL ROBOT

Path Planning & Control Integration Initially, we first explored Autoware's OpenPlanner and Freespace modules, but their weak connections across perception, planning, and control required significant effort to identify and bridge gaps between perception, planing, and control subsystems. To overcome this, I co-developed a custom framework that allows hybrid A* global planning with TEB local planning in ROS1. This ensured smoother integration of perception data into real-time navigation.

Obstacle Avoidance: To address the blind spots in the 16-line LiDAR, I use the ultrasonic array at first to detect closing obstacles. However, due to the high noise and low sampling rates (3-4 Hz), a 1D LiDAR is integrated. Data from the 1D LiDAR and one line of the multi-line LiDAR is combined to achieve stable, high-sensitivity detection. Meanwhile, the ultrasonic sensor array is also under testing for low-height obstacles like small animals and stones.

Auto Docking: AprilTag is used for relative positioning during final alignment. Early iterations addressed unstable vehicle dynamics by developing multi-route strategies: I developed one strategy utilized multi-routes planning and odom-based localization, and another segmented approach angles with speed-specific zones for large directional deviations. However, after field validation, the navigation accuracy enabled simplification to a single directional calculation function, with a secondary protocol triggering a stop-and-wait state upon tag loss.

Dual-Rate Detection: A two-layer detection strategy is implemented for balancing response speed and contextual precision. Locally on the vehicle, a self-trained YOLO model runs at high rate, detecting unwanted behaviors such as smoking or walking dogs in real time. In parellel, selected frames are sent at low rate to a multi-modal model deployed on company-server, which delivers higher-precision judgements on subtle cases like stepping on grass or fighting.

Voice Interaction: Live audio streams are continuously sent to a large language model, where detected words are classified by an agent into commands, casual dialogue, or emergency cases such as tourists asking for help. Each category triggers corresponding strategies, from executing control actions to issuing alerts. On the hardware layer, a dual-microphone array fileters input from the front direction, while speakers are positioned on the vehicle's side to prevent feedback from its own output.

LiDAR-Camera Fusion: LiDAR points are projected into the camera coordinate system, filtered by YOLO ROIs(Region of interest), and then clustered using a density-based method. The most representative cluster is selected as the final result, providing depth and localization for detected objects.

Voice Strategies: After audio is converted to text by ASR, the recognized text is processed by an LLM combined with a knowledge base. This layer classifies the input into three categories: casual dialogue, control commands, or emergencies. Casual dialogue prompts lightweight responses to maintain engagement; commands such as navigation requests are translated into vehicle actions; and emergency cases, like tourists asking for help, immediately trigger protocols to notify police or park stuff.

Visual Strategies: Visual perception outputs are also mapped into interaction strategies. When unwanted behaviors such as smoking or walking dogs are detected, the robot sends a signal to the backend, issues a voice warning to the individual involved, and directs the navigation system to keep following the case. In more serious scenarios - such as detecting a person who has fallen or groups engaged in fighting - the system sends emergency calls to the backend to alert part guards for assistance.