Camera-Based Human Tracking and Following Algorithm for Mobile Robots
Autonomous ground vehicle (AGV) following system using YOLOv11 and Range Imaging
Swipe left or right to navigate
Real-time tracking and following in the sim
Project Overview
Developed a camera-based human tracking pipeline to enable the safe and efficient relocation of airport robots. This algorithm detects and tracks human movement in real time using Ultralytic's YOLOv11 model and a 3D depth camera to create a dynamic path plan.
Technical Implementation
- Developed a ROS2 node that subscribes to an Intel RealSense D455's RGB and depth streams along with the camera info containing its camera intrinsics
- Used Ultralytic's YOLOv11 tracking model to publish an annotated image and an ENU pose
- Created a Python script to generate custom Gazebo SDF worlds for simulation
- Built and optimized CUDA-enabled Dockerfiles to enable GPU-acceleration on an NVIDIA Jetson Orin NX
- Built an app using HTML and Javascript to adjust parameters, toggle tracking, and visualize the current person being tracked in real time
Key Features
- Distinguishes between multiple people in a scene
- Detection publishes at ~10hz with GPU-acceleration enabled
- Maintains a fixed distance between the robot and the tracked person (adaptive cruise control)
- Seamless integration with ROS navigation stack
- Comprehensive testing in simulation as well as real life
- Parameters can be adjusted before tracking is toggled
- Visualization occurs in real time
Results & Future Work
The system successfully worked on the sim as well as the real robot. In the future, I plan to enhance the system's capabilities by integrating gesture-based control to remove the dependence of a carried mobile device and enabling more intuitive human-robot interaction. Additionally, implementing facial recognition will allow the system to identify authorized personnel, ensuring only those approved can be followed which will enhance overall security.