Overview
Crafted a state-of-the-art person-following system for Teladoc Health's commercial tele-presence robot. The system was integrated into the existing autonomous navigation stack, enabling the robot to autonomously follow a designated person through hospital environments while avoiding obstacles and maintaining safe distance.
The system was deployed as a production feature in a commercial product used in hospital environments across the United States, making the engineering constraints around reliability, latency, and robustness non-negotiable.
The Challenge
Person following in real hospital environments is significantly harder than lab conditions. Challenges include: crowded corridors with many people present, frequent occlusions as the target passes doorways and corners, varied lighting conditions across different hospital zones, and the need to distinguish the target person from other hospital staff wearing similar attire.
The system also needed to integrate seamlessly with the existing navigation stack without degrading obstacle avoidance performance — the robot had to follow its target while still safely navigating around patients, carts, and other obstacles. All inference had to run in real-time on onboard compute without a dedicated GPU.
Technical Approach
- Real-time person detection using edge-optimized computer vision models with TensorFlow Lite (TFLite) for efficient onboard inference
- Depth camera integration for 3D person position estimation, enabling precise distance control independent of camera zoom or image scale
- Person re-identification (ReID) system to recover tracking after occlusions — when the target disappears behind a corner or door, the system re-acquires the correct person upon reappearance
- Target person lock-on at initial selection, maintained robustly through crowd and occlusion scenarios using appearance features
- 1.2-meter following distance maintained continuously using depth-based control loop integrated with the navigation stack
- Tight integration with the autonomous navigation stack for simultaneous obstacle avoidance and person following without performance degradation
- Real-time performance on onboard compute without GPU acceleration — optimized model architecture and inference pipeline for edge deployment