Learning to learn to swim: investigating the use of reinforcement learning for the development of multi-limbed swimming gaits

Anthony C. Drago III

doi:10.17918/00011133

Underwater bio-inspired robotics offers a promising path for advancing autonomous aquatic locomotion by emulating the efficient swimming strategies of marine organisms. However, developing robust, multi-limbed swimming gaits that perform reliably in complex underwater environments remains a significant challenge. This thesis investigates the use of reinforcement learning (RL) techniques to generate effective multi-limbed swimming gaits in bio-inspired underwater robots, focusing on fish-inspired fins and tail designs as well as sea lion foreflippers. The research follows a three-phase approach. First, empirical testing of bio-robotic propulsors demonstrated that variations in fin and flipper kinematics significantly influence propulsive forces, establishing bio-inspired heuristics that informed the development of the RL environment and defined its action space for targeted swimming tasks. Second, free-swimming bio-robotic models and coupled numerical simulations were developed in MATLAB Simscape. These multi-body simulation environments incorporate hydrostatic and hydrodynamic forces calculated using closed-form equations, rather than computational fluid dynamics (CFD), providing an efficient and physics-grounded training platform for RL-based gait optimization. Finally, RL was applied to tune controller parameters, optimize open-loop kinematics, and implement closed-loop velocity control across three robotic platforms: a two-fin system, the fish-inspired SAMUNO, and the sea lion-inspired SEAMOUR. Results show that RL-generated gaits consistently outperformed biologically inspired, hand-tuned gaits for the tasks they were designed to accomplish. In the two-fin system, RL-based tuning of a Central Pattern Generator controller yielded higher thrust and reduced lateral forces compared to manual tuning. For SEAMOUR, RL-derived flipper stroke patterns produced greater forward velocity and enhanced stability relative to bio-inspired strokes. In SAMUNO, RL developed a straighter swimming gait through coordinated fin and body joint movements compared to the bio-inspired baseline. Additionally, RL enabled closed-loop velocity control for speeds up to 0.9 m/s in simulation and 0.5 m/s on the physical robot. Discrepancies between simulation and physical performance--particularly at higher speeds--highlight the need for more accurate hydrodynamic modeling and improved state estimation in future work. Overall, these findings demonstrate the potential of reinforcement learning to generate robust, task-specific swimming gaits that outperform traditional hand-tuned approaches, advancing the performance and reliability of underwater bio-inspired robotic systems.

Learning to learn to swim: investigating the use of reinforcement learning for the development of multi-limbed swimming gaits

(1)