Last year saw a host of humanoid robots captivate audiences, who watched these robots perform backflips, stack blocks, and prepare cups of coffee. But as the dust begins to settle and the public’s “AI IQ” rises, some demonstrations have begun to spark skepticism.
Understandably, this sharpens the critical eye on all such demos and raises the question: “Is this robot really operating autonomously?” At Sanctuary AI, we’d argue that while transparency is key, this question misses the bigger picture.
The intrigue behind whether or not a robot is performing autonomous work is understandable and to be clear, Sanctuary AI’s embodied artificial general intelligence (AGI) robots are capable of working autonomously. However, in most of the videos we post for our YouTube series “Robots Doing Stuff,” our robots are being teleoperated by a person. Here’s why: If you want to actually build a robot capable of doing things like people, you must begin by gathering an enormous amount of data from the activities that people would do if they were a robot.
That's where teleoperation comes in – in order to train generative AI models, you must perform an enormous number of demonstrations, so that the AI systems can learn from them. This is how we can build “Large Behavior Models”. Gathering human behavioral data through teleoperation is what builds the foundational blocks toward embodied artificial general intelligence. This data doesn’t just include what doing a task looks like, but also how it feels. Our human behavioral data consists of all kinds of sensory data, including one critical element not all competitors are focused on, touch. Our tactile sensor technology allows our AI control system, Carbon, to understand the interaction between visual and haptic data. This means that Carbon can understand the difference between picking up something fragile, like a glass, and something more robust of the same appearance, like a plastic cup, and amend its actions accordingly. Without a person doing this first through the robot, we wouldn’t be able to collect all the data needed for a robot to interact with and manipulate objects in a changeable environment.
Teleoperation is also a critical component of hardware testing. There’s little point training an AI system to do something, like pick up a glass, if you then find out the robotic hands aren’t dexterous enough to grasp the glass in the first place. Teleoperation allows us to ensure our general purpose robots can physically perform the task at hand, while also allowing Carbon to find the best possible method to complete the task through data collected.
We can’t create the world’s first human-like intelligence in general purpose robots without training it through the lens of human experience. Teleoperation is a crucial piece of this puzzle, but it should by no means exist in isolation. Building a mind requires many other components such as reinforcement learning, simulation, and natural language processing to name a few. At Sanctuary AI, it’s these things combined with expert demonstrations from our human pilots that are guiding us toward mission success. If you want to learn more about this process, watch our Sanctuary AI Ground Truth podcast episode that takes a deep dive into our workflow process.
So, the next time you see a robot being teleoperated, remember that it’s a crucial part of teaching AI control systems what it’s like to experience our world.
