Robots Need More Than a Model: They Need a System

Tech Insights2026.09.1219 min read

Scaling embodied AI requires KyOS, a closed-loop architecture coordinating perception, task planning, and reflex control across different timescales.

From World Models and Physical AI to KyOS: How to Think About a Robot's "Brain"

A service robot gets a task: take a cup of coffee to a meeting room.

It has to know where it is, where the coffee is, and where the meeting room is. It has to recognize people, doors, tables, chairs, and obstacles that appear without warning. Delivering coffee is more than going from A to B. The cup should stay upright, the robot should not hit anyone, and it should avoid walking into a meeting that is already in progress.

When a child runs in from the side, the robot cannot pause for several seconds to think. It has to slow down, keep its balance, and change path immediately. If the planned route is blocked by construction, reflexes are not enough either. It has to replan, and it may need to decide whether to ask a person for help.

A robot that can work in the Real World cannot run on one intelligent model. It needs an architecture in which different systems operate together on different timescales.

In my view there are at least three core systems: perception, real-time control, and reasoning. Across all three there should also be a Human-in-the-Loop. People set goals at critical points, give demonstrations, correct mistakes, and judge outcomes, so the system can run on its own and still keep learning from people.

1. The Perception System: Giving Robots a Continuous Understanding of the World

Perception is more than mounting a few cameras, and more than running object recognition once.

A robot may carry cameras, LiDAR, depth cameras, microphones, inertial measurement units, joint encoders, torque sensors, and tactile sensors. The perception system has to fuse those streams, which differ in origin, frequency, and accuracy, to answer a few basic questions:

Where am I, and what is the current state of my body?
What is around me, where is it, and how is it moving?
Which information in the current environment is relevant to the task?
What intent is actually being expressed by the language, gesture, or demonstration a person provides?

The output should be a continuously updated world state, not a pile of labels. Spatial structure, object relationships, dynamic changes, risk areas, task objectives, and the robot's own state need to sit in one representation the rest of the stack can use.

Spatial Intelligence belongs here. The robot needs to see the cup, know it sits near the table edge, know there is liquid inside, and know that grasp angle and acceleration will affect whether the liquid spills. The job of perception is to build the context that action needs.

KyOS Executable Spatial Intelligence System diagram with world model, intelligence brain, and embodied execution layers

2. The Real-Time System: The Robot's "Cerebellum" and Reflex Arc

The second system is real-time feedback. It is closer to the cerebellum, brainstem, and reflex arc: balance, motion control, collision protection, trajectory tracking, force control, and emergency response.

The requirement is determinism, not average speed. Computations that must finish inside a time window have to finish inside that window. The ROS 2 real-time documentation says the same thing: real-time computing requires "the right computation to happen at the right time." For some robots, missing a deadline is itself a failure.[1]

A large model can take longer to compare options. Real-time control cannot wait. When a quadruped steps on a loose stone, posture has to be corrected immediately. When an arm hits a person or meets unexpected resistance, the safety response has to outrank the original task. When a drone hits a gust, it cannot wait for a cloud model to finish inference before it holds attitude.

The real-time system therefore needs clear execution boundaries and the highest priority. A high-level system can say, "Put the cup on the table." Joint updates in every control cycle, speed and torque limits, and handling of slip and disturbance belong in a constrained, verifiable real-time Closed Loop.

A larger model does not automatically make a safer robot. Thinking and responding on time are different capabilities.

3. The Reasoning System: Understanding Goals and Making Judgments

The third system is reasoning, the layer people usually call the "brain." It handles higher-level reasoning, planning, and decision-making:

Breaking down "tidy the room" into executable subtasks
Weighing efficiency, safety, and human preferences
Replanning when something goes wrong, instead of repeating a failed action
Calling on the appropriate tools, skills, and knowledge
Using past experience to decide when to continue, when to pause, and when to ask a person for confirmation

It does not need to run at motor-control frequencies. It does need more context, a longer horizon, and a way to combine information. Language models, multimodal models, task planners, memory systems, and policy models can all sit in this layer.

It should not do everything. If one high-level model both interprets "what I want" and emits every motor command, semantic reasoning, real-time control, and safety sit on a single path. As the system grows, that path is harder to debug, verify, and extend.

Let reasoning decide what to do and why. Let the real-time system do it on time and within safety limits. Let perception keep telling both of them what the world is now.

KyOS chart mapping real-time, cognition, and perception systems to different operating time scales

The Fourth Thread: Human-in-the-Loop, so Robots Keep Learning From People

Autonomy does not mean taking people out of the architecture. While robot capabilities are still incomplete and the Real World is full of long-tail cases, people should remain part of the system.

Human-in-the-Loop is not only remote teleoperation, and it is not only a "good" or "bad" click after the fact. People play different roles at different stages:

Stage	What humans do
High-level decision-making	Define goals, boundaries, values, and preferences; provide confirmation when there is ambiguity or high risk
Operation	Demonstrate and correct, and take over when necessary
Outcome evaluation	Compare solutions and mark out what counts as "good"
Learning and iteration	Write demonstrations, interventions, and preferences into the next round of training and validation

In High-Level Decision-Making, People Set Values and Boundaries

A robot can compute the shortest route. It does not, on its own, know which route is appropriate. Cutting through a children's activity area, walking onto a film set that is shooting, or coming up fast behind someone may all be geometrically valid and still violate ordinary social norms.

More human-like high-level decisions means capturing the factors people actually weigh: safety, courtesy, privacy, risk, priorities, and individual preferences. When an instruction is ambiguous, confidence is low, or the stakes are high, the robot should hand the choice back to a person rather than hide uncertainty behind a fluent answer.

The reasoning system therefore needs planning, and it needs a map of its own limits: what it can decide alone, when it should ask, and when it must stop. Asking for help at the right moment is part of that competence.

During Operation, People Teach and Watch for Safety

Many physical skills are hard to write down as language or rules. Picking up a soft cup of liquid, inserting a flexible cable into a narrow connector, or changing gait on a wet floor all depend on small, continuous details in the motion.

People can show those details through teleoperation, hand-guiding an arm, motion capture, video demonstration, or taking over at a critical moment. Reproducing demonstrations is generally imitation learning. The system enters reinforcement learning from human feedback only when it also uses corrections, ratings, or preferences between options to learn a reward and improve the policy.

People are useful because the data they give is high value, not because there is more of it. During autonomous runs the robot spends most of its time in familiar states. People tend to step in when failure is close, risk is rising, or the system hesitates. Those interventions mark the edge of the policy: where it is unreliable, why a person does not trust the current action, and which way recovery should go. Existing robotics research has shown this path: people monitor autonomous execution, correct by teleoperation in hard states, and the deployment data is reused for later policy learning.[8]

Reinforcement Learning: The Reward, and Who Decides What "Good" Is

Traditional reinforcement learning depends on reward functions, but in real tasks "good" is often not a number. Fast delivery can still be a poor experience. A successful grasp can still be unsafe. The shortest path can still violate human preference. A slightly wrong reward lets the system game the rules.

Human-in-the-Loop can turn those hard-to-write goals into learning signals. People can compare two behaviors and say which is closer to the goal; those preferences train a reward predictor, which then helps optimize the policy. Work in this area has shown that people do not need to write a reward for every action. Continuous comparison and feedback can teach complex objectives.[7]

For multi-task robots, human feedback also helps new tasks reuse old experience. Opening a window, a drawer, and a door are different objects with shared structure: approach, contact, apply force, confirm state. Research has shown that a preference model pretrained on existing tasks can cut the number of human queries needed to learn new robotic tasks.[8] That is the other half of task scalability: code and models have to be reusable, and so do the results of human teaching.

Keep the Execution Loop and the Learning Loop Apart

Doing reinforcement learning with a person in the loop does not mean a robot in public should trial-and-error its way through a live control policy. Exploration in the physical world has real costs. Bad actions can break equipment or injure people.

A safer design uses two loops that connect, with different permissions:

The execution loop handles the current task. The robot runs a validated policy. People can confirm high-level decisions, correct actions, or take over in an emergency.
The learning loop is for later versions. The system collects demonstrations, interventions, failures, and preference data; trains in simulation or controlled settings; and ships the next version only after regression tests, safety checks, and a phased rollout.

Human feedback can then improve the system without letting online learning eat into real-time performance or safety. Intervention does not have to stay forever. As the robot covers more hard states, takeovers should become rarer, and people can spend their attention on new long-tail cases.

KyOS diagram separating the real-time operating loop from the slower human-guided learning loop

Three Systems, Not Three Isolated Islands

Splitting a robot into three systems is still one Closed Loop:

The perception system updates the world state, the reasoning system selects goals and strategies, and the real-time system executes and feeds back the results; humans confirm, demonstrate, and correct at critical points; execution and human feedback together change the next round of perception, decision-making, and learning.

This loop runs on several timescales at once. Low-level control may update every millisecond. Local obstacle avoidance and skill execution run on shorter cycles. High-level task planning may take seconds or more. Exact frequencies depend on the robot and the task. The principle does not: fast systems hold stability and safety; slow systems handle understanding and choice. Slow systems can guide fast ones. They must not block them.

This layering is familiar. Robotics has long discussed mixing reactive and deliberative architectures. Classic three-layer designs already put a task-execution layer between fast feedback control and slower high-level reasoning. Large models improved semantic understanding and generalization. They did not remove the engineering boundaries between timescales. They made coordination across the stack more important.[2]

KyOS loop linking perception, cognition, and real-time control with a human in the loop

World Models Matter. They Are Still Only One Part of the System

"World Model" has been one of the most discussed ideas in robotics and AI in recent years. In short, it is the machine's internal picture of the world and how the world changes: what is happening now, and what might happen next if a given action is taken.

A World Model can hold geometry, semantics, physical properties, and dynamic relationships. It can also predict future states from images, video, sound, and actions. NVIDIA's definition likewise stresses understanding and predicting the dynamics, physical properties, and spatial characteristics of the Real World.[3]

That representation is important. Without one, a robot has a hard time anticipating what may happen, and a harder time comparing options before it acts. A World Model is still not a complete robot:

It can predict that a cup will fall, but that does not mean the motors can necessarily catch it in time
It can generate a realistic training environment, but that does not mean perception results will remain reliable under real-world noise
It can produce a reasonable plan, but that does not mean the plan can be executed safely and deterministically
It can perform well on one task, but that does not mean a new task can be integrated at low cost

Some World Model work now focuses on prediction quality, generation quality, and benchmark scores. That is a normal stage of research. The risk in industrial deployment is treating progress on one model as if the whole machine were done. What a robot has to deliver is not a polished video. It is completing tasks over a long period, stably and recoverably, in an open environment.

Physical AI Is Not the Full Robotics Answer Either

Physical AI is about getting AI to perceive, understand, reason about, and act in the physical world. The usual toolkit leans on physical laws, high-fidelity simulation, synthetic data, reinforcement learning, and sim-to-real transfer. NVIDIA's own definition is broader than "physical laws." In the current industry story and toolchain, though, simulation, physical consistency, and training data tend to sit up front.[4]

Those capabilities address how robots learn about the physical world. They do not, by themselves, answer another set of engineering questions:

How can capabilities be reused across robots from different manufacturers and with different forms?
How can a new task be quickly decomposed, combined, tested, and deployed?
How should high-level models, real-time control, safety systems, and sensors work together?
How can a system detect failure, degrade gracefully, and recover?
How can data generated during real-world operation re-enter the training and validation loop?

World Models and Physical AI each supply important pieces. They still have to be assembled into a complete robotic system.

KyOS: Bringing Perception, the Cerebellum, and the Brain Into a Closed Loop

XGRIDS's core capability, built up over years, has been turning real spaces into three-dimensional data machines can use: multimodal spatial sensing and high-fidelity 3D reconstruction, forming spatial representations that AI can use for understanding, reasoning, and training.

At the NVIDIA GTC 2026 showcase, XGRIDS showed a Real2Sim path: capture real environments, build a high-fidelity 3D world, then connect that world to simulation and robot-training workflows. Public reports also say its spatial sensing and modeling systems have been deployed on quadruped platforms, so robots can plan paths, make behavioral decisions, and execute tasks from a fuller 3D spatial structure, rather than doing local obstacle avoidance from onboard sensors alone.[5]

XGRIDS's official description of KyOS is direct: a Spatial Operating System made of "environmental perception, a Smart Brain, and an Embodied Cerebellum," rather than a single model. The gap it aims to fill is the spatial intelligence robots lack when they enter the Real World.[6]

Those three pieces map onto the three-system frame above.

KyOS diagram closing the loop across the world model, smart brain, and embodied cerebellum

Environmental Perception: The World Model Has to Start From the Real World

KyOS first builds a 3D world of the current environment. It restores the shapes of streets, buildings, and storefronts, and it also projects semantics and relationships into that space: where objects are, how large they are, where motion is possible, and how doors, steps, counters, and obstacles relate to the current task.

The system has to answer "What does this place look like?", "Can I move through here?", "Should I go around it or pass through?", and "How will this spatial element affect the task?" That is the perception system's job: fold multiple sensors into a world state that can support action.

XGRIDS calls its spatial cameras "the entry point to the world," and says "world models originate from reality, not imagination." Robots can use generative models to fill in and predict, but the geometry, semantics, and traversability that actions depend on have to be anchored in the real environment first.

The Smart Brain: Task Chains Generated From the Environment

When a person makes a request, KyOS's Smart Brain reads the instruction against the current spatial state: where is the destination, which route, what might change along the way. Official materials stress that task chains are generated from the real environment, not a fixed process written in advance.

High-level decision-making is then more than translating language into a string of actions. It has to combine the task's intent with the spatial conditions at that moment. The same instruction, "Go to that store and pick something up," should yield different task chains when the entrance is closed, foot traffic has changed, or the road is blocked.

Before the robot leaves, the whole trip can also be simulated in the World Model, to check collision risk and motion stability. Here the World Model is the predictor and testbed inside the reasoning system. It helps compare consequences before acting. It does not replace real-time execution afterward.

The Embodied Cerebellum: Turning Understanding Into Timely, Safe Action

Once the robot is in the real environment, the Embodied Cerebellum turns the high-level plan into motion. According to information disclosed by XGRIDS, the onboard system combines real-time perception and uses reinforcement learning to control gait, obstacle avoidance, and posture. The route should also follow spatial semantics, not only the shortest geometric path.[6]

The shortest path on a map may cut through a crowd, cross steps the current body cannot handle, or enter a space that does not fit the task. The real-time system has to execute while adjusting to the robot's physical limits and to changes in the environment, and while holding stability and safety.

The Smart Brain and the Embodied Cerebellum do different jobs. The Brain understands goals, builds task chains, and handles anomalies. The Cerebellum finishes the motion on time in a physical environment that keeps changing.

Execution Feedback: The Environment, and Then People

The Closed Loop KyOS describes does not stop when the task is done. Execution results are written back, and new preferences are recorded. For the same route or the same store, different people's understanding and choices also update through experience.

That adds a fourth piece: memory and continuous learning. Data from perception, decision-making, and execution returns to the world state and the task model. It shapes the next action, and it also shapes how the whole system understands spaces, tasks, and people.

Remembering preferences is only part of it. A complete KyOS-style system should also let people step into both high-level judgment and actual operation: confirm the system's reading when the goal is unclear; demonstrate or take over when the robot hits a motion it has not learned; choose among feasible routes so the system can see which option fits the current situation.

That matches XGRIDS's line that "robots do not replace humans; they are an augmentation of human capability, and also an extension of the self." A person issues the request; the robot enters the Real World on that person's behalf; people stay in the loop for value judgments, teaching, and final responsibility. As autonomy grows, people need less step-by-step control. As uncertainty grows, the system should hand control or the decision back sooner.

What KyOS is showing, then, is a loop rather than a one-way pipeline of "input instruction, output action":

Capture the Real World and build a World Model; understand the task and simulate it; execute in real time through the Embodied Cerebellum; have humans confirm, demonstrate, and correct when necessary; then write the results, experience, and preferences back into the system.

What KyOS Changes: Task Scalability

From a systems view, KyOS is not a replacement for ROS, World Models, Physical AI, or any one large model. It puts those pieces in positions where perception, reasoning, simulation, control, and learning can be combined, orchestrated, and updated over time.

That architecture is also why XGRIDS says "one croissant is just the beginning." The same abilities to perceive, understand, decide, and move through space are not limited to food delivery. If sensors, embodiment, skills, and task rules can be connected in a standard way, the same stack can extend to factory inspection, film production, urban spatial services, and home robotics.

Robotics does not scale by writing a new program for every customer, or by retraining the whole stack every time a task is added. Navigation, grasping, following, inspection, mapping, and interaction need to be composable, replaceable, and verifiable skills. Task workflows then get generated from the real environment.

Human-in-the-Loop can lower the cost of adding a task. Adding a task used to mean engineers writing rules, collecting a lot of data, and tuning parameters over and over. Later, a business user could demonstrate a task a few times, correct the serious mistakes while the robot runs, and mark preferences on a set of outcomes. Combined with existing skills, World Models, and simulation, the system has a chance to learn new tasks with less data.

Task scalability is then a teaching mechanism with a low cost, not only a software interface. The party that makes it easier for ordinary people to teach robots is more likely to move robots from a few standardized settings into many industries.

XGRIDS's Real2Sim path also supplies a data loop for that kind of scaling: bring real scenes into simulation, train and validate in the World Model, deploy to real robots; new cases from operation become inputs for the next round of models, policies, and tests.

KyOS is a systematic try at an approach where "the embodiment and the computing are provided by an ecosystem." Competitiveness is whether the whole set of capabilities can still be reused when the body changes, sensors are added, or new tasks appear, not only how strong any one module is.

KyOS panel of five system properties: executable, generalizable, scalable, safe, and evolvable

The Next Stage of Robotics Competition: From Model Capability to System Capability

Evaluating a robotic platform may need to look past model size, single-task success rates, or demo videos, and at questions closer to real use:

How long does it take to integrate a new type of sensor?
When the robot embodiment is changed, how much of the existing skill set can be reused?
When a new task is added, does the entire model need to be retrained, or can existing capabilities be combined?
If the network goes down, a sensor malfunctions, or a model times out, can the system still operate safely?
After failing in an unfamiliar environment, can it accumulate experience and iterate quickly?
When faced with ambiguity and high-risk choices, does it know when it should ask a person?
After a person provides one demonstration or correction, can that experience be turned into a reusable skill?
As the system learns, does the frequency of human takeovers genuinely continue to decline?
Can the same system move from a demonstration to continuous, stable operation over months at a time?

That is what "task scalability" means here: the system can add tasks, move across platforms, find faults, and upgrade at a cost that stays under control.

World Models cover how robots represent and predict the world. Physical AI covers how AI learns and follows physical laws. Perception, real-time control, and high-level reasoning take understanding, action, and decision-making. Human-in-the-Loop keeps feeding in human goals, experience, preferences, and responsibility. Combined into a Closed Loop, robots can move from performing a demonstration once to completing real tasks over a long period.

The next useful breakthrough in robotics may not be a larger "brain." It is more likely a nervous system and a learning setup around that brain: able to understand the world and still respond on time; able to reason and still know when to ask a person; able to finish familiar tasks on its own and, after demonstration, correction, and evaluation, pick up new tasks at lower cost.

When we look at KyOS and at the next generation of robotic systems, that is the thread worth following.

References

ROS 2 Design: Introduction to Real-time Systems
Jiang et al.: LAAIR: A Layered Architecture for Autonomous Interactive Robots
NVIDIA: What Is a World Model?
NVIDIA: What Is Physical AI?
XGRIDS / PR Newswire: XGRIDS Showcases at NVIDIA GTC 2026: Spatial Intelligence Drives AI into the Physical World
Google DeepMind: Learning through human feedback
Liu et al.: Robot Learning on the Job: Human-in-the-Loop Autonomy and Learning During Deployment
Hejna & Sadigh: Few-Shot Preference Learning for Human-in-the-Loop RL