Humans spend very little time thinking about how they recognize behavior in other people. We notice hesitation, uncertainty, engagement or skepticism almost automatically, and most of the time we struggle to explain exactly why we reached those conclusions. That intuition is useful in everyday life. It is much less useful when the goal is to train a machine.
A model cannot learn "hesitation" simply because humans recognize it when they see it. It first requires a definition of what hesitation actually is, what observable evidence supports it, where that evidence becomes ambiguous, and how it differs from closely related behaviors such as confusion or uncertainty. That turns out to be a much harder problem than it first appears.
One of the assumptions people often make about behavioral AI is that the difficult part lies in the modeling. Collect enough data, train a sufficiently capable architecture, and the patterns should emerge naturally. Our experience has been almost the opposite. The difficult part begins long before the first model is trained, because someone has to decide what the model is supposed to learn in the first place.
One behavior rarely means one thing
Imagine someone answering a question during a meeting. They pause halfway through a sentence and look away before continuing. What happened? Even though the pause and gaze might be obvious, the underlying meaning behind them is not as clear.
One person may interpret that moment as hesitation. Another may see someone carefully choosing their words. A third may think they are recalling information. Someone who knows the speaker well may consider the behavior completely typical, while somebody meeting them for the first time may reach a different conclusion altogether. None of these interpretations is obviously unreasonable.
Human behavior rarely provides a single correct answer.
The relationship between behavioral cues and social signals is almost never one-to-one. Hesitation may appear through pauses, repetitions, filler words, changes in speaking rate or gaze aversion, often several at once. Equally, a single cue such as looking away may signal more than just hesitation and support several different interpretations depending on the broader conversational context.
That many-to-many relationship is one of the defining characteristics of social behavior. It is also one of the reasons why behavioral science has become such an important part of our work.
To teach a model any of this, someone has to turn hours of video into structured labels it can actually learn from. That process is annotation, and it is where the ambiguity described above stops being a theoretical observation and becomes something you have to make a concrete decision about, clip by clip.
The problem with forcing a single answer
One possible response would be to remove ambiguity altogether. A data annotation protocol could simply specify that whenever behaviors A, B and C occur together, the correct label is hesitation. The resulting dataset would probably become more consistent, but consistency alone is not necessarily the same thing as accuracy. Human interaction does not follow rigid behavioral rules, and a protocol that ignores context risks teaching both annotators and models to ignore the very thing that gives behavior its meaning.
The opposite approach is equally problematic. If annotators are simply asked to rely on intuition, each person inevitably brings a slightly different understanding of what counts as hesitation, engagement or skepticism. The labels become inconsistent, not because the annotators lack expertise, but because the concepts themselves remain underspecified. Neither extreme is particularly satisfying.
Building a shared language
Our approach has been to treat behavioral science as the conceptual foundation rather than as a validation step after the model has already been built.
Before annotating large amounts of video, we needed a behavioral taxonomy with clear definitions, examples, distinctions between related signals, and descriptions of the behavioral cues that typically support each interpretation. Developing that taxonomy meant drawing on behavioral science, cognitive psychology, affective computing, linguistics and related disciplines. Even then, the taxonomy is not intended to eliminate judgment. Behavior is too context-dependent for that. Instead, it provides a common language that allows trained annotators to reason about the same interaction using shared concepts rather than entirely personal intuition.
In practice, that means our annotators come in with at least a bachelor's-level background in behavioral or social science, several with graduate degrees, and go through an onboarding process followed by a calibration phase: a set of annotated videos reviewed individually against expert feedback before anyone annotates independently. The taxonomy only does its job if the people using it have actually been trained to use it the same way.
This distinction matters because annotation is not simply a data-labeling exercise. It is the point at which behavioral theory becomes machine-readable structure. Every decision about what counts as evidence, how overlapping signals should be handled, or how uncertainty should be represented eventually shapes the data the model learns from.
A shared language doesn't mean a single answer
People often assume disagreement between expert annotators is evidence that somebody made a mistake. Sometimes that is true. More often, disagreement reflects something inherent about the behavior itself. That is, social signals are "fuzzy" and occur simultaneously: a person may appear both confident and uncertain in the same answer, or remain engaged in a conversation while also feeling frustrated by it. Two behavioral scientists observing exactly the same interaction may simply assign different weight to the same behavioral evidence. One may see uncertainty where another sees careful deliberation. Neither interpretation is necessarily incorrect. They may simply represent different, yet defensible, readings of genuinely ambiguous and multifaceted behavior.
This is why every clip goes through more than one annotator, at minimum three independently, and why we measure agreement rather than assume it.
Rather than treating disagreement as noise to eliminate, it becomes information in its own right.
It tells us something about the confidence we should place in a particular interpretation, and it reminds us that the uncertainty exists in the behavior itself rather than in the people observing it. A useful model should be capable of reflecting that uncertainty instead of presenting every prediction as equally certain.
What this means for social AI
As AI systems increasingly move into environments where they interact with people rather than simply process information, the quality of their understanding will depend less on whether they can produce fluent language and more on whether they can interpret human interaction with appropriate caution.
That does not mean avoiding conclusions altogether. It means grounding those conclusions in observable evidence, acknowledging where several interpretations remain plausible, and making the reasoning behind a prediction available for inspection.
Behavioral science becomes important for exactly that reason. It provides a way of translating something as ambiguous and context-dependent as human interaction into concepts that can be defined, validated, annotated and eventually learned by a model. The goal is not to remove ambiguity from human behavior, because that would also remove much of what makes behavior meaningful. It is to build models that know when it is ambiguous and hold that same kind of calibrated uncertainty our annotators already do, rather than presenting every prediction as equally certain.
