Inter-2 reads interpersonal signals from video with audio, producing predictions across 10 scored social signals, a separate engagement status, and a confidence level for each detected signal. The model is produced through curriculum post-training of an omni-modal architecture, using benchmark data assembled specifically for social perception.
This post describes the task and evaluation framework behind Inter-2, the curriculum used to train it, what each stage is designed to achieve, and where the reported numbers come from. Three design decisions account for much of the resulting performance: a clean and densely annotated dataset, the separation of output structure from audio grounding during training, and maintaining strict parity between the configuration used during post-training and the configuration used during serving.
The task: from describing interaction to grounding social signals
Frontier multimodal models are trained for captioning, visual question answering, and general video question answering, objectives that primarily reward describing what is visible or answering questions about the content of a scene. Social signal scoring asks something narrower and harder to fake: the model must temporally ground a signal within a short clip, identify the evidence supporting that signal, and do so under a latency budget appropriate for real-time inference.
The label space is the signal ontology carried over from Inter-1 and consists of 10 scored signals: agreement, disagreement, skepticism, confidence, hesitation, frustration, interest, uncertainty, confusion, and stress, alongside a separate engagement status of engaged, neutral, or disengaged. Several signals may be present within the same clip, making the task inherently multi-label rather than single-class: a speaker can sound confident and skeptical in the same sentence, while a listener can show interest while simultaneously disagreeing.
The output contract is fixed because the downstream product code consumes it directly. Each prediction contains an engagement status, a set of detected signals, a categorical confidence level for each signal (high, medium, or low) and a short rationale citing the evidence used by the model.
Signal-Bench and EW-F1
Signal-Bench is our multi-label social-signal benchmark, consisting of video with audio annotated by trained raters at dense temporal resolution. Annotator disagreement is substantial and should not be treated simply as noise to be averaged away, because two raters can both be correct about a moment while assigning different signals when the available evidence is partial or compatible with more than one interpretation (Uma et al., 2021).
Plain macro F1 handles this structure poorly because it treats every label instance as equally certain, meaning that a model can be penalized in the same way for missing a signal that three raters marked with clear evidence and one that a single rater inferred from an ambiguous glance. This led us to EW-F1, which weights each instance according to the strength of the annotation evidence behind it, making the resulting score more closely track the cases in which a human observer would agree on the presence of a signal. We also built our own evaluation suite to evaluate both closed and open frontier models at scale.
Results: accuracy and inference speed
| Model | EW-F1 | Relative inference speed |
|---|---|---|
| Inter-2 | 47.0 | 3.5× |
| Closed frontier | 44.7 | 0.9× |
| Open frontier | 43.8 | 1.25× |
Inter-2 reaches an EW-F1 of 47.0, compared with 44.7 for the strongest closed frontier baseline and 43.8 for the strongest open-weight frontier baseline. The baseline groups consist of closed frontier models served through APIs and open-weight frontier models, with the table reporting the strongest entry from each group; to make the comparison as fair as possible, we give the frontier models the best available evaluation setup by running them through our purpose-built optimized harness for signal prediction.
The speed figures are relative to the 1.0× reference on the release latency sheet, while the EW-F1 figures come from our shared evaluator running on frozen benchmark splits rather than from the training code, which only emits predictions. Inter-2 leads the closed frontier by 2.3 EW-F1 points and the open frontier by 3.2 points, while also running faster than every baseline in either group.
The shape of the plot matters more than the single lead. The baseline models trade accuracy against speed along a visible frontier: the strongest scores are concentrated at 1.25× relative inference speed and below, while the fastest entries give up around ten points of EW-F1. Inter-2 is therefore not simply further along the same accuracy-speed trade-off; it sits outside that frontier.
Serving configuration and train/serve parity
Inter-2 runs at 3.5× on the release latency sheet, making it the fastest evaluated entry and the only one above 45 EW-F1. The next-fastest baseline runs at 2.8× and scores 36.3, which is 10.7 points lower, while every baseline above 43 EW-F1 operates at 1.25× relative speed or slower.
A common pitfall in multimodal modeling is the divergence between offline benchmark harnesses and production inference pipelines. When sequence layouts, frame sampling, or multimodal interleaving differ between training and serving, real-world performance drops sharply.
Training and serving use a single configuration: the same frame sampling, the same audio path, and the same corrected audio positional encoding. Every released checkpoint records the configuration under which it was trained so that an evaluation run and a served model can be checked directly against one another.
Why a curriculum
One-shot fine-tuning fits this data poorly for reasons that are specific to the task rather than generally true of post-training. Clips vary in how many signals co-occur, which changes the effective difficulty of the label decision; they vary in how much of the answer is carried by audio rather than vision, which changes which encoder must carry the relevant evidence; and they vary in how strictly the output contract binds, since some clips admit a short rationale while others require the model to explain competing signals.
Training across all of these conditions simultaneously produces a model that can become mediocre at each individual component. In particular, the model can learn the easiest available shortcut by predicting the modal signal from visual context and writing a rationale that sounds plausible without actually depending on the audio. Once that shortcut is established, later training does not necessarily remove it, because the loss is already low on the clips where the shortcut succeeds.
The curriculum exists to control the order in which those pressures reach the model (Bengio et al., 2009). We begin with output structure, then establish audio grounding, and only afterwards increase the density and complexity of the signals presented during training.
Stage 1: output structure on interaction video
Stage 1 uses supervised fine-tuning on the curated base mixture of interaction footage assembled for Inter-2. The objective at this stage is target behaviour rather than difficulty: the model learns the signal taxonomy, evidence-style rationales, engagement and confidence fields, and a stable output contract across naturalistic social video.
We deliberately do not begin with the hardest clips. A model that has not yet learned the contract produces malformed outputs on difficult examples, and those malformed outputs introduce training noise for reasons unrelated to social perception. The development split at this stage is therefore used to determine whether outputs parse correctly and carry the required fields, rather than whether accuracy is already competitive.
Stage 2: audio-grounded counterfactual continuation
Stage 2 continues from Stage 1 using matched original and muted versions of the same clips. For the muted copies, rationales are rewritten so that the model cannot cite audio evidence that is absent, while labels remain consistent where the visual evidence continues to support them. The pairing is the point: the same clip appears twice with a different evidence budget, forcing the model to condition its explanation on the evidence that is actually available rather than on the evidence it has learned to expect.
We measure the effect using counterfactual audio probes. Muting removes the audio stream, shifting misaligns it in time, and swapping substitutes audio from a different clip. Each intervention answers a different question: muting asks whether audio contributes at all, shifting asks whether temporal alignment matters rather than only global tone, and swapping asks whether the model can recognize contradictory evidence introduced by an unrelated audio stream. Muting is the probe we treat as the floor for how much of the prediction depends on sound rather than on visual priors.
The stage closes with the positional correction described above and produces the release checkpoint. Without this stage, models suffer from severe shortcutting: they predict signals from facial expression alone while generating rationales that cite vocal tone that does not actually exist. In diagnostic probes, un-grounded models hallucinated audio evidence on silent clips more than 12% of the time.
Intervention-style audio diagnostics in prior work informed the design of these probes (Wen et al., 2026). The contribution here is to combine counterfactual-audio robustness with a social-signal curriculum under train/serve parity, rather than treating the probes as a post-hoc diagnostic applied to a model trained without them.
Stage 3: density resampling
Stage 3 rebalances the training mixture toward clips in which several signals compete simultaneously. It runs as an ablation on top of the release recipe rather than as part of the shipped Stage 1 and Stage 2 path, because uniform sampling underrepresents dense clips: they are a minority of the corpus while representing a substantial share of the cases that matter in conversational interaction.
The failure mode this stage targets is precision collapse under density. A model tuned predominantly on single-signal clips tends to predict too few labels when several signals are present, while its confidence calibration can drift because it has rarely had to divide probability mass across competing signals. Stage 3, therefore, runs only after the audio-grounded path has stabilized, allowing density pressure to be introduced without competing with audio grounding for the same capacity.
Stage 4: complexity-aware selection
Stage 4 replaces heuristic resampling with structured selection, choosing training subsets against explicit density and complexity objectives. Instead of simply upweighting a predefined part of the dataset, it selects a set of clips that jointly covers the label combinations and complexity levels that are relevant for the task.
We use this stage for follow-on gains and ablations. It has produced our strongest results in later experiments.
Release path and selection policy
The shipped recipe is curriculum fine-tuning followed by the audio-grounded continuation, after which the resulting model is exported as a unified standalone model for serving. Checkpoint selection uses EW-F1 on frozen benchmark splits, scored by our shared evaluator rather than by the training code itself.
The model is post-trained with frozen perception towers. Freezing the vision and audio encoders keeps their general features intact and confines specialization to the decision layers, which also keeps each stage cheap enough to run several variants and choose between them on evidence.
The result is a model whose performance is not defined solely by whether it can identify a social signal, but by whether it can identify that signal within a temporal interaction, ground its prediction in the evidence available across modalities, express the result through a consistent structured output, and maintain that behaviour under the same configuration used for real-time serving.
References
- Bengio, Y., Louradour, J., Collobert, R., & Weston, J. (2009). Curriculum Learning. Proceedings of the 26th Annual International Conference on Machine Learning (ICML).
- Uma, A., Fornaciari, T., Dumitrache, A., Chamberlain, J., Kruschwitz, U., Le Brun, K., ... & Poesio, M. (2021). Learning from Disagreement: A Survey. Journal of Artificial Intelligence Research, 72, 1385–1470.
- Vinciarelli, A., Pantic, M., & Bourlard, H. (2009). Social Signal Processing: Survey of an Emerging Domain. IEEE Signal Processing Magazine, 26(4), 42–53.
- Wen, X., Mo, W. J., Fu, X., Cai, R., Zhu, T., Li, W., Xie, Y., Chen, M., & Qi, P. (2026). When Vision Speaks for Sound. arXiv:2605.16403.
