From Self-Report to Authentic Voice: Language as Measurement Signal

A behavioral health screener asks a student to rate a checklist item against a shared scale — "not at all," "several days," "more than half the days" — and treats the resulting number as the measurement. The instrument does the same work whether the student circling "several days" is guarded, articulate, distracted, or silently translating the item from another language in their head; the scale absorbs all of that into one number and discards the rest. Unconstrained language works differently. When a student answers an open-ended prompt in their own words, the response carries lexical choice, structure, elaboration, hedging, and framing that a five-point item cannot capture, and a growing body of clinical and educational measurement research treats that language itself as a scorable signal rather than as color commentary around the "real" score. This section lays out what that research base currently supports, what it does not, and why IMPACTER's production scoring architecture is built the way it is in response to it.

The clinical evidence base for language as a measurement signal is largest around depression detection. A 2026 systematic review and meta-analysis in npj Digital Medicine screened 892 records, included 123 studies covering 129 effects, and pooled results across 50 independent datasets to estimate what natural-language and machine learning methods can currently achieve when detecting depression from spoken or written language (Fisher et al., 2026). The pooled estimate across 43 studies and 40,983 text samples was 0.80 accuracy (95% CI 0.76–0.83), with precision of 0.78 across 28 studies, recall of 0.76 across 33 studies, an AUC of 0.79 across 14 studies, and a more conservative balanced-accuracy figure of 0.71 across the 16 studies that reported it — a spread across metrics that itself signals how much the achievable number depends on which metric a given study chose to report. The review's subgroup analysis, run by language, text source, feature type, and classifier, found accuracy was highest specifically in studies built on structured clinical interviews rather than casual or social-media text, in non-English-language samples rather than English-language ones, and in studies using linguistic or embedding-based features rather than surface word counts — a pattern on the language axis that argues the signal lives in how carefully something is elicited, not simply in which language it happens to be said in. A broader review the same year extended the picture beyond depression to automated speech-based assessment across schizophrenia, bipolar disorder, PTSD, anxiety, and eating disorders, screening 1,395 studies and including 127 (Low et al., 2020). Most of that included literature still clustered around depression, schizophrenia, and bipolar disorder specifically; PTSD, anxiety, and eating disorders — despite being explicitly in scope for the review — turned up far less evidence, which is exactly the pattern this section keeps returning to. Set side by side, these two reviews establish a real but bounded ceiling: language-based detection of a single, well-studied clinical construct, assessed under a structured elicitation format, performs meaningfully above chance, and the evidence thins quickly once the target construct, the population, or the elicitation format moves away from that well-trodden center. The diagram below sets that pooled, metric-dependent spread next to the single-number mechanism the opening paragraph described, so the contrast is a comparison rather than a claim to take on faith.

That evidence base also documents its own exposure. A 2024 review in the JMIR-affiliated Interactive Journal of Medical Research surveying natural-language-processing approaches to depression screening names the specific risks a system built on this method has to design against (Teferra et al., 2024). It flags reidentification risk in clinical text even after standard anonymization and recommends differential privacy as a companion safeguard rather than treating anonymization alone as sufficient. It documents bias in common word-embedding models — GloVe and Word2Vec by name — along religion, race, gender, nationality, sexuality, and age lines that can degrade accuracy specifically for underrepresented groups rather than uniformly across a sample. It names interpretability limits inherent to black-box deep learning as a validity concern in its own right, not just a usability inconvenience. And it calls explicitly for culturally and linguistically sensitive models, citing evidence that how distress gets expressed in language varies across languages and cultures. That last point lands directly on a design constraint BHPM cannot defer: students will answer BHPM prompts in whatever language they think in, and a model trained mostly on English clinical text is exactly the kind of system this review is warning about. None of this is a reason to treat language-based measurement as unusable — the clinical NLP field would not be publishing systematic reviews of its own aggregate performance if the answer were simply "don't." It is a reason to treat every one of those failure modes — reidentification, embedding bias, opacity, and monolingual training data — as a design constraint to be engineered against from the start, not a problem discovered after deployment.

The multilingual angle threads through several of these findings, not just Teferra et al.'s caution. Fisher et al.'s own subgroup analysis found accuracy highest in non-English-language samples, not lowest — a result that runs against the intuition that English-trained pipelines transfer best, and suggests the studies driving that subgroup were built around structured, purpose-built elicitation in the target language rather than a repurposed English-first pipeline translated after the fact. Lopez et al. and Ober et al., discussed below, sit on the same side of that finding from the education-measurement side: both evaluate systems interpreting language from students who are actively acquiring English, not fluent adult clinical patients, and both report the system performing well on that population specifically. None of that licenses treating multilingual administration as a solved problem — Teferra et al.'s call for culturally and linguistically sensitive models exists precisely because the underlying embedding models still carry documented bias, and one positive subgroup finding in one meta-analysis is not a validated multilingual pipeline. But it does mean the research base points toward purpose-built, in-language elicitation and scoring as the direction with actual evidentiary support behind it, not away from it, which matters directly for a screening population that will answer BHPM prompts in whatever language they think in.

Educational measurement has its own, more directly relevant track record with open-ended student language, and it predates the current wave of clinical NLP research. ETS's conversation-based assessment program builds learner models that combine evidence from multiple-choice items, constructed responses, and dialogue with a conversational agent into a single account of what a student knows or can do, with the conversation's complexity adapting dynamically as the system updates its read on the student's real-time progress (Zapata-Rivera et al., 2022). That adaptive design matters here because it treats open-ended dialogue as a live measurement channel, not a static text sample scored after the fact once the conversation is over. A 2021 ETS research report evaluated one such prototype, ELLA-Math, specifically on its ability to interpret middle-school English learners' written responses to math prompts, and found the system highly accurate at correctly classifying the sentences those students actually produced (Lopez et al., 2021). The scope of that finding is worth stating precisely: it is one prototype, evaluated in one subject domain, with one population — a real limitation, not a broad validation of automated interpretation across every context. But the population it was tested on is the relevant one. It is not a claim about fluent adult English text; it is a claim about interpreting open-ended written language from students who are actively acquiring the language they are writing in, which is closer to the linguistic diversity a multilingual, K-12 student population will actually produce than anything in the depression-detection literature above. That result matters specifically because it is a precedent for automated interpretation of open-ended language from students across a range of linguistic backgrounds, in an assessment context, rather than a claim about clinical text from adult patients.

The most directly load-bearing study for how IMPACTER's pipeline is built, though, is a 2026 study in the Journal of Educational Data Mining that compared several large language model configurations against expert human coders on more than 10,000 conversational turns from 107 English learners, coding constructs including persistence, self-efficacy, interest, perceived challenge, and prior knowledge from a conversation-based math assessment (Ober et al., 2026). The study varied both model temperature and model size across multiple ChatGPT-4o configurations and compared each configuration's coding against expert human coders construct by construct, rather than reporting one blended agreement figure across all five constructs at once. The finding was not that a larger or more capable model closes the gap with human coders. It was that how precisely a construct is operationally defined governs human-model agreement far more than model scale does: self-efficacy, the most tightly and operationally defined construct in the study, showed the strongest human-model alignment and needed a low, near-deterministic temperature setting to hold that alignment steady, while the more theoretically diffuse constructs behaved inconsistently across different temperature settings — the same construct, coded by the same model family, produced different agreement levels depending on a decoding parameter that has nothing to do with the construct itself. That is a documented reproducibility problem, not a one-off measurement artifact, and it is open-access research — JEDM publishes without a paywall, with a supplemental materials record on OSF, which is part of why it is verifiable rather than taken on faith. A separate 2023 study accepted at NeurIPS reinforces that same instability from a different angle. It shows that large language models' own chain-of-thought explanations can misrepresent the actual basis for an output: when researchers quietly injected a biasing feature into the input — for instance, reordering which multiple-choice option came first — models still produced fluent, plausible-sounding reasoning that omitted the injected feature entirely, even though that feature measurably changed the model's answer, with accuracy dropping by as much as 36 percent across 13 benchmark tasks once the bias was in place (Turpin et al., 2023). Set against each other, these two studies describe a specific, well-documented failure mode of generative language models used for construct coding: reasoning and consistency that look plausible turn by turn but are not reliably faithful to the same construct, in the same way, twice.

That construct-clarity argument is not new to Ober and colleagues; it echoes a long-standing position in validity theory. A chapter on the cognitive foundations of automated scoring makes the same case from first principles rather than from an empirical LLM comparison: scoring-system design should be traceable to an explicit cognitive model of the construct the assessment is meant to elicit, not just statistical fit to a set of human ratings, and the transparency of a scoring system's logic is itself a validity concern rather than a mere usability nicety (Bauer et al., 2020). Held up against Ober et al.'s empirical finding, the two arguments converge on the same conclusion from opposite directions — one from measurement theory, one from an LLM-coding experiment — that a scoring system's trustworthiness rests on how explicitly its target construct is defined and how transparently its logic can be traced back to that definition, not on how large or fluent the underlying model is.

Ober and colleagues' own prescription is rubric-first construct definition, and IMPACTER's production architecture is built to satisfy that prescription structurally rather than through prompt engineering after the fact. The scoring pipeline is discriminative, not generative: a fine-tuned transformer encoder is paired with a CORAL ordinal regression head. The encoder descends from the disentangled-attention family introduced by He, Liu, Gao, and Chen, which represents each token with separate content and relative-position vectors and adds an enhanced mask decoder that incorporates absolute position information during pretraining — a design that, trained on half the data, still outperformed RoBERTa-Large on MNLI, SQuAD v2.0, and RACE (He et al., 2021). A later refinement in that same lineage swaps in ELECTRA-style replaced-token detection and gradient-disentangled embedding sharing for further gains in sample efficiency (He et al., 2021) — the specific line IMPACTER's production DeBERTa-v3-base encoder sits on. CORAL, for its part, enforces rank-consistent, monotonic score bands by construction, through a weight-sharing constraint across the model's binary sub-classifiers, rather than leaving ordinal consistency to chance the way independent binary decisions would (Cao et al., 2020); it was introduced for a very different task, image-based age estimation, and has since become enough of a standard reference implementation for ordinal deep learning that it ships as an open-source library other teams reuse directly. In production, this pairing is IMPACTER's Model Suite v6.0, per the organization's published Model Card: a DeBERTa-v3-base encoder plus CORAL ordinal regression head, 184 million parameters, run under frozen, versioned model releases, with an average quadratic-weighted kappa of 0.918, average exact-match accuracy of 77.3%, and average adjacent accuracy of 98.0% (IMPACTER Education Corp. (2026). IMPACTER Model Card, 2026). A given student response passes through the model once, deterministically, and produces one ordinal score; the architecture performs no sampling at inference and returns no second, different answer on a second pass over the same text. That is a categorically different failure surface than the one Ober et al. and Turpin et al. document, because it is not a generative model reasoning its way to an answer — it is a fixed function evaluated once against a rubric-trained ordinal scale. Encoder-compression research points at the same design space from a different angle: DistilBERT, produced by applying knowledge distillation during pretraining rather than only at fine-tuning time, cuts parameter count by roughly 40 percent and inference time by roughly 60 percent while retaining about 97 percent of the larger model's language-understanding performance (Sanh et al., 2019). That result matters here less for the compression itself than for what it demonstrates about the family this section is describing: encoder architectures in this lineage are well-studied enough that researchers can trade size for speed in predictable, measured increments, which is a different kind of engineering maturity than the reproducibility problems documented above in generative, decoder-based coding. The encoder-plus-ordinal-head choice is a deliberate point in that well-studied design space, not an ad hoc one.

The distinction that follows from all of this is worth stating plainly rather than leaving implicit. The clinical NLP literature above establishes what language-based detection of a diagnostic-adjacent construct like depression can currently achieve in the aggregate, across many studies, many populations, and many methods — a ceiling that, per Fisher et al., tops out around 80 percent accuracy even for the single best-studied construct in the field. IMPACTER's scoring pipeline is not making that claim, and is not trying to clear that particular bar, because it is not answering that question. It is scoring authentic, multilingual student language against IMPACTER's own rubric-defined ordinal bands for durable-skills constructs, using a deterministic, versioned model built specifically to avoid the reproducibility failure mode the construct-coding literature documents in generative alternatives. That is a narrower, more tightly specified measurement task than clinical detection: the construct is IMPACTER's own, rubric-defined and expert-authored rather than inherited from a diagnostic manual; the scale is ordinal rather than binary; and the architecture is discriminative rather than generative. Evaluated on its own terms, that narrower task is exactly what Section 3 turns to next.