The Measurement Problem at the Heart of Value-Based ABA

Two Measures, Opposite Directions

An observational study of 154 children aged 3 to 16 who received 24 months of ABA within an integrated health system compared three ways of judging improvement: progress on individualized treatment goals, a caregiver measure of progress on treatment participation goals, and the Vineland Adaptive Behavior Scales adaptive behavior composite. They did not agree. No patient-centered measure correlated significantly and positively with change on the adaptive behavior composite at 12 or 24 months, and neither did the caregiver measure.

The sharpest finding is directional. Between the first year and the second, the share of children achieving clinically meaningful gains on their individualized goals rose. The share gaining on adaptive behavior fell. Two defensible definitions of progress, tracking the same children, moved opposite ways.

That study, by Kristen Choi and colleagues in BMC Pediatrics in 2022, was observational, drawn from a single health system, and modest in size, so it cannot settle the question on its own. It matters because one of the two measures it compared is already carrying financial weight. Kyo Autism Therapy’s value-based arrangement with Magellan Healthcare, which has run in California for more than three years, assesses longitudinal quality on the Vineland scales alongside capitated monthly payments, and the roughly 80% progress figure the companies cite is a Vineland figure.

Two defensible definitions of progress, applied to the same children across the same months, moved opposite ways. A payment model has to pick one.

Validated for Diagnosis, Deployed for Payment

A 2025 review in Behavioral Sciences by Mirko Uljarevic appraised the published psychometric evidence and technical manuals for the Vineland-3 and the Adaptive Behavior Assessment System, Third Edition, examining how well they characterize social communication and interaction in comprehensive and intensive behavioral intervention programs. The review, which analyzed existing evidence rather than generating new data, found that evidence for several key psychometric characteristics is either unavailable or suggests less-than-desirable properties, and that practical evaluation revealed weaknesses in ongoing intervention monitoring and clinical decision support.

The appraisal separated use cases, and that separation is the substance of the critique. Both instruments have significant strengths for cross-sectional outpatient assessment, particularly identifying intellectual disability. Their substantial weaknesses show up when applied to outcome assessment during intervention. The review recommended adopting measures developed specifically for the intervention context.

A counterweight belongs here. The Vineland-3 also offers growth scale values, an alternative scoring approach that a 2025 study in the Journal of Child and Adolescent Psychopharmacology found may be useful for measuring within-person change, reporting excellent split-half and test-retest reliability across subdomains except coping, where reliability was lower, with floor effects in some domains including domestic. That study was conducted in children with SCN2A-related disorders in preparation for clinical trials, not in autism populations receiving ABA, so its findings do not transfer directly. Taken together, the evidence suggests the problem is the composite score in common use rather than measurement itself, and that better-suited scoring approaches exist without yet having been validated for this purpose.

No Agreed Definition of Success

Beneath the instrument question sits a definitional one. ABA treatment targets are individualized by design, a clinical strength that leaves a common benchmark with nothing uniform to count. Industry participants describe the consequence directly: standardized data sets are a precondition for genuine value-based contracting, and until the field has them, such contracts will remain few and far between.

Deepa Shah, chief growth officer at Kyo, put the design question at the center at an industry conference in March. “I think value-based care can drive quality, but it’s not guaranteed if it’s not designed well,” she said. “Incentives shape behavior. If the metrics are well-designed and not narrowly defined or overly prescriptive, which can then distort care, we have seen it drive immense quality.” Her position is that metric design determines whether the model works, and that her own company’s experience shows it can.

The Risk-Adjustment Gap

A valid progress measure would still leave a second problem. Autism spans a wide range of support needs, and a model that fails to adjust for that range will penalize providers who take the hardest cases. A 2025 analysis in Behavioral Sciences examining value-based care and accountable care organizations argued that while such payment models allocate more for medically complex patients, there is little evidence they adequately account for complexity related to neurodevelopmental disability, and that models failing to do so may unfairly penalize clinicians caring for children with higher support needs.

What Would Have to Be Built

Some standards work is underway. The Council of Autism Service Providers released version 3.0 of its practice guidelines, which address outcome measures and add attention to tracking longer-term outcomes, and the organization operates an accreditation program for clinical quality. Those are process and quality frameworks rather than a settled outcome metric, and the gap between the two is where value-based contracting currently stalls.

What the field lacks is an agreed measure, or a small set of them, that clinicians accept as valid and payers accept as auditable, with risk adjustment attached and evidence that it detects real change during treatment. Until that exists, value-based contracts in ABA will keep being negotiated one at a time, each carrying its own definition of success. The next article in this series examines the organizations running those arrangements and what their contracts actually pay for.

AT A GLANCE

SOURCES & REFERENCES