In late 2017, a junior at Ladue Horton Watkins High School took the ACT twice. He scored a 28 the first time, a 29 the second time.
Those aren't bad scores, by any means, but the kid was a straight-A student with ambition, and neither was good enough for his purposes. So he did test prep — an hour a day. This being Ladue, he also hired a tutor. And he took the test again in February 2018.
This time, success — the kid scored a 33, high enough to place in the 98th percentile.
The bad news came seven months later.
Out of the blue, the student got a letter in September 2018 from ACT Test Security. The Iowa-based nonprofit that administers the ACT had flagged his February score as the product of possible cheating.
It invoked no details from the day of the test — nothing about where he was sitting, or who he might have cribbed answers from. The letter noted simply that another examinee who used the same test form had made similar mistakes in the multiple-choice sections. "The probability of these identical responses and identical incorrect responses occurring given chance is very small," its letter said.
And with that, the student had three choices: He could agree to cancel the results in question. He could take a private retest in hopes of confirming his score. Or he could try to persuade the organization's "review panel" that his scores were valid.
This being Ladue, the kid hired a lawyer — and that lawyer hired a statistician. And what the statistician found was alarming: The ACT's statistical analysis, the lawyer concluded, was "fatally flawed."
Albert Watkins, the Clayton-based attorney who represented the student, says his expert, a data scientist named Adam Gold, ran a simulation showing that the threshold for "identical incorrect responses" should be fifteen, not eleven. "For ACT to utilize flawed analyses as a basis for its concerns is not simply reckless, but unpardonable," Watkins wrote in a letter to the non-profit on September 19. He demanded ACT back down and let the student use the score.
Edward Colby, senior director for media and public relations for ACT, said the test service could not comment on any individual cases due to "privacy reasons." While he provided a link to an overview of the process
, he also wouldn't say how many students find their scores flagged by the service — or provide statistics on how many end up getting thrown out after that.
In this case, the news was good for the Ladue student. Last month, ACT quietly closed its review of the student's score, telling Watkins it would "continue to treat his scores as valid." The two-paragraph letter to Watkins does not provide a reason or explain which evidence provided ACT found compelling — or whether it was merely the existence of a lawyer armed with a statistical analysis of his own threatening to escalate the matter.
Questioned about whether ACT has concerns about its statistical model, Colby says it does not. "ACT has complete confidence in the statistical analyses that are performed in our score review process," he writes in an email. "Statistical analysis is only one piece in identification and review of testing irregularities."
But Watkins thinks it should. "Nobody likes a bully," he says. "And second of all, nobody likes a bully that is, by the very nature of event in which they're bullying, dealing with people at a real disadvantage." Like students who are months removed from taking a test where they thought they earned the score they needed. Students who can't, for the most part, afford to fight back without suffering significant cost to their reputation ... or to their parents' finances.
"How many of these kids are out there?" Watkins says, pointing out that few are likely to take on ACT publicly once receiving an ominous letter. "When you think about it, it's a lot like getting the clap," he says. "You don't want it being the subject of public record."
Colby won't say how many students found themselves flagged for eleven to fourteen identical incorrect responses — and therefore might fall into the camp where Gold's analysis would be significant.
Watkins thinks that's because ACT knows it has a problem. "Their response was such that whatever evidence they had, they didn't deem it compelling enough or worthy enough to stick to their guns," he says.
He's speaking out in hopes of raising awareness among others who might find their scores flagged without good cause.
"No one tells you it's flawed," he says of the ACT's cheating-detection analysis. "My hope is that the public knows enough to be wary of it."
We welcome tips and feedback. Email the author at email@example.com