By Richard Wexler, Executive Director, National Coalition for Child Protection Reform
November, 2016; Updated August, 2022, December, 2023
Introduction: "The machine has labeled you as high-risk."
In the dystopian science fiction movie Minority Report, people are arrested and jailed based
on the predictions of three psychics in a bathtub that at some point in the
future they will commit a crime. The film was meant to be a cautionary tale.
But
in the field of child welfare, there are many who seem to be responding to Minority Report the way Mike Pence might respond if he read The Handmaid’s
Tale – seeing it as a blueprint instead of a warning.
No, they are not proposing to rely on psychics in a
bathtub. Instead they’re proposing
something even less reliable: using “predictive analytics” to decide when to tear
apart a family and consign the children to the chaos of foster care. Giant software firms claim they can use
demographic information and other “data points” to create algorithms that predict
who will commit a crime, or who will neglect their children.
 |
| "Big Data" told us she would be president. (Photo by Gage Skidmore) |
It’s the same approach used so brilliantly by organizations
such as FiveThirtyEight and The New York
Times to predict the outcome of the 2016 Presidential Election.
But, as a
Times analysis points out, there
is reason for concern about predictive analytics that goes far beyond that one
“yuuuge” failure. And those concerns should extend to child welfare.
● Predictive Analytics already has gone terribly wrong in
criminal justice, falsely flagging Black defendants as future criminals and
underestimating risk if the defendant was white.
●In child welfare, a New Zealand experiment in predictive
analytics touted as a great success wrongly predicted child abuse more than
half the time.
●A predictive analytics model spreading all over the country, called Rapid Safety Feedback, has failed spectacularly in Illinois.
● In Los Angeles County, another experiment was hailed as a huge success in spite of a “false positive” rate of more than 95 percent. And that experiment was conducted by the private, for-profit software company that wants to sell its wares to the county. (Los Angeles has now quietly dropped this experiment, but still is pursuing predictive analytics.)
●A predictive analytics model spreading all over the country, called Rapid Safety Feedback, has failed spectacularly in Illinois.
● In Los Angeles County, another experiment was hailed as a huge success in spite of a “false positive” rate of more than 95 percent. And that experiment was conducted by the private, for-profit software company that wants to sell its wares to the county. (Los Angeles has now quietly dropped this experiment, but still is pursuing predictive analytics.)
● The same company is developing a new approach in Florida. This one targets
poor people. It compares birth records to three other databases: child welfare system
involvement, public assistance and “mothers who had been involved in the
state’s home visiting program.”
So listen up “at-risk” new mothers: In the world of predictive analytics,
the fact that you reached out for help when you needed it and accepted
assistance on how to be better parents isn’t a sign of strength – it’s a reason
to consider you suspect, and make it more likely that your children will be
taken away.
 |
| In Pittsburgh, they try to slap an invisible "scarlet number" risk score on every child - at birth. |
The Associated Press reports that when, finally, this model, known as the Allegheny Family Screening Tool (AFST), the subject of so much fawning news coverage, faced
an independent evaluation - one where the county itself didn't chose the
evaluators, they found that the model had a serious racial bias problem. An
earlier "ethics review" may have been unethical. Even that ethics
review was premised on the fact that the algorithm only would be applied where
there was an allegation of abuse or neglect - not to every child at
birth. The U.S. Department of Justice is is investigating whether AFST is biased against families where a parent is disabled. This is one of the cases they're examining, the case of Lauren and Andrew Hackney. And that case includes a revelation that punctures one of the key defenses of AFST: The claim that the algorithm is just advisory, humans still make the decisions - and the humans who make the decisions don't even know the risk score.
We’ve always known that the claim about workers not knowing
the risk score was b.s. They may not
know the precise risk number (it can range from 1 to 20) but if they’re told to rush out and
investigate this family, they know AFST has labeled the family “high risk.” So of course that’s going to bias the
investigator.
And now we know more. In another story about the Hackneys, this one in the ABA Journal, we learn who really calls the shots - human or machine. According to the Hackney’s lawyer:
when the Hackneys asked their caseworker what they could do to get their daughter back, “the caseworker said, ‘I’m sorry. Your case is high-risk. The machine has labeled you as high-risk.’ [Emphasis added.]
There
is a prescient discussion of the failure in Pittsburgh in this excerpt in Wired from Prof. Virginia
Eubanks' book, Automating Inequality. Among other
things, she was first to reveal Pittsburgh human services officials
seriously were considering something they subsequently
did: implementing a second, Orwellian algorithm that stamps a "risk
score" - a kind of invisible "scarlet number" on every child born in the county - at birth (unless
the family takes advantage of a very limited opportunity to opt out). The
county claims the score from this algorithm, will be used only to target
"prevention" as part of a program called "Hello Baby."
How
did the county solve the ethics review problem? By commissioning two more
ethics reviews. But the information the county released about those other
ethics reviews turned out to be incomplete.
At
no point does the county address the issue of informed consent - the fact that
vast amounts of data are being taken, disproportionately from poor people,
without their permission and then potentially used against them.
When Facebook misused data it obtained from users voluntarily, it was fined more than $5 billion. But Allegheny County can take data from poor people and turn it against them with impunity.
 |
| As we said in 2019 |
In fact, the designers of AFST and "Hello, Baby" and similar algorithms elsewhere want to go further. They are part of the team offering up the most Orwellian algorithm yet: the Cross Jurisdiction Model Replication project.
Like “Hello, Baby,” CJMR generates a risk score for every child at birth. Unlike Hello, Baby there is no opt-out at all. And while with AFST the developers bragged about not explicitly using race (while using poverty), and with Hello Baby they claimed (falsely) that they weren’t using poverty, this time there’s no more let’s pretend. The use of race and poverty is out in the open.
The campaign is led
largely, though not exclusively, by the field’s worst extremists – those who
have been most fanatical about advocating a massive increase in the number of
children torn from everyone they know and love and consigned to the chaos of
foster care – and also by those most deeply “in denial” when it comes to the
problem of racial bias in child welfare.
For example, the co-designer of the Pittsburgh algorithms and the CJMR project, Emily Putnam-Hornstein, denies the field has a racism problem, demeans Black activists, and takes pride in being part of a group that defends a self-described "race realist" law professor who hangs out with Tucker Carlson. She has said "I think it is possible we don’t place enough children in foster care or
early enough" and signed onto an extremist agenda that proposes requiring anyone reapplying for public benefits, and not otherwise seen by "mandatory reporters," to produce their children for a child abuse inspection.
In the spirit of CJMR, others –
leading researchers in the field - have
even argued that “prenatal risk assessments could be used to identify
children at risk of maltreatment while still in the womb.” As with "Hello, Baby" and CJMR these
researchers argue that such targeting should be used in order to provide help
to the mothers, that’s not how child welfare works in the real world.
Predictive analytics is a fad that presents serious new dangers to children in impoverished families, especially children of color. That’s because predictive analytics does not eliminate the racial and class biases that permeate child welfare, predictive analytics magnifies those biases. Predictive analytics amounts to data-nuking impoverished families. It is computerized racial profiling.
Indeed, when ChildTrends, an organization that specializes
in analyzing data on children’s issues, published its 2015 list of Top Five Myths About Child Maltreatment, #1
was: “We can predict which children will be maltreated based on risk factors.”
ChildTrends explains:
Risk factors associated with child maltreatment include extreme poverty, family unemployment, caregiver substance abuse, lack of understanding of child development, and neighborhood violence. However, each of these only weakly predicts the likelihood of maltreatment.
For example, although maltreatment is more common among families living in poverty than among other families, the majority of parents with low incomes do not maltreat their children. When risk factors are present, protective factors can mitigate the likelihood of maltreatment. Such protective factors include parental social connections, knowledge of parenting and child development, concrete support in times of need, and children’s social-emotional competence.
Because maltreatment is so difficult to predict, prevention approaches that strengthen protective factors among at-risk families broadly—even if the risk is low—are likely to be most effective in reducing maltreatment.
The stakes
As is always the case when advocates of finding new ways to
interfere in the lives of impoverished families try to justify trampling civil
liberties, they misrepresent the “balance of harms.” They claim that investigating families based
on the potential for what is known in Minority
Report as “future crime” is a mere inconvenience – no harm done if we
intervene and there’s no problem, they say.
But if they don’t intervene, something terrible may happen to a child.
But a child abuse investigation is not
a benign act. It can undermine the
fabric of family life, creating years of insecurity for a child, leaving severe
emotional scars. The trauma is compounded if, as often happens, the
investigation includes a stripsearch of a child by a caseworker looking for
bruises. If anyone else did that it would be, in itself, sexual abuse.
And, of course, the trauma is vastly
increased if the investigation is compounded by forcing the child into foster
care.
Thanks to decades of what has been aptly called "health terrorism" - misrepresenting the true nature and scope of a problem in the name of "raising awareness" -- when
we think of child abuse the first images that come to mind are of children
brutally beaten, tortured and murdered.
But the typical cases that dominate the caseloads of child welfare
workers are nothing like the horror stories. Far more common are cases in which
family poverty has been confused with “neglect.” Other cases fall between the
extremes.
So
it’s no wonder that two massive studies involving more than 15,000 typical cases found that children left in their own homes fared
better even than comparably-maltreated children placed in foster care. Several other studies A third, smaller study, using various methodologies, have reached the same conclusion.
· When a child is needlessly thrown into foster care,
he loses not only mom and dad but often brothers, sisters, aunts, uncles,
grandparents, teachers, friends and classmates.
He is cut loose from everyone loving and familiar. For a young enough child it’s an experience
akin to a kidnapping. Other children
feel they must have done something terribly wrong and now they are being
punished. One need only recall the cries of children separated at the Mexican border to understand that the emotional trauma can last
a lifetime.
· That harm occurs even when the foster home is a good
one. The majority are. But the rate of abuse in foster care is far
higher than generally realized and far higher than in the general
population. Multiple studies have
found abuse in one-quarter to one-third of foster homes. The rate of abuse in group homes and
institutions is even worse.
· But even that isn’t the worst of it. The more that workers are overwhelmed with
false allegations, trivial cases and children who don’t need to be in foster
care, the less time they have to find children in real danger. So they make even more mistakes in all
directions.
None
of this means no child ever should be taken from her or his parents. But foster care is an extremely toxic
intervention that should be used sparingly and in very small doses. Predictive analytics almost guarantees an increase in the dose - that’s why the
biggest champions of predictive analytics often are also the biggest supporters
of throwing more children into foster care.[1]
Algorithms make all this worse in still another way: The very fact that a child was labeled "high risk" to be a victim while growing up, also makes it more likely that she or he will be labeled high risk to abuse her or his own children. Yes, predictive analytics manages to be Orwellian and Kafkaesque. Pittsburgh's scarlet number algorithm and its counterparts can haunt families for generations.
To
understand why predictive analytics does so much harm, one need only look at
what has happened in criminal justice – and what the evidence is
revealing in child welfare itself.
The bias already in the system
As is noted above, the overwhelming majority of cases
that come to the attention of child protective services workers are nothing
like the images that come to mind when we hear the words “child abuse.” On the contrary, typically, they involve
neglect. And neglect often is a euphemism for poverty.
What is “neglect”? In Illinois, it's failure to provide "the proper
or necessary support ... for a child's well-being." In Mississippi, it's
when a child is "without proper care, custody, supervision, or
support." In South Dakota and Colorado, it's when a child's
"environment is injurious to his welfare."
 |
| Three studies have found that 30 percent of foster children could be home right now if their parents just had adeqate housing |
The biggest single problem in American child welfare is the
confusion of poverty with neglect. For reasons discussed
below, predictive analytics worsens the confusion.
The class bias is compounded by racial bias. Obviously, a system that confuses poverty
with neglect will do disproportionate harm to children of color, since they are
more likely to be poor. But study after study has found racial bias over and above the class bias.
This should come as no surprise. After all, racial bias permeates every other
facet of American life. What makes child
welfare unusual is the extent to which the field is, to use one of its own
favorite terms, “in denial” about this basic truth. And, as is discussed in more detail below,
that’s one reason why using predictive analytics in child welfare is likely to
be even more harmful, and even more prone to abuse, than its use in criminal
justice.
Predictive analytics does nothing to counteract these
biases. On the contrary, predictive
analytics makes these biases worse.
The criminal justice experience
 |
| Eric Holder warned of the dangers of predictive analytics |
Although these measures were created with the best of intentions, I am concerned that they inadvertently undermine our efforts to ensure individualized and equal justice. They may exacerbate unwarranted and unjust disparities that are already far too common in our criminal justice system and in our society.
ProPublica found that
Holder was right.
ProPublica looked at
7,000 cases in Broward County, Fla., which uses a secret algorithm created by a
for-profit company to assign risk scores to people arrested in that county,
much as Los Angeles County once planned to use a secret algorithm from a for-profit
company to apply predictive analytics to its child abuse investigations.
According to the
story, when it came to predicting violent crime, the algorithm did a lousy job
in general – four times out of five, people the algorithm said would commit a
violent crime within two years did not.
In addition,
according to the story:
The formula was particularly likely to falsely flag black defendants as future criminals, wrongly labeling them this way at almost twice the rate as white defendants. White defendants were mislabeled as low risk more often than black defendants.
The company that came
up with the algorithm disputes the findings, saying its own analysis of the
data found no racial disparities. But a second analysis of ProPublica's data confirmed the news organization's findings.
Poverty is equated with risk
Since the algorithm
itself is secret, we can’t be sure why the results came out racially biased.
But Prof. Sonja Starr of the University
of Michigan Law School has written that
the factors used to create these sorts of algorithms typically include
“unemployment, marital status, age, education, finances, neighborhood, and
family background, including family members’ criminal history.”
Or as Prof. Starr put it to Bloomberg Technology: “Every
mark of poverty serves as a risk factor.”
Similarly, the algorithm Los Angeles planned to use for
child abuse investigations includes risk factors such as whether the child has
been taken often to an emergency room or whether the child often changes
schools, both factors closely correlated with poverty. Perhaps that helps
explain why, when the Los Angeles model predicted a catastrophe in a family, 95
percent of the time, the prediction was wrong.
Virginia Eubanks estimates that 25% of the variables in Pittsburgh's AFST algorithm are direct measures of poverty. Another 25% measure interaction with the child welfare and juvenile justice systems themselves. Eubanks calls it "poverty profiling."
In a classic example of the disingenuous way Allegheny County has been selling predictive analytics, they falsely claimed to have fixed the poverty profiling issue in their “Hello Baby” algorithm, the one that seeks to stamp a scarlet number risk score on every child at birth.
Thus, they claim: “Unlike the Allegheny Family Screening Tool model, the Hello Baby model only relies on data where the County has the potential to have records for every family; it only uses universal (rather than means tested) data sources.”
But the key weasel word there is potential.
Because right before making this claim, the county acknowledges that they probably will use “child protective services, homeless services and justice system data.”
So unless Allegheny County’s jails are filled with wealthy white-collar corporate criminals, and its homeless shelters are filled with CEOs spending the night because they misplaced the keys to their McMansions, this is still poverty profiling.
There is a similar problem when it
comes to the use of “criminal history.”
As The Marshall Project and
the website FiveThirtyEight explain:
Heavy policing in some neighborhoods … makes low-income and nonwhite residents more likely to be arrested, whether or not they’ve committed more or worse crimes. … Even using convictions is potentially problematic; blacks are more likely than whites to be convicted of marijuana possession, for example, even though they use the drug at rates equivalent to whites.
The same, of course, is true when it
comes to “reports” alleging child abuse – some communities are much more
heavily “policed” by child protective services. Indeed, a better term for such agencies is "family police."
So just as predictive analytics in
criminal justice puts black defendants at greater risk of prolonged sentences,
predictive analytics in child welfare puts black children at greater risk of
being sentenced to needless foster care – with all of the attendant harms noted
earlier in terms of abuse in foster care itself and
other rotten outcomes.
Predictive analytics as computerized racial profiling
The parallels to child welfare don’t end there.
● In criminal justice, the use of
predictive analytics is far outrunning objective evaluation. ProPublica found
that evaluations were rare and often done by the people who developed the
software. ProPublica had to do its own test for racial bias because, it seems,
no one else has bothered. Similarly, Los Angeles initially moved ahead with
predictive analytics in child welfare based on tests and evaluations run by the
software company that wants to sell its product to the county. In Pittsburgh, as we've seen, proponents stacked the deck on evaluations and "ethics reviews."
●Predictive analytics originally was
sold in criminal justice as a benevolent intervention – meant to help agencies
custom tailor rehabilitation and supportive services to the needs of high-risk
defendants and reduce incarceration. But it’s quickly metastasizing into use at
all stages of the criminal justice projects, including, most ominously,
sentencing. Child welfare will be even
less likely to keep the use, and abuse, of predictive analytics under control.
That’s partly because
at least in criminal justice, there is a vibrant community of progressives and
civil libertarians on guard against abuse. But too often, in child welfare, if
you want to get a liberal to renounce everything he claims to believe in about
civil liberties and due process, just whisper the words “child abuse” in his
ear.
This can be seen most
clearly through another comparison to criminal justice.
Predictive analytics: The stop-and-frisk of child welfare
In 2016, The Daily Show did
a superb analysis of “stop-and-frisk” – the policing tactic pioneered in New York
City under former Mayor Rudy Giuliani and struck down by a judge who branded it
“indirect
racial profiling.”
In the clip, available here and embedded below, Trevor Noah goes through the problems
with stop-and-frisk one after the other:
§ The rate of false positives – innocent people stopped and
frisked – is staggering.
§ Though the name suggests a gentle, benign process, the
reality is a deeply frightening, humiliating experience to those who must
undergo it.
§ It is racially biased.
§ Defenders say it’s not biased, it’s based on applying a
series of risk factors said to be associated with criminal behavior.
§ It backfires by sowing so much fear and distrust in poor
communities of color that it undermines law enforcement and compromises safety.
But backers of stop-and-frisk – overwhelmingly white and
middle class – say they know better than people who actually live in
communities of color. Former House Speaker Newt
Gingrich put it this way:
 |
| Too many liberals start to sound like Newt Gingrich when you whisper the words "child abuse" in their ears. (Photo by Gage Skidmore) |
You run into liberals who would rather see people killed than have
the kind of aggressive policing … And a lot of the people whose lives were
saved because of policing in neighborhoods that needed it the most, were
minority Americans.
But what else would you expect from
right-wing Republicans like Gingrich, or Giuliani or Donald
Trump? Liberals would
never tolerate such a harmful, racially biased intrusion on civil liberties.
Or would they?
One self-proclaimed liberal, used exactly the same logic, and almost the same words, as Gingrich to defend the use of predictive analytics in child welfare. Here's what she wrote for what was then known as the Chronicle of Social Change:
As you watch the Daily Show clip, try this: Whenever Trevor Noah says “crime” or “criminal” substitute “child abuse” or “child abuser.” And whenever he says stop-and-frisk, substitute “predictive analytics.”
One self-proclaimed liberal, used exactly the same logic, and almost the same words, as Gingrich to defend the use of predictive analytics in child welfare. Here's what she wrote for what was then known as the Chronicle of Social Change:
As to the concern about racial profiling that has haunted the predictive analytics debate, I find it very hypocritical. Letting more black children die to protect them from racial profiling cannot possibly be an ethical approach or one that is endorsed by responsible child welfare leaders.
The Chronicle persuaded the writer to delete the paragraph, but
it was briefly published by mistake.
As you watch the Daily Show clip, try this: Whenever Trevor Noah says “crime” or “criminal” substitute “child abuse” or “child abuser.” And whenever he says stop-and-frisk, substitute “predictive analytics.”
As with stop-and-frisk, predictive analytics puts a
pseudo-scientific veneer on indirect racial profiling. ProPublica proved it. And as with stop-and-frisk,
predictive analytics leads to an enormous number of false positives,
guaranteeing that many more innocent families will be swept into the system.
If anything the collateral damage of
predictive analytics can be worse than stop-and-frisk. With stop-and-frisk, a
child may see his father thrown up against a wall and roughed up, but at least
when it’s over the child still will have his father.
Yet, as has been the case so many times
before, for many years, the Left failed to mobilize to counter a threat to civil liberties disguised as
fighting child abuse. That is beginning to change. The American Civil Liberties Union has stepped up.
Other reasons the risk is greater in child welfare
The failure of many on the Left to
stand for civil liberties when the issue is child abuse has created a whole
series of other problems. They add up to still more reasons for concern about
adding predictive analytics to the mix:
At least in criminal justice, every accused is
entitled to a lawyer – though not necessarily an effective one. At least in
criminal justice conviction requires proof beyond a reasonable doubt. At least
in criminal justice, the records and the trial are public. At least in criminal
justice, almost everyone now admits that racial bias is a problem, even if they
disagree about how much of a problem.
And at least in criminal justice, the leader of one of the largest and
most important law enforcement organizations, the International Association of
Chiefs of Police, issued
a public apology to communities of color for the actions of police
departments.
In contrast, none of these protections is universal
– and most never apply at all – in cases where the stakes often are higher: cases in
which a child protective services agency decides to throw a child into foster
care.
The right to counsel, and whether hearings are open or
closed, vary from state to state. In every state, child protective services can
hide almost every mistake behind “confidentiality” laws. Homes can be searched
and children can be strip-searched – and seized – without a warrant.
The
standard of proof for a court to rubber-stamp removal of a child typically is only
“preponderance of the evidence,” the same standard used to determine which
insurance company pays for a fender-bender.
And not only has there
been no apology to the Black and Native American communities for the ongoing racial bias
that permeates child welfare, there is an entire coterie in child welfare
insisting that people in the field are so special, so superior to the rest of
us, that racial bias isn’t even an issue.
Indeed, imagine for a
moment the uproar that would follow if a prominent figure in law enforcement,
asked which states were doing the best job curbing crime replied that it’s
complicated “but I will tell you the states that do the best overall are the
ones that have smaller, whiter populations.”
If a police official
said that he might have to resign, and probably would be effectively, and
rightly, exiled from debates over criminal justice.
 |
| Michael Petit |
Petit
said it was a complicated question, “but
I will tell you the states that do the best overall are the ones that have
smaller, whiter populations.”
Not only was Petit not asked to apologize and not
ignored from then on in the child welfare debate, he was named to a national
commission to study child abuse and neglect fatalities, where he was the most
outspoken advocate for taking away more children – and for predictive
analytics.
For all these reasons predictive
analytics probably is even more dangerous in child welfare than in criminal
justice.
Research specific to child welfare
But
we don’t need to draw analogies to criminal justice to see the failure of
predictive analytics. Consider what a
researcher found in New Zealand, a world leader in trying to apply predictive
analytics to child welfare:
“New Zealand Crunches Big Data to Prevent Child Abuse,” declared a headline on a
2015 story about predictive
analytics.
The story quotes Paula Bennett, New
Zealand’s former minister of social development, declaring at a conference: “We
now have a golden opportunity in the social sector to use data analytics to
transform the lives of vulnerable children.”
If implemented, the story enthuses, it
would be a “world first.”
All this apparently was based on two studies that, it now
turns out, used methodology so flawed that it’s depressing to think that things
ever got this far. That’s revealed in a detailed analysis by Professor Emily Keddell of the University of Otago.
The analysis zeros in on a crucial flaw in most such algorithms: They don't actually try to predict if someone will commit an act of child abuse. Rather, they predict only if someone is likely to be involved with the family policing system itself at some point in the future.
The Pittsburgh algorithm predicts the likelihood that a child will be placed in foster care in the next two years. The New Zealand algorithm attempted to predict which children would turn
out to be the subjects of “substantiated” reports of child maltreatment.
Among the children identified by the
software as being at the very highest risk, between 32 and 48 percent were, in
fact, “substantiated” victims of child abuse. But that means more than half to
more than two-thirds were false positives. (And, as we've seen the record in Los Angeles and Illinois was even worse.)
Think about that for a moment. A
computer tells a caseworker that he or she is about to investigate a case in
which the children are at the very highest level of risk. What caseworker
is going to defy the computer and leave these children in their homes, even
though the computer is wrong more than half the time?
But there’s an even bigger problem.
Keddell concludes that “child abuse” is so ill-defined and so subjective, and
caseworker decisions are so subject to bias, that “substantiation” is an
unreliable measure of the predictive power of an algorithm. She writes:
How accurately the substantiation decision represents true incidence is … crucial to the effectiveness of the model. If substantiation is not consistent, or does not represent incidence, then identifying an algorithm to predict it will produce a skewed vision …
Turns out, it is not consistent, it
does not represent incidence, and the vision is skewed. Keddell writes:
Substantiation data as a reflection of incidence have long been criticized by researchers in the child protection field … The primary problem is that many cases go [unreported], while some populations are subject to hypersurveillance, so that even minor incidents of abuse are identified and reported in some groups.
That problem may be compounded, Keddell
says, by racial and class bias, by whether a poor neighborhood is surrounded by
wealthier neighborhoods (substantiation is more likely in such places),
and even the culture in a given family policing agency office.
Predictive analytics becomes self-fulfilling prophecy
Algorithms don’t counter these biases,
they magnify them.
Having a previous report of
maltreatment typically increases the risk score. If it’s “substantiated,” the
risk score is likely to be even higher, though in Pittsburgh it raises the risk score by the same amount either way. So then, when another report comes in,
the caseworker, not about to overrule the computer, substantiates it again,
making this family an even higher “risk” the next time. At that point, it
doesn’t take a computer to tell you the children are almost certainly headed to
foster care.
Writes Keddell:
“prior substantiation may also make practitioners more risk averse, as it is likely to heighten perceptions of future risk to the child, as well as of the practitioner’s own liability, and lead to a substantiation decision being made.”
So predictive analytics becomes a
self-fulfilling prophecy.
We also know that workers are more likely to investigate a case and more likely to substantiate an allegation if the family is Black or Native American. So if the algorithm increases the risk score based on prior reports and substantiations, it bakes in racial bias.
The problem is compounded by the fact that though race is not explicitly listed as a risk factor, multiple risk factors are correlated to poverty. So parenting-while-poor inherently raises the risk score. Virginia Eubanks refers to the Pittsburgh algorithm as "poverty profiling."
The focus on past reports builds in bias in other ways. A lawsuit alleges that the University of Pittsburgh Medical Center often drug tests pregnant women and their newborns without consent, then reports even false positive results for marijuana to the county family police agency - which automatically inflicts a child abuse investigation. Ironically, if the lawsuit is correct, the algorithm is bypassed. But the existence of that previous report, no matter how absurd, automatically raises the risk score for this family if anyone reports them for anything else ever again.
And there's still another problem: The Pittsburgh algorithm is so fixated on a family's past - using factors heavily correlated to poverty -- that it does not consider the actual allegation when deciding if a child is at high risk! As the Associated Press story explains, that means a very serious allegation against a wealthy family might lead to a risk score that’s too low, even as a minor allegation against a poor family leads to a risk score that’s too high.
But it turns out there may be one area
where predictive analytics can be helpful. Keddell cites two studies in which
variations on analytics were used to detect caseworker bias. In one, the
researchers could predict which workers were more likely to recommend removing
children based on questionnaires assessing the caseworkers’ personal values.
In another, the decisions could be predicted by which income
level was described in hypothetical scenarios. A study using similar methodology uncovered racial bias.
So how about channeling all that energy
now going into new ways to data-nuke the poor into something much more useful:
algorithms to detect the racial and class biases among child welfare staff?
Then we could teach those staff to recognize and overcome those biases, and protect
children from biased decisions by “high risk” caseworkers.
The reality on the ground
To
see how the brave new world of predictive analytics likely would play out in an
actual American case, let’s take a trip into
the very near future and consider a hypothetical case.
Child Protective Services has just
received a child maltreatment report concerning a father of five. With a few
keystrokes, CPS workers find out the following about him:
He’s married, but the family lives in
deep poverty. He has a criminal record, a misdemeanor conviction. He and his
wife also had the children taken away from them; they were returned after six
months.
These data immediately are entered into
a computer programmed with the latest predictive analytics software. And
quicker than you can say “danger, Will Robinson!” the computer warns CPS that this
guy is high-risk.
When the caseworker gets to the home,
she knows the risk score is high, so if she leaves those children at home and
something goes wrong, she’ll have even more than usual to answer for.
That means the allegations in the actual report – and
whether or not those allegations are true – barely matter. In this new, modern
age of “pre-crime” making determinations based on what actually may have
happened is passe. Instead we make
decisions based on what the computer says might happen. So those children are likely to be taken
away, again.
So, now let’s return to the present and
meet the family at the center of the actual case, from
Houston, Texas, on which this hypothetical is based. Unfortunately the original broadcast story no longer is online.
The case involved no
accusation of abuse. The children were
not beaten, or tortured. They were not left home alone. They were not locked in a closet. They were not starved. The children were taken because their father
was panhandling while they were nearby – with their mother. Period.
In the hypothetical, I changed two
things about this story. First, the story mentions no criminal charges, and, in
fact, panhandling is illegal only in some parts of Houston. But predictive
analytics tends not to factor in Anatole France’s famous observation that “the
law, in its majestic equality, forbids rich and poor alike to sleep under
bridges, to beg in the streets, and to steal bread.”
So had there been a criminal
conviction, or even a charge, regardless of the circumstances, it almost
certainly would have added to the risk score.
But even without adding a criminal conviction to the hypothetical, we’re
still talking about a family which not only had a previous report of child
neglect “substantiated” but already had the children taken away.
And second, I’m assuming the father, Anthony
Dennison, and his wife actually will get
their children back. In fact, there’s no telling what will happen.
What we do know is that if this case occurred in, say Pittsburgh instead of Houston if Dennison’s children ever were returned,
and if Dennison ever was reported again, the children would be likely to be removed
again. And, what with it then being the second time and all, they’d be more likely to stay removed forever.
Then, when they grew up, if they ever were accused of abuse or neglect, their history would raise the risk score for them and make it more likely they would lose their children.
Now, lets compare this with another actual case - involving an incident in Pittsburgh. The allegations are as follows:
A man storms into an office in downtown Pittsburgh. His 12-year-old daughter is with him. The man appears intoxicated. The man screams and yells and pounds his fists against a bulletin board. He demands to have his picture taken.
He forcefully grabs his daughter’s forearm, pulling her into the picture as she tries her best to pull away from him. She screams “Please, please Daddy, no!” multiple times. And multiple times he yanks on her arm, trying to pull her to his side so a photo could be taken of both of them. He yells at his daughter and repeatedly jabs his finger in her shoulder.
The daughter is crying hysterically and red-faced. The father rips a cell phone out of her hand because he thought she was trying to call her mother.
As one eyewitness said:
I was extremely concerned for his daughter‘s safety, and I actually noticed that my heart was racing. … Having to watch as [the father] terrorized his teenage daughter — with his hair disheveled and his face twisted — was something I’m never going to forget.
The father denies all of this – but remember, an allegation is all it takes to put someone through the AFST database and cough up a risk score.
We don’t know if anyone ever called the Allegheny County family police agency. Indeed, we don’t know if the father actually lives in Allegheny County or even Pennsylvania. But if the answer to those questions is yes, and if the father’s name were run through AFST, odds are his risk score would be very low. That’s because the father is John Robinson Block. The office in question is that of the Pittsburgh Post-Gazette – which Block publishes. That makes him way too affluent for much to turn up in the AFST database. And, as we’ve seen, AFST doesn’t factor in the seriousness of the allegation.
(The account on the incident comes from the Post-Gazette chapter of the News Guild. As noted above, Block denies the allegations.)
The child welfare response: We can control our nukes
Although the strongest pressure for predictive
analytics comes from those who also want to see far more children taken away, occasionally, reformers running child welfare systems also want to use it.
Their argument boils down to this: We understand the enormous power of Big
Data, and you know we don’t want to
take away lots more kids. So you can
trust us. We’ll only use our enormous
power for good!
There are a number of variations on
the theme, the most prominent being: We’ll only use the data to target help to
families, we won’t use it to decide whether to remove the children. And then
there are those who paraphrase the National Rifle
Association: Computers don’t decide whether to take away children, they say,
people do! Human beings are free to
override any conclusions reached by an algorithm. In Pittsburgh, the county family policing agency chants that one like a mantra.
But that’s not how it’s likely working in real life.
For starters, let’s return to that
case in Houston cited above:
When child protective services in
Houston encountered the Dennison family, they did not offer emergency cash
assistance. They did not offer assistance to Dennison to find another job, or
train for a new one.
They took the children and ran. Just as Houston CPS did in another case, where they rushed to confuse poverty with neglect.
An algorithm won’t make these decisions
any better. They’ll just make it easier to take the child and run.
But, say some reformers, we’re more
enlightened than the people in Houston, we wouldn’t misuse analytics in a case
like this. Maybe not. But child welfare agency chiefs don’t tend to stay on
those jobs for very long. To those reformers I would respond: What about your
successor? And her or his
successor?
And, as noted earlier, what caseworker
will leave a child in a home rated by a computer as high risk, knowing that if
something goes wrong, she’ll be all over the local media as the caseworker who
ignored “science” and “allowed” a tragedy to occur? Of course she won’t leave the children in
that home. So children will be taken in
many, many cases where the algorithm got it wrong and produced a “false
positive.”
 |
| Big Data is like nuclear power at best, nucelar weapons at worst |
Few in child welfare are truly
malevolent. But there are lots and lots
of typically fallible human beings. And
nothing in the history of child welfare indicates that it can responsibly
handle the nuclear weapon of Big Data.
That’s why predictive analytics actually amounts to data-nuking poor
families.
Efforts to abuse analytics already are underway
There’s no need to speculate about
whether America’s child welfare bureaucrats would misuse predictive analytics –
some already are trying.
Misusing analytics was, in fact, the
primary recommendation of a group that was known as the Commission to Eliminate
Child Abuse and Neglect Fatalities (CECANF). The
commission was the brainchild of Michael Petit – the same Michael Petit whose
appalling comments on race are noted above.
The recommendation to misuse analytics also was Petit’s idea.
As
I noted in the trade journal Youth Today, the commission, created by an act of Congress, was
chaotic, angry, dysfunctional and secretive. It made decisions based on
newspaper horror stories. One of only
two Black commissioners was treated with appalling disrespect as the
Commission rushed to complete deliberations.
(That commissioner, Patricia Martin, presiding judge of the Child
Protection Division of the Circuit Court of Cook County, Illinois, wrote a scathing dissent.) In other words,
the Commission to Eliminate Child Abuse and Neglect Fatalities didn’t study the "child welfare" system, it recreated the
child welfare system.
The Commission used predictive
analytics to justify what some commissioners called a “surge,” in which states
would use “multi-disciplinary teams” to reopen huge numbers of cases – but only
cases in which children were allowed to remain in their own homes. There would be no checking on children in
foster care to see if they really need to be there. (Other commissioners, with
no sense of irony, referred to this idea as an “accelerant.” Both terms were
dropped after commission PR staff was instructed to find something more
palatable.)
The surge/accelerant calls for
demanding that states look at every child abuse death for the past five years,
try to find common risk factors, and then reopen any case where even one such
risk factor may be present.
The flaw in the logic is one for which
we all should be grateful. Though each is among the worst imaginable
tragedies, the chances of any parent killing a child are
infinitesimal. The chances of a parent with a given “risk factor” killing
a child are ever-so-slightly less infinitesimal.
It should not have been necessary for
the federal Department of Health and Human Services to point out something so obvious,
but, since they were mandated by Congress to respond to the Commission’s
report, they did so. Here’s
what they said:
States frequently have significant year-to-year swings in the number and rate of fatalities. In small states, a single incident rather than a systemic issue can dramatically affect annual statistics. In addition, in small states an analysis of data from the past five years…would include too few cases to draw definitive conclusions.
The surge isn’t the only area where the
Commission misused analytics. They also
used it as the basis for a recommendation that would prohibit child protective
hotlines from screening out any call involving a child under age 3, and another
one to bar screening out any case in which someone calls more than once.
The hotline recommendations alone, if
implemented, probably would increase the number of cases investigated every year
by nearly 40 percent. (For NCCPR’s full analysis of the Commission report, see our full rebuttal here.)
The Commission recommendations add up
to a regime of domestic spying that would make the NSA blush. And even if
you’re not fazed by the enormous harm inflicted on children by needless
investigations and needless foster care, consider what a 40 percent increase in
caseloads would do to the quality of investigations. Workers would have
far less time for any case – and more children in real danger would be missed.
All because the commission fell in love
with the concept of predictive analytics.
These posts to the NCCPR Child Welfare
Blog describe in detail the
chaos and
incompetence that characterized
the deliberations of what turned out to be the Keystone Kops of commissions, not
to mention the time and money they wasted writing blog posts that could be turned
into Mad Libs). And these are
supposedly national leaders in the field. Do we really want to give people like
this, running secret systems with no real due process protections, and no real
accountability the nuclear weapon of predictive analytics?
 |
| CECANF was the Keystone Kops of commissions |
Lessons from the elections of 2016
In 2016, predictive analytics told us
Hillary Clinton would be president.
Predictive analytics told us the Democrats would take control of the
Senate. And The New York Times says there are lessons from that – lessons that
go far beyond election forecasting.
According to the Times:
It was a rough night for number crunchers. And for the faith that people in every field … have increasingly placed in the power of data. [Emphasis added]
[The election results undercut] the belief that analyzing reams of data can accurately predict events. Voters demonstrated how much predictive analytics, and election forecasting in particular, remains a young science …
[D]ata science is a technology advance with trade-offs. It can see things as never before, but also can be a blunt instrument, missing context and nuance. … But only occasionally — as with Tuesday’s election results — do consumers get a glimpse of how these formulas work and the extent to which they can go wrong. … The danger, data experts say, lies in trusting the data analysis too much without grasping its limitations and the potentially flawed assumptions of the people who build predictive models.
As
we’ve seen, flawed assumptions, built into the models, were the root of the
rampant racial bias and epidemic of false positives that ProPublica found when
analytics is used in criminal justice. And as we’ve seen, Prof. Emily Keddell and researchers in Pittsburgh found much the same when they examined predictive analytics in child welfare.
The Times story also includes a lesson for
those who insist they can control how analytics is used – those who say they’ll
only use it to target prevention – not to decide when to tear apart families:
Two years ago, the Samaritans, a suicide-prevention group in Britain, developed a free app to notify people whenever someone they followed on Twitter posted potentially suicidal phrases like “hate myself” or “tired of being alone.” The group quickly removed the app after complaints from people who warned that it could be misused to harass users at their most vulnerable moments.
Conclusion – Big Data is watching you
If you really want to see the world
envisioned by proponents of predictive analytics, forget the bland reassurances
of proponents in child welfare. Consider how those pushing
the product market it to other businesses. (I used to have a link to the commercial described below, but the company took it off YouTube).
It showed a bunch of data analysts, presumably working for a firm that sells
sporting goods, spying on a woman’s recreational habits. They have amassed
so much data and their algorithms are so wonderful that it’s like having a
camera watching her 24/7. Not only do they know her preferences, they know
exactly why she prefers one sport over another and exactly what she’ll do next.
In other words, they’re stalking her.
But this is not presented as a warning of the dangers of
predictive analytics. On the contrary, virtual stalking is what they’re
selling.
That’s because the commercial is not aimed at consumers –
such as the woman being stalked. The target audience is potential stalkers; in
this case people who want to sell her stuff.
The maker of the stalking – er, analytics – software, and
maker of the commercial, is SAP – described in one story as one
of the “market leaders” in predictive analytics and a potential competitor in the
child welfare market.
Unlike the bland reassurances given when people raise
concerns about predictive
analytics, the commercial reveals the real mindset of some of the human beings
pushing Big Data.
Apparently, no one at SAP was creeped out by the ad’s
Orwellian overtones. The slogan might as well have been “Big Data Is Watching
You.” That alone ought to be enough to make anyone think twice about turning
these companies loose in the child welfare field.
And it ought to make anyone think twice about giving this
kind of power to secretive, unaccountable child welfare bureaucracies that have
almost unlimited power to take away people’s children.
[1] In addition to
Michael Petit, and Emily Putnam-Hornstein, whose roles in the debate are discussed in the text, big boosters
of predictive analytics included the late Richard Gelles, a strong proponent of taking
away more children and expanding the use of orphanages, Elizabeth Bartholet,
whose own proposals for changing the system would require the removal of
millions more children from their homes, and Naomi Schaefer Riley, a far right extrremist who proudly analogizes her work to that of Charles Murray.
