Aug 29, 2018

Schools turn to AI to monitor students' mental health

Sixth graders at Flint Hill School in Oakton, Va. Photo: Katherine Frey/Washington Post/Getty

Jolted by rising suicide and self-injury among young Americans, schools are using software to monitor student browsing history and logs for signs of distress, in what they hope will curb the problem.

What's going on: Kids and teens increasingly rely on the internet to answer their mental-health questions, creating browsing patterns that schools hope could identify potential harm before it happens. But that requires sweeping online surveillance that critics say could leave a lasting mark on students.

The context: Between 2006 and 2016, the suicide rate for white children rose 70%, to more than five per 100,000 children, according to the CDC. The rate for black children increased 77%, to 2.62 suicides per 100,000 children.

The details: Many monitoring tools are on the market, including one by GoGuardian, a Los Angeles-based education-tech company.

  • GoGuardian's early filtering products were aimed at keeping students away from online porn, Tyler Shaddix, the company's chief product officer, tells Axios.
  • After a student committed suicide at a school testing the software, an administrator asked whether better monitoring could have prevented it.
  • That led to a new system that evaluated students’ online activity for signs they may harm themselves.

At first, GoGuardian used keyword matching, a standard approach among companies like it. This generated thousands of extraneous alerts, so in its latest version — due out in October — GoGuardian has added artificial intelligence techniques that it says improves the software's accuracy.

How it works: GoGuardian installs an extension on school-issued computers running Chrome OS or Windows. When the system detects a potential problem in chat conversations, emails, Google Docs, or browsing, it sends an alert to whomever the school has designated to receive them.

  • With the help of mental-health experts, the company trained a machine-learning algorithm so it flags content most closely associated with potential harm, like searches for suicide methods.
  • The system now sends a much more manageable 2,000 alerts a week across 5 million students, within seconds of being triggered.
  • A human makes the final call, reading over the alert — including five minutes of browsing history before and after the incident — to decide whether or not to act.

The software can stumble either by sending sending schools unneeded messages, drowning out real flags, or by missing important signs and failing to send a critical alert.

  • GoGuardian CEO Advait Shinde says the system errs on the side of alerting more often, but the company refused to share stats about its software's failure rate.
  • Gennie Gebhart, a researcher at EFF and co-author of a 2017 report on student privacy, said companies that help monitor students are not transparent enough about the accuracy of their software.

Electronic snooping always raises weighty ethical questions, even when it’s meant to safeguard young people.

  • Privacy experts worry about early, all-encompassing monitoring. Gebhart said pervasive surveillance normalizes electronic snooping, and can keep kids from testing out new ideas and identities as they grow.
  • "This piece of software is going beyond ed-tech and toward a new kind of surveillance in the classroom," Gebhart said.
  • When I asked GoGuardian if students have a right to privacy, Shaddix said the benefits of monitoring "outweigh the negatives."

Go deeper: Wired reports on schools that monitor students’ social media feeds for threatening posts or cyberbullying.

Go deeper

International coronavirus treatment trial uses AI to speed results

Hydroxychloroquine is one of the drugs that will be included in the trial. Photo: John Philips/Getty Images

The first hospital network in the U.S. has joined an international clinical trial using artificial intelligence to help determine which treatments for patients with the novel coronavirus are most effective on an on-going basis.

Why it matters: In the midst of a pandemic, scientists face dueling needs: to find treatments quickly and to ensure they are safe and effective. By using this new type of adaptive platform, doctors hope to collect clinical data that will help more quickly determine what actually works.

We can't just flip the switch on the coronavirus

Illustration: Sarah Grillo/Axios

It feels like some big, terrible switch got flipped when the coronavirus upended our lives — so it’s natural to want to simply flip it back. But that is not how the return to normalcy will go.

The big picture: Even as the number of illnesses and deaths in the U.S. start to fall, and we start to think about leaving the house again, the way forward will likely be slow and uneven. This may feel like it all happened suddenly, but it won't end that way.

Go deeperArrow18 mins ago - Health

The Fed rescues Wall Street, but Main Street is another story

llustration: Aïda Amer/Axios

In less than a month, the Federal Reserve has unleashed a multi-trillion dollar tour de force to buoy the U.S. economy against the COVID-19 pandemic.

Why it matters: While it has steadied the markets, the Fed is poorly equipped to offset the hit being absorbed by small business owners and the close to 17 million Americans who have filed for unemployment in just the past three weeks.