Happy with a 20% Chance of Sadness
In the winter of 1994, a young man in his early twenties named Tim was a patient in a London psychiatric hospital. Despite a happy and energetic demeanour, Tim had bipolar disorder and had recently attempted suicide. During his stay, he became close with a visiting US undergraduate psychology student called Matt. The two quickly bonded over their love of early-nineties hip-hop and, just before being discharged, Tim surprised his friend with a portrait that he had painted of him. Matt was deeply touched. But after returning to the United States with portrait in hand, he learned that Tim had ended his life by jumping off a bridge.
Matthew Nock now studies the psychology of self-harm at Harvard University in Cambridge, Massachusetts. Even though more than two decades have passed since his time with Tim, the portrait still hangs in his office as a constant reminder of the need to develop a way to predict when people are likely to try and kill themselves. There are plenty of known risk factors for suicide—heavy alcohol use, depression and being male among them—but none serve as tell-tale signs of imminent suicidal thoughts. Nock thinks that he is getting close to solving that.
Since January 2016, he has been using wristbands and a phone application to study the behaviour of consenting patients who are at risk of suicide, at Massachusetts General Hospital in Boston. And he has been running a similar trial at the nearby Franciscan Children’s Hospital this year. So far, he says, although his results have not yet been published, the technology seems able to predict a day in advance, and with reasonable accuracy, when participants will report thinking of killing themselves.
Nock’s trial is one effort to make use of the burgeoning science of mood forecasting: the idea that by continuously recording data from wearable sensors and mobile phones, it will be possible not only to track and perhaps identify signs of mental illness in a person, but even to predict when their well-being is about to dip. Nock collaborates with Rosalind Picard, an electrical engineer and computer scientist at the Massachusetts Institute of Technology (MIT) in Cambridge. Picard leads a team that has tracked hundreds of undergraduates in universities in New England with phones and wristbands, and reports being able to predict episodes of sadness in these students a day before symptoms arrive.
Hints that it might be possible to track impending emotional vulnerability have sparked strong commercial interest. Mindstrong Health, a company in Palo Alto, California, which has raised US$29 million in venture capital, tracks how people tap, type and scroll on their phones, to spot shifts in neurocognitive function. Paul Dagum, a physician and computer scientist who founded the firm, says that data from a person’s touchscreen interactions can identify oncoming episodes of depression, although that work has not yet been published. Other companies are also researching the use of such ‘digital phenotyping’ to recognize symptoms of mental illness. Among them is Verily, a life-sciences firm owned by Google’s parent company, Alphabet.
At this stage, the reliability of mood-prediction technology is unclear. Few results have been published, and groups that have released results say they have achieved only moderate rather than outstanding accuracy when it comes to forecasting moods. Picard, however, is confident that the concept will hold up. “I suffered from depression early in my career and I do not want to go back there,” she says. “I am certain that by tracking my behaviours with my phone I can make it far less likely I will return to that terrible place.”
But researchers including Picard have reservations about possible downsides of their creations. They worry that scientists and clinicians haven’t thought enough about how to inform users of an imminent emotional downturn. There are also questions about whether such warnings could cause harm. And some wonder whether corporations or insurance companies might use the technology to track the future mental health of their employees or customers. “The [potential for] misuse of this technology is what keeps me up at night,” Dagum says.
Predicting depression
Picard got into mood-prediction research indirectly. A decade ago, she showed that it was possible to use wristbands to detect seizures, sometimes minutes before spasms shook the body, by tracking the electrical conductance on a person’s skin. In 2013, she co-founded Empatica, a company in Cambridge that sells sensors, including a smartwatch approved by the US Food and Drug Administration to monitor signs of seizures and issue alerts to caregivers.
Working with her PhD student at the time Akane Sano, now at Rice University in Houston, Texas, Picard saw potential for wider applications. They hypothesized that it might be possible to combine data from wrist sensors and mobile phones to monitor stress, sleep, activity and social interactions to predict general mental health and well-being.
Sano and Picard collaborated with a team at Harvard Medical School to design a study that would track university students on a daily basis. Since 2013, the team has studied 300 students—50 each semester, for 30 days at a time—by giving them watch-like devices to wear. The instruments measure the students’ movements, note the amount of light they are exposed to, monitor their body temperature and record the electrical conductance of their skin. Sano and Picard also developed software, installed on participants’ phones, which records data about their calls, text messages, location, Internet use, ‘screen on’ timing and social interactions. The team also recorded much of their e-mail activity. Students filled out surveys twice a day about their academic, extracurricular and exercise activities. They described their sleep quality, their mood, health, stress levels, social interactions and how many caffeinated and alcoholic drinks they were consuming. The students also reported their exam scores and filled out extensive surveys at the beginning and end of the 30-day studies.
By 2017, the team had reported, training an algorithm to learn from these surveys and to weight the importance of hundreds of measurements. The system can accurately forecast, a day in advance, the students’ happiness, calmness and health, Picard’s team says. In the experiment, individuals had to be monitored for 7 days to reach forecast-accuracy levels of around 80%. Picard’s analysis suggests that wristbands and mobile phones are not able to predict slight changes in mood. But when changes in well-being are large, predictions are more reliable. Some of the signals make intuitive sense—moving around before bed might suggest agitation, for instance—but the details are not always understood. As an example, social interactions might modify stress levels, which can be reflected in skin electrical conductance, but it’s unclear whether many peaks of skin conductance in a day is good or bad, because it increases both when people are problem solving and when they are stressed.
Simply interpreting someone’s mood using such signals is a great achievement, says computer scientist Louis-Philippe Morency at Carnegie Mellon University in Pittsburgh, Pennsylvania, who thinks artificial-intelligence technology could help with mental-health assessments. But he is cautious about its ability to forecast moods. “Since tomorrow’s mood is often similar to today’s mood, we need more research to be able to clearly decouple these two phenomena. It is possible that current forecasting technologies are mostly predicting spillover emotion from one day to the next,” he says.
Picard thinks improvements will come: “We are the pioneers saying that this is truly possible and are showing data to back this claim up. Reliability will grow and grow with more data.” She has made her algorithms open-source, so that others with access to the technology can try to reproduce her work.
“Picard is on to something, and her track record of transparency with her algorithms, models and data sets makes me even more confident of that. People don’t make it so easy to recreate their work when they are unsure about their results,” says Jonathan Gratch, a psychologist at the Institute for Creative Technologies at the University of Southern California in Playa Vista.
Nock’s trial on suicidal thoughts grew out of a collaboration with Picard. So far, he has monitored 192 people, mainly using wristbands and by asking them how they are feeling, through a phone app or interview. For now, he has trained devices not on an individual’s data, but on those of the entire group of participants, and he says that he has identified a few measurable signs that can predict later suicidal thoughts with an accuracy of 75%. Some of the most important factors, he says, are considerable movement in the evening, perhaps denoting restlessness or agitation at night, mixed with spikes in skin electrical conductance and an elevated heart rate. But he declined to give more details because his paper is under review at a journal.
Moving to market
Commercial firms are less willing than are academics to discuss their results. But in March, Mindstrong, which is only 16 months old, reported finding digital biomarkers — patterns of swipes and taps on a phone—that correlate with scores on neuropsychological performance tests3. On its website, the firm says it has completed five clinical trials, the results of which have not been disclosed, and in February, it announced a partnership with Tokyo-based Takeda Pharmaceuticals to explore the development of digital biomarkers for conditions such as schizophrenia and treatment-resistant depression. It has competition: Verily says its digital phenotyping projects include one designed to detect post-traumatic stress disorder using smartphones and watches.
Mindstrong says it’s moving beyond measuring brain function with smartphones, to predicting it. “When we take in the trajectory of numerous biomarkers over the course of six or seven days, we can predict episodes of depression up to a week in the future,” says Dagum—although he declined to say which signals his firm is using, because the company was submitting papers on its work to journals.
The plan for Mindstrong’s phone-based app (the company is not using wristbands) is to embed its touchscreen-interaction measures into a digital mental-health-care system. It has been sharing results with the state of California, which sees enough clinical potential to have granted the firm $10 million over 3 years from a state-managed, $60-million mental-health innovation fund. “Will all of this data that we are collecting ultimately have clinical utility? We don’t know yet,” says psychiatrist Tom Insel, who co-founded Mindstrong and had previously started the mental-health unit at Verily after a 13-year stint as head of the US National Institute of Mental Health.
Picard questions Insel’s approach at Mindstrong. “I believe he has made a company with an idea that is not proven to work as well as other ideas,” she says. Neither she nor Nock yet have commercial plans for their mood-prediction technology. (Besides Empatica, however, Picard has co-founded Affectiva, a firm in Boston that sells technology to analyse facial and vocal expressions.)
Insel says the technology needs testing in real-world settings, with patients and health providers. “We are not running before walking. California is paying us to learn how to walk,” he says. He adds that he doesn’t view Picard as a rival. “This is a hard problem that no one has solved. My best guess is that it will take all of us using many approaches to prove the clinical value of this technology—and, frankly, I’d love to have at least ten other groups of Roz’s lab’s calibre working on digital phenotyping,” he says.
Changing behaviour
Picard is confident that mood forecasting—even if it requires individualized training from a consenting user—will become a perfected art. The real question, she says, is whether it can be used to help change a forecasted dark mood.
Nock and psychologist Evan Kleiman, also at Harvard University, are working with 150 patients to encourage them to reappraise things that they are viewing negatively by using cognitive reframing exercises. These exercises are activated on the patients’ phones when their wrist monitors detect signals that predict upcoming suicidal thoughts. Beyond this, Nock is unclear what to do with the data. “If we have someone who is predicted to be at high risk for suicidal thoughts, or who notes that they are 100% likely to kill themselves, what do we do? Do we send an ambulance? Contact their doctor? Do nothing?” he wonders. “The ethics of this are extremely challenging.” Nock says he knows that those in his trial want the technology. “Patients say all the time how useful they would find an alert or guidance system,” he says.
Morency thinks that it is too soon for computers to be giving mental-health advice on their own. His research involves teaching computers to study facial expressions and language so that they can work out what is on a person’s mind, and he is now collaborating with psychiatrists to install this technology in hospital mental-health wards. The goal is for machines to study people during their interactions with doctors, to discern whether psychiatric disorders are present. The physicians still do the diagnosis; the computer analysis provides a separate assessment that doctors can compare with their own. “The risks presented by a computer giving mental-health advice are significant. We need more research to understand the long-term impact of such technology,” Morency says.
Another issue, says Picard, is that actions to improve mood are different for different people. In one of her experiments, Picard found that one cluster of students who had conversations with friends before going to sleep enjoyed brighter moods the following day, whereas another cluster experienced the inverse effect.
Barbara Fredrickson, a psychologist at the University of North Carolina at Chapel Hill, is concerned that the act of predicting a mood could affect how people feel. “It seems likely that people will give negative mood forecasts a great deal of attention, and for some, this could start an emotional negativity tailspin that could be truly damaging,” she says.
Justin Baker, a researcher in mental illnesses who is the scientific director of the McLean Institute for Technology in Psychiatry in Belmont, Massachusetts, says: “I think it will be just as difficult for us to determine what advice a person needs as it will be to determine how to present that advice to them in a manner that does not get ignored or make them worse.”
Picard has grand visions for digital mood forecasting. She thinks it could improve the health of the general public, and in particular that it might benefit corporations. “Why do so many amazing companies that give their employees every perk under the sun still lose so many staff to depression? Can we catch the coming transition before it takes place?” she says. But she also worries that the technology might be misused. Picard thinks that new regulations might be needed to prevent, say, corporations from targeting advertising at those whose bad or good moods can be seen coming, or to keep insurance companies from setting prices based on signs of their customers’ mental health.
“A few bad actors who misuse this technology could spoil the benefits for patients with serious mental-health issues,” says Insel. Mindstrong, he says, is working with a bioethics group at Stanford University, California and plans to publish a paper on these matters shortly.
Picard argues the research efforts are worthwhile. “Clinical depression is often emotional death by a thousand cuts,” she says. “If we can help to identify the many little things that weigh us down over time and drive us into a perpetual sorrowful state, we can make a big difference.”
This article is reproduced with permission and was first published on October 30, 2018.