Abraham de Moivre, a French mathematician and the godfather of probability theory, was also the first-known person to correctly predict the day he would die. At the age of 87, he noticed that he was sleeping 15 minutes longer each night. He theorized that when those extra 15 minutes per day added up to a full 24 hours, he would die. The date he predicted: Nov. 27, 1754. Sure enough, he passed away from “somnolence” that day.
Though there is some doubt about the veracity of this story, many researchers have since tried to use statistics to tell us how long we will live. More than 250 years later, however, the science of predicting mortality has remained stagnant, left to insurance actuaries using antiquated statistical techniques based on limited data.
But the advent of Big Data analytics has reraised the questions that de Moivre considered: Can we use mathematics to predict the timing of death? Do people want to know when they will die? Recent insights using computer analytics say yes to both.
Predicting one’s mortality is an important question for many stakeholders. As a physician who studies end-of-life care, I have come across cases for which an accurate estimate of one’s longevity would dramatically improve patients’ lives. I recently cared for a frail woman whose knee arthritis rendered her unable to walk. “Doctor, you’ve gotta help me with this pain,” she regularly told me. I tried to justify to her orthopedic surgeon why a knee replacement surgery could improve her quality of life. But with advanced hepatitis C and heart failure, the patient was far from an ideal surgical candidate. The surgeon could justify the elective surgery only if the patient would live long enough to benefit from it. “How long do you think she has to live?” the surgeon asked. I responded honestly: “I don’t know.” Without an accurate prognosis, her surgeon was left to assume the worst. She still hasn’t received her surgery.
Doctors, even those with an intimate knowledge of their patients and diseases, are notoriously bad at prognosis. In a 2000 study published in the BMJ by Nicholas Christakis, a Yale physician and sociologist, more than 300 physicians were asked to estimate the survival of terminally ill patients they had referred to hospice. Doctors, on average, overestimated survival more than fivefold. In his book Death Foretold, Christakis calls this “the ritualization of optimism,” arguing that the difficulty of prognosis causes most doctors to shirk difficult conversations around end-of-life planning. Instead, he says, they opt to hope for the best. This well-intentioned neglect leaves patients struggling for answers.
Clearly, a doctor’s intuition alone predicts death poorly. And even traditional statistics, such as logistic regression, cannot handle the enormous number of variables that play a role in predicting when someone will die. Enter advanced computer algorithms, which can mine countless amounts of data and account for thousands of variables—and theoretically do a much better job predicting outcomes.
Experts are already applying advanced analytics to other areas of health care. Ruben Amarasingham, physician and former CEO of the Parkland Center for Clinical Innovation in Dallas, is one of the world’s experts in studying how to apply predictive modeling in routine clinical care. Amarasingham’s algorithms estimate important outcomes like a patient’s real-time risk of being readmitted to the hospital. Using these predictive algorithms, Amarasingham targeted scarce care-planning resources to high-risk patients, lowering their odds of readmission by 27 percent over one year and saving Parkland Hospital more than $500,000.
Machine-learning algorithms, an example of advanced analytics, learn from large amounts of data to make predictions. In other words, there is no need to manually code these algorithms—they are programmed to learn from prior data and code themselves. Machine-learning algorithms can curate, analyze, and learn from vast arrays of data, raising the potential for better prediction. Machine learning plays a role in dozens of more mundane activities, from suggesting Amazon purchases and Netflix movie choices to projecting the optimal lineup for your favorite baseball team. Researchers even used it to predict correctly the next death on Game of Thrones.
Spurred by such successes, doctors and researchers are now applying machine learning to predict an individual’s risk and timing of death. At the recent the Coalition to Transform Advanced Care annual summit, the most interesting insight came from Ziad Obermeyer, an emergency physician who co-leads the Harvard Laboratory for Systems Medicine at Brigham and Women’s Hospital. He said: “When people are forced to have end-of-life conversations in the emergency room, it’s too late. We need—and now have—better ways to predict mortality using one’s electronic data.”
Obermeyer uses machine learning to predict death in patients with life-threatening diseases like cancer. Unpublished evidence suggests that his algorithms can accurately identify a large group of patients who are virtually certain to die within a year. Algorithms like Obermeyer’s outperform many widely used prognostic risk tools: A 2013 study published in the American Journal of Epidemiology found that machine learning outperformed any single algorithm or risk score by up to 44 percent when predicting mortality in an elderly population.
Researchers in the United Kingdom are betting on the promise of advanced analytics to predict death, undertaking a mammoth effort, funded by a $1.1 million grant from the Institute and Faculty of Actuaries, to apply analytics to health care, insurance, and socioeconomic data. Led by Elena Kulinskaya, professor in statistics at the University of East Anglia, the project will bring together medical staff, data analysts, and computer scientists. According to Kulinskaya, her team will use individual-level health data and advanced analytics to “establish the drivers of increasing longevity, and to predict how they may change over time and how this would affect life expectancy.”
Many skeptics argue that even older algorithms would perform well today with prediction just because of the tremendous amount of electronic data now available. Amarasingham’s algorithms, for example, have been able to predict clinical outcomes so well because they account for socioeconomic variables, such as access to transportation or nutritious food, which have a tremendous impact on health.
But as Obermeyer writes in a recent piece in the New England Journal of Medicine, “To be useful, data must be analyzed, interpreted, and acted on. Thus, it is algorithms—not data sets—that will prove transformative.” Machine learning succeeds in handling thousands, even hundreds of thousands, of predictor variables and combining them in nonlinear ways to predict death. Additionally, traditional risk scores predict an average risk of mortality for a similar group of patients. This allows doctors and patients to make the counterargument, “Sure, but that number doesn’t apply to me.” But machine-learning algorithms—by virtue of the huge amount of data they account for—can predict an individual’s risk of death.
As machines become better than physicians at prognosis, some worry that we may lose the humanity inherent in emotion-ridden conversations about death and dying. But Obermeyer suggests the opposite may be true. He told me:
When algorithms have access to so much data, in a counterintuitive way, it can be very humanizing: the algorithm sees an incredibly rich picture of you, your medical history, your social support, how well you’re breathing today versus last week. It doesn’t change its behavior if you get angry and snap at your doctor because you’re in pain. I think there’s that warmer, fuzzier side of algorithms that we overlook. They don’t miss anything. And that can be very reassuring.
As Obermeyer suggests, while their potential health care impact is huge, these algorithms may serve a better purpose in the hands of patients and families. Imagine how differently patients and their doctors would approach health care if they knew someone’s one-year mortality risk. Survey research suggests that half of patients with cancer want an exact quantitative prognosis, but only half of those receive one (and, of course, what they get isn’t terribly reliable). People may use prognostic information for many non–health care decisions, including whether to take a vacation or how much money to save for retirement. Just as a pregnant woman may elect not to know the gender of her baby, many patients and families may choose not to know their prognosis. But for a majority of people, an accurate knowledge of longevity can make ambiguous decisions much clearer.
As terms like precision medicine and patient-centered care abound, we have grown more accustomed to the idea of personalizing health care. Accurate prognosis is key to that effort. Physicians and others have what Christakis calls “a duty to prognosticate” but don’t do so because they don’t have accurate resources. Machine learning and other analytics can change that—and in doing so, can change the way patients and doctors approach health care and the prospect of death. In his book Being Mortal, author and physician Atul Gawande writes, “how we seek to spend our time may depend on how much time we perceive ourselves to have.” The machine can help with this, freeing us from trying to live longer so that we can just live.