Part 2: Cost as a proxy for need
A risk score in health care
A hospital uses a commercial risk score to decide who is offered extra care. Patients above the 97th percentile are identified for a care program automatically; those above the 55th are flagged to their doctor.
The score predicts next year’s medical cost. Race is not one of its inputs.
By industry estimates, tools of this class are applied to roughly 200 million people in the United States each year.
Obermeyer, Powers, Vogeli and Mullainathan (2019), “Dissecting racial bias in an algorithm used to manage the health of populations,” Science 366(6464), 447–453, doi:10.1126/science.aax2342; free full text at escholarship.org/uc/item/6h92v832.
Sketch it
If cost tracked need equally for Black and White patients, what would the two curves look like?
The same score, more illness
At every level of the score, the Black patients it selects are sicker.
Obermeyer and colleagues (2019), Figure 1A and text: 4.8 against 3.8 conditions at the 97th percentile (\(P < 0.001\)).
A model: an offset for one group
\(Y^\star\) need, assumed here to be the same in both groups; \(A = 1\) for one group, \(0\) for the other; selected if \(Y > t\), where \(Y = Y^\star + \beta_A A\).
For group \(1\), \(Y > t\) is \(Y^\star > t - \beta_A\): a threshold moved for that group.
With \(\beta_A = -0.5\): is group \(1\) selected more or less often?
Who is selected, and how ill
\(Y^\star \sim N(0, 1)\) in both groups; \(t\) is the 97th percentile.
| 0.0 |
3.0% |
3.0% |
1.00 |
2.27 |
2.27 |
| -0.5 |
3.0% |
0.9% |
0.29 |
2.27 |
2.71 |
| -1.0 |
3.0% |
0.2% |
0.07 |
2.27 |
3.17 |
With \(\beta_A < 0\), group \(1\) is selected with probability \(P(Y^\star > t - \beta_A) < P(Y^\star > t)\), and its selected members have higher mean need than group \(0\)’s.
The mean of a standard normal above \(z\) is \(\phi(z)/(1 - \Phi(z))\). Where did that appear before?
Changing the label
Removing the gap between the groups, in a simulation: the share of Black patients among those identified automatically rises from 17.7% to 46.5%.
Training a new model on each of three labels (the paper’s Table 2), the Black share of patients above the 97th percentile:
| total cost |
14.1% |
| avoidable cost |
21.0% |
| active chronic conditions |
26.7% |
Which label would you choose for this program, and who is worse off under it?
Obermeyer and colleagues (2019), body text and Table 2. The abstract states the first figure loosely, as “the percentage of Black patients receiving additional help”; the body states it as the Black share of those auto-identified at the 97th percentile.
Why cost is lower at the same need
At the same number of chronic conditions, Black patients in the sample generated about $1,801 less in medical costs a year. Everyone in the sample was insured.
Two routes the authors name: barriers to care (poverty, geography, transport, work and child care, knowing when to seek care), and the effect of race on the care a patient receives.
What are we optimizing or predicting? Who chose cost as the label, and why?
Who bears the error
| ill enough to need the program |
right |
left out |
| not |
a place used |
right |
A threshold \(t\) trades the two errors. An offset \(\beta_A\) in the measure moves the second error, being left out though ill, onto the group with the offset.
Who benefited from that choice being invisible? And would a more competent analyst have chosen differently?
A noisy measure flattens a slope
\(Y = Y^\star + \varepsilon\), noise independent of need. The best guess of need from the measure (normal case) is \[
E[Y^\star \mid Y = y] = \lambda y, \qquad \lambda = \frac{\operatorname{Var}(Y^\star)}{\operatorname{Var}(Y^\star) + \operatorname{Var}(\varepsilon)} < 1 .
\]
Noise and an omitted variable
Health \(Y\); visits \(X^\star\), reported as \(X = X^\star + \varepsilon\); illness \(Z\), not in the data. \[
Y = \beta X^\star + \gamma Z + U ,
\qquad
\text{slope of } Y \text{ on } X = \lambda\,(\beta + \gamma\delta),
\] with \(\lambda = \operatorname{Var}(X^\star)/\operatorname{Var}(X)\) and \(\delta = \operatorname{Cov}(X^\star, Z)/\operatorname{Var}(X^\star)\).
Visits help, \(\beta > 0\). The ill visit more and are less healthy next year. Can the slope be negative? Can noise make it so?
Part 3: What gets rewarded
Reinforcement learning
Notation: in this part, the letters of reinforcement learning: \(S\) state, \(A\) action, \(R\) reward, \(\pi\) policy, \(U\) utility.
Learning: change \(\pi\) to make the expected reward \(E_\pi[R]\) large.
A utilitarian wants the largest \(E_\pi[U]\), with \(U\) the good we care about. The agent is given \(R\).
A cleaning robot
Notation: reinforcement learning (\(S\), \(A\), \(R\), \(\pi\), \(U\)).
- \(S\): the camera’s view
- \(A\): move, or clean
- \(R\): 1 if no mess in view
- \(U\): a clean room
Amodei, Olah, Steinhardt, Christiano, Schulman and Mané (2016), “Concrete Problems in AI Safety,” sections 2 and 4.
A job center as an agent
Notation: reinforcement learning (\(S\), \(A\), \(R\), \(\pi\), \(U\)).
A variant of week 3’s job center: 100 places on the course, which helps some people more than others. Take the rates below as what would happen if the course were assigned, or withheld.
| worked last year |
70% |
64% |
| did not |
40% |
14% |
- \(S\): a person’s record
- \(A\): refer or not
- \(R\): placement rate, the share of referrals in work
- \(U\): extra people in work
Who does a center judged on \(R\) send? Who would a utilitarian send?
Cream skimming
Notation: reinforcement learning (\(S\), \(A\), \(R\), \(\pi\), \(U\)).
Cream skimming: choosing people who would have found work anyway. In the US Job Training Partnership Act programs, centers were judged on short-run measures (\(R\)) that were “weakly, and sometimes perversely, related to long run impacts” (\(U\)).
Heckman, Heinrich and Smith (2002), “The Performance of Performance Standards,” NBER Working Paper 9002, doi:10.3386/w9002, abstract.
Need, or benefit
Notation: reinforcement learning (\(S\), \(A\), \(R\), \(\pi\), \(U\)).
The job center chose between two utilities: people likely to be in work, or people the course helps most.
The hospital’s program mostly prevents acute episodes in people with chronic illness. Two candidates for \(U\): how ill a patient is, and how much the extra care would help them.
Benefit is an effect, a causal question.
Should the program go to the sickest, or to those the care would help most? What would you need to know to tell them apart, and what are you deciding when you choose?
Utilitarianism with a reward
Notation: reinforcement learning (\(S\), \(A\), \(R\), \(\pi\), \(U\)).
An agent judges a policy only by the reward it expects. That is consequentialism, with the good defined by whoever chose \(R\).
Last week listed four difficulties of utilitarianism. Which are they?
| defining utility |
someone chooses \(R\), and \(R\) is not \(U\) |
| trading off between people |
reward is added up over the people served |
| judging acts or rules |
the policy is a rule: it treats alike everyone in the same state |
| predicting consequences |
a model of what the actions do, which is a causal model |
Part 4: Systems that adapt
When a measure becomes a target
Before a rule rewards \(Y\): \(\;Y = Y^\star + \varepsilon\).
After: each person can add \(g \ge 0\) to \(Y\), at a cost, without changing \(Y^\star\): \[
Y = Y^\star + \varepsilon + g .
\]
Replacing the equation for \(Y\) is an intervention. The relation between \(Y\) and \(Y^\star\) that held before the rule need not hold after it.
Two ways to respond
300 places go to the 300 highest \(Y\), where \(Y = Y^\star + \varepsilon\). Then the rule is announced.
- Everyone adds \(1\) to their \(Y\).
- Half of the people are able to add \(1\); the rest cannot.
In each case: who gets the places, compared with before? Does the mean need of the people selected change?
The same score, before and after
Everyone adding the same amount leaves the ranking unchanged. A response that differs between people moves the places toward those who can respond: \(Y\)’s equation has changed.
Gaming and improvement
A course is judged by its students’ test scores. A response is improvement if it raises \(Y^\star\), and \(Y\) with it; gaming if it raises \(Y\) by a route that does not pass through \(Y^\star\).
A third response: learning the examiner’s favorite phrasing. Draw its graph. Gaming or improvement, and what does it depend on?
Miller, Milli and Hardt (2020), “Strategic Classification is Causal Modeling in Disguise,” ICML, PMLR 119, 6917–6926. The example is ours.
Two models of a system that adapts
Strategic classification
- A classifier is published; people change their features, at a cost, for the better label. True labels stay fixed: every response is gaming.
- Design: accuracy after people respond, against their original labels.
Performative prediction
- The deployed model changes the distribution it is judged on, by any mechanism: a traffic forecast changes routes. Labels can move too.
- Question: what does retraining converge to?
In both, the rule is an intervention, not a sample.
Hardt, Megiddo, Papadimitriou and Wootters (2016), “Strategic Classification,” ITCS 2016, 111–122, doi:10.1145/2840728.2840730; Perdomo, Zrnic, Mendler-Dünner and Hardt (2020), “Performative Prediction,” ICML, PMLR 119, 7599–7609.
Engagement
Notation: reinforcement learning (\(S\), \(A\), \(R\), \(\pi\), \(U\)).
YouTube, August 2012: its discovery features had been “designed to drive views,” which rewarded videos that got clicked over videos that kept viewers watching. The reward was changed to watch time.
\(R\): engagement, whatever the platform can measure. \(U\): a user who is informed, and not manipulated. Where raising \(U\) would lower \(R\), revenue pulls the other way.
Who benefits from the gap between \(R\) and \(U\) staying invisible?
Meyerson, “YouTube now: why we focus on watch time,” YouTube Official Blog, 10 August 2012, blog.youtube/news-and-events/youtube-now-why-we-focus-on-watch-time.
Lenses this week
- What are we optimizing or predicting? Cost, standing in for need. Citations, for contribution. A placement rate, for jobs created. Engagement, for a user’s good.
- Who bears the error? A threshold makes two errors; an offset moves the error of leaving people out onto one group. A robot’s owner bears the gap between its reward and a clean room.
- What can be changed, and how? What things are considered “controlled for” or “held fixed,” and is that realistic? Benefit compares care with no care for the same patient. A slope of health on visits is read with illness held fixed. A rule on the score assumed that the score-need relation would stay fixed.
…and why?
Your score, again
The score you wrote down at the start. Recommend one change to how it is used. Which mechanism from today does your change address, and who might lose from it?
Was the choice of cost as the label a technical error or a value choice?
For next time: bring one dataset or study where some people who should be in it are not, with who is missing and in which direction that pushes the result.
