Consequences and causal models

Class

Author

Joshua Loftus

Note

This is the student version of the class sheet. Parts marked (class) are for the class; the rest are practice. Answers are revealed in class.

In class

  1. A game, then a discussion of it.
  2. A discussion of the arguments you brought.
  3. Problems 1 and 2, by hand.
  4. A discussion of a model that makes decisions about people.
  5. A debate, then a vote.

Discussions 1 to 3 are in groups of three or four, then with the room. For discussion 4 the room splits into four parts. Each problem is gone through as soon as the room has tried it. Everything under “Optional, can be done at home” is practice for afterward.

Discussion 1: absurd trolley problems

In your group, open Absurd Trolley Problems on one device and play for about six minutes, choosing together. Note the scenarios where the group disagreed, and the ones you personally found hard to choose.

  1. Take one scenario where the group disagreed. Which framework was each side using?
  2. Did any of your choices depend on a prediction of what would happen? How sure of it were you?

Discussion 2: the arguments you brought

Put your arguments side by side and choose one.

  1. What is the action, and which outcome does the argument count? Whose good is counted, and whose is left out?
  2. Draw the causal graph the argument needs, with an arrow for each cause-and-effect claim you listed. Mark the action.
  3. Try to draw a second graph that fits the same evidence and under which the action does nothing. If the evidence makes every such graph hard to believe, say what in the evidence does that.
  4. Argue against the policy without disputing any of its causal claims. Which framework are you using?

Each group’s graph, and its argument from question 4, go on the screen.

Problems

Before you start, from memory: what does the adjustment formula average, and with what weights?

1. Conditioning and intervening

A program in the style of the reading’s. \(U_1, \dots, U_5\) are independent, with \(U_1 \sim \mathrm{Ber}(0.5)\), \(U_2 \sim \mathrm{Ber}(0.6)\), \(U_3 \sim \mathrm{Ber}(0.2)\), \(U_4 \sim \mathrm{Ber}(0.4)\) and \(U_5 \sim \mathrm{Ber}(0.1)\), where \(\mathrm{Ber}(0.6)\) is 1 with probability 0.6 and 0 otherwise.

  1. \(X := U_1\) (exercise)
  2. \(W :=\) if \(X = 1\) then \(U_3\) else \(U_2\) (overweight)
  3. \(H :=\) if \(X = 1\) then \(U_5\) else \(U_4\) (heart disease)

  1. Draw the graph.

  2. Compute \(P(W = 1)\) and \(P(X = 0 \mid W = 1)\).

  3. Compute \(P(H = 1)\), \(P(H = 1 \mid W = 1)\) and \(P(H = 1 \mid W = 0)\).

  4. A program changes everyone’s weight to normal: \(W := 0\). Compute \(P(H = 1 \mid do(W := 0))\). What does the program do to heart disease?

  5. Compute the average treatment effect of exercise on heart disease, and compare it with \(P(H = 1 \mid X = 1) - P(H = 1 \mid X = 0)\). Are they equal? Why?

2. Adjusting for a confounder

A school offers a revision app before a test. Students who passed the previous year’s test (\(Z = 1\)) are more likely to use it (\(X = 1\)). The outcome is passing this year (\(Y = 1\)).

used the app (\(X = 1\)) did not (\(X = 0\))
passed last year (\(Z = 1\)), 500 students 400, of whom 320 passed (80%) 100, of whom 70 passed (70%)
did not (\(Z = 0\)), 500 students 100, of whom 50 passed (50%) 400, of whom 120 passed (30%)

  1. Compute the pass rates of users and non-users, and the difference.

  2. Draw the graph you would assume, and compute the average treatment effect of the app with the adjustment formula.

  3. Compute the difference within each row. How does the average treatment effect combine the two?

  4. A second school has the same pass rates within each cell of the table, but only 20% of its students passed last year. What is the average treatment effect of the app there?

  5. A head teacher asks whether to buy the app for every student. Which of your numbers would you report, and what would you say about it?

Discussion 3: a model that decides about people

Think of a predictive algorithm that makes, or helps to make, decisions with large effects on people: which applicants get a loan, which patients a GP sees first, which benefit claims are checked for fraud. Choose one in your group.

  1. Describe the scenario: who uses the model, what it predicts, from what data, and what is decided with it.
  2. Half the groups take the four Belmont–Menlo principles, and half take the five Cs (consent; clarity; consistency and trust; control and transparency; consequences). For each one, say what would have to be in place: a feature of the system, or a procedure around it.
  3. Which of your requirements could be checked only with a causal claim?

Two or three groups describe their scenario and its requirements to the room.

Discussion 4: the case for one framework

The room splits into four parts. Each part prepares the strongest case it can for one statement.

  1. Ethical data science and AI development requires us to lean mostly on consequentialist ethics.
  2. Ethical data science and AI development requires us to lean mostly on deontological ethics.
  3. Ethical data science and AI development requires us to lean mostly on virtue ethics.
  4. Ethical data science and AI development requires us to lean mostly on pragmatic ethics.

Five minutes to prepare, then a minute for each part. Each case should answer the strongest objection to it. For consequentialism, one objection is the lecture’s last question: does it become more or less defensible once you have to write the causal model down?

Then everyone votes, whichever part they argued for. Which account are you most convinced of: consequentialist, deontological, virtue or pragmatic ethics, a combination, or none of them?

If there is time

  • “Correlation (more generally, association) is not causation.” This is true and important, but can it be misused?
  • Can causality be useful in the context of reproducibility? In designing and communicating studies?
  • A company tests two versions of its app on its users, without telling them, to see which version keeps them using the app longer. Which of the Belmont–Menlo principles does this touch? How would each of the four frameworks judge it?
  • Find a sentence in a news report of an observational study that avoids saying “causes” but implies a cause. What graph does it assume?

To finish

One sentence, individually: what is the difference between \(P(Y = 1 \mid X = 1)\) and \(P(Y = 1 \mid do(X := 1))\)?

For next time

Bring one score or algorithm used to make decisions about people. Write down what it measures and what it stands in for. Every group’s examples will go on the screen.

Optional, can be done at home

Practice for afterward: problems 3 to 7, then coding A to C. The problems can be done by hand. Attempt each part before you open a hint.

WarningHints

Do not open a hint unless you have been stuck for at least a few minutes, and open them in order.

3. Omitted-variable bias

In the linear model of the notes, \(Z := U_Z\), \(X := \alpha Z + U_X\), \(Y := \beta X + \gamma Z + U_Y\), with independent mean-zero noise, take \(\alpha = 2\), \(\gamma = 1\), \(\beta = 0.5\) and \(\operatorname{Var}(Z) = \operatorname{Var}(U_X) = 1\).

  1. Compute \(\operatorname{Var}(X)\), \(\operatorname{Cov}(X, Z)\) and the slope of \(Y\) on \(X\) alone.

  2. Repeat with \(\gamma = -1\).

  3. Repeat with \(\gamma = 1\) and \(\beta = 0\). What would someone with data on \(X\) and \(Y\) alone conclude?

  4. Is the error in (a) a bias or a variance? What happens to it as the sample grows?

\(Z\) and \(U_X\) are independent, so \(\operatorname{Var}(X) = \alpha^2 \operatorname{Var}(Z) + \operatorname{Var}(U_X)\).

4. Choosing an adjustment set

  1. List the backdoor paths from \(X\) to \(Y\).

  2. Which of these sets satisfy the backdoor criterion: the empty set, \(\{Z\}\), \(\{M\}\), \(\{Z, M\}\), \(\{W\}\), \(\{Z, W\}\)?

  3. In a linear model, what does the coefficient on \(X\) estimate if you adjust for \(\{Z, M\}\)?

  4. If more than one of the sets in (b) satisfies the criterion, which would you use, and why?

A backdoor path starts with an arrow into \(X\). How many arrows point into \(X\)?

5. The graph behind an argument

A university says: “Students who use the library more get better grades. We will extend the library’s opening hours, because using the library improves grades.”

  1. Draw the causal graph the argument presupposes, and mark the action.

  2. Add one variable that would make the observed difference larger than the effect, with its arrows.

  3. What comparison would support the university’s claim?

The university’s evidence compares students. Its action changes opening hours. Are those the same thing?

6. Prediction letters and reinforcement-learning letters

The decision whether to prescribe hormone therapy to prevent heart disease can be written with the letters used for prediction or with those of reinforcement learning.

  1. Write it with the letters used for prediction. Say what the treatment, the outcome and a confounder are, and write the quantity a doctor needs with \(do(\cdot)\).

  2. Write it with the letters of reinforcement learning, with the one-line declaration at the top. Say what the state, the action, the reward, the policy and the utility are.

  3. Which letters would mean different things if you used them in both answers?

For (a), what does the doctor choose, and what could affect both that choice and heart disease? For (b), what does the doctor know when choosing?

7. Hormone therapy

  1. The 1996 estimate (0.39) and the 2008 estimates (1.42 over the first two years, 0.96 over the whole follow-up) come from the same cohort. Which comparison did each make?

  2. Draw a graph under which the 1996 estimate is confounded.

  3. Which comparison matches the decision a doctor makes, and why?

For (a), ask who is compared with whom in each analysis.

Coding

Open a fresh R script. You will need the tidyverse.

A. Prediction and intervention

n <- 10000
# program 1: X causes Y
X1 <- rnorm(n); Y1 <- 2 * X1 + rnorm(n)
# program 2: Z causes both
Z2 <- rnorm(n); X2 <- Z2 + rnorm(n); Y2 <- Z2 + rnorm(n)
# program 3: X causes Y through M
X3 <- sqrt(2) * rnorm(n); M3 <- X3 + sqrt(2) * rnorm(n); Y3 <- 0.5 * M3 + rnorm(n)
  1. Write down the slope of \(Y\) on \(X\) you expect in each program. Then fit lm and compare.
  2. For each program, intervene: set \(X\) to 1 for everyone and run the statements after the one for \(X\) again, then the same with \(X\) set to 0. Compute the difference in the means of \(Y\).
  3. In program 2, fit lm(Y2 ~ X2 + Z2). In program 3, fit lm(Y3 ~ X3 + M3). Compare the coefficients on \(X\) with your answers to 2.

In each program, which statements after the one for \(X\) use \(X\), directly or through a variable computed from it? Those are the ones whose values change when you set \(X\).

B. Adjustment on data

cells <- data.frame(Z = c(1, 1, 0, 0), X = c(1, 0, 1, 0),
                    n = c(400, 100, 100, 400), passed = c(320, 70, 50, 120))
students <- bind_rows(
  cells |> mutate(Y = 1, count = passed),
  cells |> mutate(Y = 0, count = n - passed)
) |>
  select(Z, X, Y, count) |>
  tidyr::uncount(count)
nrow(students)
[1] 1000
  1. Compute the pass rates of users and non-users from students, and their difference.
  2. Compute the adjusted difference: the difference in pass rates within each value of \(Z\), averaged with weights \(P(Z = z)\).
  3. A randomized version: give each of 1,000 students the app with probability 0.5, whatever their \(Z\), and draw whether they pass from the table’s rates. Compute the difference in pass rates. Repeat 200 times. What is the average difference, and how much does it vary?

For 2: students |> group_by(Z) |> summarize(...) gives one row for each value of \(Z\); compute the difference in pass rates and the share of students in each row there.

C. Intervening in a program

The job center from the lecture, as a program. As in the lecture, everyone referred attends and nobody else does. Each line generates one variable from the ones above it.

n <- 10000
Z <- rbinom(n, 1, 0.5)                        # worked last year
X <- rbinom(n, 1, ifelse(Z == 1, 0.8, 0.2))   # attended the course
Y <- rbinom(n, 1, 0.3 + 0.1 * X + 0.3 * Z)    # in work six months later
mean(Y[X == 1]) - mean(Y[X == 0])             # attenders against the others
  1. Run it. Compare the difference with the lecture’s first job-center table (64% against 36%).
  2. Compute the adjusted difference: the difference between attenders and non-attenders within \(Z = 1\) and within \(Z = 0\), averaged with weights mean(Z == 1) and mean(Z == 0).
  3. Intervene. Replace the line for X by X <- rep(1, n), run the line for Y again, and record mean(Y). Do the same with X <- rep(0, n). Compare the difference with your answer to 2.
  4. The center changes its referrals to 0.95 for \(Z = 1\) and 0.05 for \(Z = 0\). Before you run anything, write down what you expect for the differences in 1 and in 3. Then run it.

For step 3, replace the line for X and leave the line for Z alone. Then run the line for Y again, so that it uses the new X.