Ethics and causation

Class

Author

Joshua Loftus

Discussion 1: absurd trolley problems

In your group, open Absurd Trolley Problems on one device and play for about six minutes, choosing together. Note the scenarios where the group disagreed, and the ones you personally found hard to choose.

  1. Take one scenario where the group disagreed. Which framework was each side using?
  2. Did any of your choices depend on a prediction of what would happen? How sure of it were you?

In the original problem, pulling the lever to save five people at the cost of one is what a consequentialist who counts lives would do. Refusing to pull it, because diverting the trolley makes you the cause of one death while doing nothing lets five die, is one deontological answer: it rests on a difference between doing and allowing. Other deontologists allow pulling the lever, by the doctrine of double effect: the one death is foreseen but not intended. A virtue ethicist asks what a good person would do there, and can answer either way. Where a choice turned on a prediction, the consequentialist’s answer depended on a causal claim, and the game gives no way to check it.

Discussion 2: a model that decides about people

Think of a scenario in which a predictive algorithm is deployed to make, or help to make, decisions with large effects on people: which applicants get a loan, which patients a GP sees first, which benefit claims are checked for fraud. Choose one in your group.

  1. Describe the scenario: who uses the model, what it predicts, from what data, and what is decided with it.
  2. Decide what features or mechanisms would have to be in place to satisfy the four Belmont–Menlo principles and the five Cs (consent; clarity; consistency and trust; control and transparency; consequences). Say which principle or C each one serves.
  3. Which of your requirements is about an effect, so that checking it needs a causal claim?

At the end, each group describes its scenario and its requirements to the other groups.

A model suggests treatments to NHS GPs, to help them get through patient cases faster. It needs private information about each patient, and may store it. It would need, among other things:

  • informed consent from the patient, or from someone legally acting for them, to the use and storage of their data, and the data held securely or not at all (respect for persons; consent);
  • a clear account to the patient of how the model is used, how their data is used and what happens to the data afterward (clarity; consistency and trust; beneficence);
  • no use of the model for a patient who says no (control and transparency; beneficence);
  • high accuracy, since a wrong suggestion can harm a patient (beneficence);
  • a model that is demonstrably fair across demographic groups, since there are documented cases of medical algorithms biased by race or by sex (justice; consistency and trust);
  • storage and processing that meet data-protection law such as the UK GDPR (respect for law and the public interest);
  • care that its records cannot be combined with other sources to identify patients (consequences).

Accuracy is a property of the predictions. Whether patients do better when their GP uses the suggestions than when the GP does not is an effect, and that requirement needs a causal claim.

A council uses a model to choose which housing-benefit claims its officers check for fraud. The model was trained on the results of past checks and uses each claim and other council records. It would need, among other things:

  • a notice to every claimant that a model helps choose which claims are checked, what data it uses and what a check involves (clarity; respect for persons). Consent is weak here, since a claimant cannot refuse and still claim, so the other requirements carry more weight;
  • an officer who looks at each chosen claim before anything happens to it, so that the model only decides which claims are looked at (beneficence; respect for law and the public interest);
  • check rates and error rates compared across groups, such as disabled claimants, single parents or claimants of one nationality, so that the burden of checks does not fall mostly on one group. The past checks the model learned from were themselves chosen by officers, so the model may learn to pick out the claimants they used to suspect (justice; consistency and trust);
  • a way for claimants to see and correct the data used and to challenge a check, and a named officer who answers for the system (respect for law and the public interest; control and transparency);
  • limits on joining other council records to the claims, and on what the results may be used for afterward (consequences).

Whether a check delays payments or causes hardship to honest claimants, and whether checks deter fraud, are effects of the council’s choice to check, so those requirements need causal claims. Whether flagged claims are more often fraudulent than claims chosen at random is a question about the model’s accuracy, and even that is hard to measure, because few claims that were not flagged were ever checked.

Problems

Before you start, from memory: what does the adjustment formula average, and with what weights?

It averages the outcome among people with \(T = t\) within each group defined by the adjustment variables, with weights equal to each group’s share of the whole population, not its share among the people with \(T = t\).

1. A safety course for riders

A food-delivery platform offers its riders a road-safety course. It pushes the course hardest to new riders and to riders who work mostly at night. New riders have more accidents than experienced riders, by day and by night, and night work raises accidents for both. New riders, who book their shifts last, more often work at night. Assume that nothing else affects both taking the course and having an accident. The platform’s records for 1,000 riders, with \(Z = 1\) for a rider who had joined in the three months before the course was offered, \(W = 1\) for one who then worked mostly at night, \(T = 1\) for one who took the course and \(Y = 1\) for one who had an accident in the six months after:

new (\(Z\)) at night (\(W\)) course (\(T\)) riders accidents
yes yes yes 70 21
yes yes no 30 12
yes no yes 100 10
yes no no 100 15
no yes yes 80 4
no yes no 120 18
no no yes 50 1
no no no 450 18

  1. Draw the graph with \(T\), \(Y\), \(Z\) and \(W\). Which set of variables satisfies the backdoor criterion for the effect of \(T\) on \(Y\)?

  2. Compute the share who had an accident among riders who took the course and among those who did not.

  3. Compute \(P(Y = 1 \mid do(T := 1))\) and \(P(Y = 1 \mid do(T := 0))\) with the adjustment formula, and the average treatment effect.

  4. A manager proposes to drop the course, citing (b). An analyst who adjusts only for whether riders are new finds that the course lowers the chance of an accident by about 2.5 percentage points. What would you tell each of them?

For (a), list the arrows into \(T\). A set that blocks the paths that start with one of them can leave the paths that start with the other open.

For (c), find the share with an accident with and without the course in each of the four groups of riders, then weight each group by its share of all 1,000 riders.

  1. \(Z \to W\), \(Z \to T\), \(W \to T\), \(Z \to Y\), \(W \to Y\) and \(T \to Y\).

Two arrows point into \(T\), and four backdoor paths start with them: \(T \leftarrow Z \to Y\), \(T \leftarrow W \to Y\), \(T \leftarrow Z \to W \to Y\) and \(T \leftarrow W \leftarrow Z \to Y\). \(\{Z, W\}\) blocks all four, and neither variable is a descendant of \(T\). \(\{Z\}\) alone leaves \(T \leftarrow W \to Y\) open, and \(\{W\}\) alone leaves \(T \leftarrow Z \to Y\) open. The graph rests on the assumption in the question. If cautious riders were both more likely to take the course and less likely to have an accident, caution would be a back door that nobody measured.

  1. With the course: \((21 + 10 + 4 + 1)/(70 + 100 + 80 + 50) = 36/300 = 0.12\). Without: \((12 + 15 + 18 + 18)/(30 + 100 + 120 + 450) = 63/700 = 0.09\). Riders who took the course had an accident rate 3 points higher.

  2. Within each group of riders:

new at night share of riders with the course without
yes yes 0.1 21/70 = 0.30 12/30 = 0.40
yes no 0.2 10/100 = 0.10 15/100 = 0.15
no yes 0.2 4/80 = 0.05 18/120 = 0.15
no no 0.5 1/50 = 0.02 18/450 = 0.04

\(P(Y = 1 \mid do(T := 1)) = 0.1 \times 0.30 + 0.2 \times 0.10 + 0.2 \times 0.05 + 0.5 \times 0.02 = 0.07\) and \(P(Y = 1 \mid do(T := 0)) = 0.1 \times 0.40 + 0.2 \times 0.15 + 0.2 \times 0.15 + 0.5 \times 0.04 = 0.12\). The average treatment effect is \(-0.05\): the course lowers the chance of an accident by 5 points. The sign of (b) is reversed because 83% of the riders who took the course were new or worked at night, against 36% of those who did not, and those riders have more accidents with or without the course.

  1. To the manager: riders who took the course had more accidents because the platform gave it to the riders most likely to have one. In every group the course lowers the rate. If the graph is right, the 300 riders who took it would have had about \(70 \times 0.40 + 100 \times 0.15 + 80 \times 0.15 + 50 \times 0.04 = 57\) accidents without it, against the 36 they had, so dropping it would cost about 21 accidents in six months among riders like these. To the analyst: adjusting for \(Z\) alone leaves the back door through night work, \(T \leftarrow W \to Y\), open, and here that hides half the effect. (Adjusting for \(W\) alone hides nearly all of it.) The 5 points and the 21 accidents rest on the graph; offering the course to riders chosen at random would test it.

2. Backdoor paths

In each graph, \(T\) is the treatment and \(Y\) the outcome.

A path is blocked by a set of variables if somewhere along it

  • a fork, \(\leftarrow V \to\), or a chain, \(\to V \to\) or \(\leftarrow V \leftarrow\), has its middle variable \(V\) in the set, or
  • a collider, \(\to V \leftarrow\), has neither \(V\) nor any descendant of \(V\) in the set.

The second rule is the reading’s: a path through a collider is blocked when you adjust for nothing, and adjusting for the collider opens it.

  1. For each graph, list the backdoor paths from \(T\) to \(Y\). Which of them are open when you adjust for nothing?

  2. For each graph, find a smallest set of variables that satisfies the backdoor criterion for the effect of \(T\) on \(Y\). Is it the only one of that size?

  3. In graph 2, an analyst adjusts for \(C\), because \(C\) was measured before \(T\) and “adjusting for more can’t hurt.” Which path does that open? Suppose \(A\), \(B\) and \(C\) are yes-or-no variables, \(A\) and \(B\) are independent, and \(A = 1\) or \(B = 1\) is enough to make \(C = 1\), as either disease was enough for admission to hospital in the reading’s example. Among people with \(C = 1\), are \(A\) and \(B\) associated, and in which direction?

  4. In graph 3, only \(C\) was measured. Can the effect of \(T\) on \(Y\) be estimated by adjusting for measured variables?

A backdoor path starts at \(T\) with an arrow into \(T\), then follows arrows in either direction to \(Y\) without visiting a variable twice.

In graph 3, \(C\) is on four backdoor paths. On which of them is it a collider?

  1. Graph 1: one, \(T \leftarrow A \to B \to Y\), open. Graph 2: one, \(T \leftarrow A \to C \leftarrow B \to Y\), blocked, because \(C\) is a collider on it. Graph 3: four, \(T \leftarrow C \to Y\), \(T \leftarrow A \to C \to Y\), \(T \leftarrow C \leftarrow B \to Y\) and \(T \leftarrow A \to C \leftarrow B \to Y\). The first three are open; the fourth is blocked at the collider \(C\).

  2. Graph 1: \(\{A\}\) or \(\{B\}\). The path can be blocked anywhere along it, so the common cause \(A\) need not be measured if \(B\) is. Graph 2: the empty set. Any set without \(C\) also works, such as \(\{A\}\); a set with \(C\) and neither \(A\) nor \(B\) does not. Graph 3: \(\{A, C\}\) or \(\{B, C\}\). Every set that blocks \(T \leftarrow C \to Y\) contains \(C\), and adjusting for \(C\) opens \(T \leftarrow A \to C \leftarrow B \to Y\), so \(A\) or \(B\) must be added to close it again. No single variable works.

  3. It opens \(T \leftarrow A \to C \leftarrow B \to Y\). Among people with \(C = 1\), \(A\) and \(B\) are negatively associated: among them, those with \(A = 0\) have \(C = 1\) for some other reason, often \(B\), so \(B = 1\) is more common among them than among those with \(A = 1\). The reading’s two diseases become negatively correlated among hospital patients in the same way. So among people with \(C = 1\), \(T\) and \(Y\) are associated through \(A\) and \(B\) even if \(T\) does nothing, and adjusting for \(C\) creates a bias that the empty set did not have. If \(A\) raises \(T\) and \(B\) raises \(Y\), the comparison within \(C = 1\) is biased downward. Measured before \(T\) does not mean safe to adjust for.

  4. No. The empty set leaves \(T \leftarrow C \to Y\) open, and \(\{C\}\) opens \(T \leftarrow A \to C \leftarrow B \to Y\). Neither satisfies the criterion. Adjusting for \(C\) may still leave less bias than adjusting for nothing; the graph alone does not say. To remove the bias, measure \(A\) or \(B\), or assign \(T\) at random.

Discussion 3: the case for one framework

The room splits into three groups. Each group develops an argument for one statement and presents it to the other groups.

  1. Ethical data science and AI development requires us to lean mostly on consequentialist ethics.
  2. Ethical data science and AI development requires us to lean mostly on deontological ethics.
  3. Ethical data science and AI development requires us to lean mostly on virtue ethics.

You have 15 minutes to develop your argument and 2 minutes to present it. Think about how to structure it to convince the listener, and answer the strongest objection to it. The group arguing for consequentialism should also answer this question: does it become more or less defensible once you have to write the causal model down?

Then everyone votes for the account they find most convincing, which need not be the one their group argued for: consequentialist, deontological or virtue ethics for data science and AI, a combination, or none of them?

Data science is used to make decisions at scale: who gets a loan, which posts are shown, which claims are checked. Such a decision is judged by what it does to the people it affects, added up over many cases, and the field’s own tools (outcomes, comparisons, trials) are built to measure that. When every option harms someone, rules give no way to weigh one harm against another, and the virtues of individual employees say little about a system that no one person runs.

The strongest objection is that adding up outcomes lets a benefit to many justify a serious harm to a few. The reply: count how the harms and benefits are shared out, not only their total.

Writing the causal model down makes the position more defensible, because its claims about consequences become explicit and open to criticism. It also shows how much of any consequentialist case rests on assumptions that the data cannot check, which is a strong argument against it.

Rules cannot anticipate every use of a model, and the consequences of a new system can rarely be predicted well enough to add up. What is left is the judgment of the people who build and run it. Virtue ethics asks what kind of practitioner to be: honest about uncertainty, careful with other people’s data, willing to say no when a manager wants a misleading result. Those are habits a team can train and a profession can expect; the ASA’s ethical guidelines, in the reading, ask statisticians to be honest about the limits of their expertise.

The strongest objection is that “be honest and careful” gives no answer in a hard case, such as whether to use a model that helps most claimants and wrongly flags a few. The reply: no framework settles that case mechanically. A consequentialist needs a causal model and a deontologist a ranking of duties, and someone with good judgment has to supply both.

If there is time

  • One sentence, individually: what is the difference between \(P(Y = 1 \mid T = 1)\) and \(P(Y = 1 \mid do(T := 1))\)?
  • The reading warns against judging an admissions test by how well it predicts success among the students who were admitted. Draw the graph. Where is the collider?
  • “Correlation (more generally, association) is not causation.” This is true and important, but can it be misused?
  • Can causality be useful in the context of reproducibility? In designing and communicating studies?
  • A company tests two versions of its app on its users, without telling them, to see which version keeps them using the app longer. Which of the Belmont–Menlo principles does this touch? How would each framework judge it?
  • Find a sentence in a news report of an observational study that avoids saying “causes” but implies a cause. What graph does it assume?

For next time

Bring one score or algorithm used to make decisions about people. Write down what it measures and what it stands in for. Every group’s examples will go on the screen.

Optional, can be done at home

Practice for afterward: problems 3 to 6, then coding A to C. The problems can be done by hand. Attempt each part before you open a hint.

WarningHints

Do not open a hint unless you have been stuck for at least a few minutes, and open them in order.

3. Omitted-variable bias

In the linear model of the notes, \(Z := U_Z\), \(X := \alpha Z + U_X\), \(Y := \beta X + \gamma Z + U_Y\), with independent mean-zero noise, take \(\alpha = 2\), \(\gamma = 1\), \(\beta = 0.5\) and \(\operatorname{Var}(Z) = \operatorname{Var}(U_X) = 1\).

  1. Compute \(\operatorname{Var}(X)\), \(\operatorname{Cov}(X, Z)\) and the slope of \(Y\) on \(X\) alone.

  2. Repeat with \(\gamma = -1\).

  3. Repeat with \(\gamma = 1\) and \(\beta = 0\). What would someone with data on \(X\) and \(Y\) alone conclude?

  4. Is the error in (a) a bias or a variance? What happens to it as the sample grows?

\(Z\) and \(U_X\) are independent, so \(\operatorname{Var}(X) = \alpha^2 \operatorname{Var}(Z) + \operatorname{Var}(U_X)\).

  1. \(\operatorname{Var}(X) = 4 + 1 = 5\) and \(\operatorname{Cov}(X, Z) = \alpha \operatorname{Var}(Z) = 2\). The slope is \(\beta + \gamma \operatorname{Cov}(X, Z)/\operatorname{Var}(X) = 0.5 + 2/5 = 0.9\).

  2. \(0.5 - 2/5 = 0.1\): the bias has the sign of \(\gamma\alpha\), and here it almost hides the effect.

  3. \(0 + 2/5 = 0.4\). They would see a positive slope and could conclude that raising \(X\) raises \(Y\), when intervening on \(X\) does nothing.

  4. A bias. The slope converges to \(0.9\) as the sample grows: the estimates concentrate around \(0.9\), not around the effect, \(0.5\).

4. Choosing an adjustment set

  1. List the backdoor paths from \(X\) to \(Y\).

  2. Which of these sets satisfy the backdoor criterion: the empty set, \(\{Z\}\), \(\{M\}\), \(\{Z, M\}\), \(\{W\}\), \(\{Z, W\}\)?

  3. In a linear model, what does the coefficient on \(X\) estimate if you adjust for \(\{Z, M\}\)?

  4. If more than one of the sets in (b) satisfies the criterion, which would you use, and why?

A backdoor path starts with an arrow into \(X\). How many arrows point into \(X\)?

  1. One: \(X \leftarrow Z \to Y\).

  2. \(\{Z\}\) and \(\{Z, W\}\). The empty set and \(\{W\}\) leave the back door open. \(\{M\}\) and \(\{Z, M\}\) contain a descendant of \(X\).

  3. The direct effect of \(X\), along the arrow \(X \to Y\): the part that does not go through \(M\). It is not the total effect of \(X\). (It is the direct effect because in this graph nothing else affects both \(M\) and \(Y\).)

  4. \(\{Z\}\) and \(\{Z, W\}\) both satisfy it. Use \(\{Z, W\}\): \(W\) is not a confounder, so leaving it out causes no bias, but it explains some of the variation in \(Y\), and the estimate that uses it is usually less variable.

5. The graph behind an argument

A university says: “Students who use the library more get better grades. We will extend the library’s opening hours, because using the library improves grades.”

  1. Draw the causal graph the argument presupposes, and mark the action.

  2. Add one variable that would make the observed difference larger than the effect, with its arrows.

  3. What comparison would support the university’s claim?

The university’s evidence compares students. Its action changes opening hours. Are those the same thing?

  1. Opening hours (the action) \(\to\) library use \(\to\) grades. The argument needs the arrow from library use to grades, and an arrow from opening hours to library use.

  2. For example motivation, or how well a student was already doing, with arrows into library use and into grades. If it raises both, the students who use the library more would have had better grades anyway, so the observed difference is larger than the effect.

  3. Grades before and after the hours were extended, against a comparable university or set of courses where they were not; or terms, or sites, chosen at random to have longer hours. The comparison of students who use the library more with those who use it less answers a question about those students.

6. Hormone therapy

The notes’ section on hormone therapy and heart disease gives three studies of whether to prescribe hormone therapy to prevent heart disease.

  1. Write the decision with \(X\) and \(Y\). Say what the treatment and the outcome are, and write the quantity a doctor needs with \(do(\cdot)\).

  2. The 1996 estimate (0.39) and the 2008 estimates (1.42 over the first two years, 0.96 over the whole follow-up) come from the same cohort. Which comparison did each make?

  3. Draw a graph under which the 1996 estimate is confounded.

  4. Which comparison matches the decision a doctor makes, and why?

For (a), what does the doctor choose, and which outcome is the choice about? For (b), ask which two groups each analysis compares.

  1. \(X = 1\) if the doctor prescribes hormone therapy, \(Y = 1\) if the woman has coronary heart disease within five years. A doctor needs \(P(Y = 1 \mid do(X := 1))\) against \(P(Y = 1 \mid do(X := 0))\), not \(P(Y = 1 \mid X = 1)\) against \(P(Y = 1 \mid X = 0)\).

  2. 1996: women who took estrogen with progestin against women who did not use hormones, adjusted for measured confounders. 2008: women who started against women who did not start, both followed from the same point, analyzed as the trial was.

  3. Socioeconomic status \(\to\) hormone use, socioeconomic status \(\to\) heart disease, with or without an arrow from hormone use to heart disease.

  4. The 2008 comparison. The doctor’s decision is whether to start, and that comparison follows women from the point at which they started or did not.

Coding

Open a fresh R script. You will need the tidyverse.

A. Prediction and intervention

n <- 10000
# program 1: X causes Y
X1 <- rnorm(n); Y1 <- 2 * X1 + rnorm(n)
# program 2: Z causes both
Z2 <- rnorm(n); X2 <- Z2 + rnorm(n); Y2 <- Z2 + rnorm(n)
# program 3: X causes Y through M
X3 <- sqrt(2) * rnorm(n); M3 <- X3 + sqrt(2) * rnorm(n); Y3 <- 0.5 * M3 + rnorm(n)
  1. Write down the slope of \(Y\) on \(X\) you expect in each program. Then fit lm and compare.
  2. For each program, intervene: set \(X\) to 1 for everyone and run the statements after the one for \(X\) again, then the same with \(X\) set to 0. Compute the difference in the means of \(Y\).
  3. In program 2, fit lm(Y2 ~ X2 + Z2). In program 3, fit lm(Y3 ~ X3 + M3). Compare the coefficients on \(X\) with your answers to 2.

In each program, which statements after the one for \(X\) use \(X\), directly or through a variable computed from it? Those are the ones whose values change when you set \(X\).

slopes <- c(coef(lm(Y1 ~ X1))[2], coef(lm(Y2 ~ X2))[2], coef(lm(Y3 ~ X3))[2])
do1 <- c(mean(2 * 1 + rnorm(n)),                    # X set to 1, later statements re-run
         mean(Z2 + rnorm(n)),
         mean(0.5 * (1 + sqrt(2) * rnorm(n)) + rnorm(n)))
do0 <- c(mean(2 * 0 + rnorm(n)),                    # X set to 0
         mean(Z2 + rnorm(n)),
         mean(0.5 * (0 + sqrt(2) * rnorm(n)) + rnorm(n)))
round(rbind(slope = slopes, effect = do1 - do0), 2)
         X1    X2   X3
slope  1.99  0.51 0.50
effect 1.99 -0.01 0.54
round(coef(lm(Y2 ~ X2 + Z2))[2], 2)    # program 2, adjusting for Z
X2 
 0 
round(coef(lm(Y3 ~ X3 + M3))[2], 2)    # program 3, adjusting for M
  X3 
0.01

The slopes are about 2, 0.5 and 0.5. The effects are about 2, 0 and 0.5. In programs 2 and 3, \(X\) and \(Y\) have the same joint distribution (jointly normal, with means 0, variances 2 and 2 and covariance 1), so \(X\) predicts \(Y\) equally well in both, and intervening on \(X\) changes \(Y\) only in program 3. Adjusting for \(Z\) in program 2 gives a coefficient near 0, the effect. Adjusting for \(M\) in program 3 also gives a coefficient near 0. That is the direct effect, zero here because \(X\) reaches \(Y\) only through the mediator \(M\); the total effect is 0.5.

B. Adjustment on data

The table of problem 1, one row for each rider.

cells <- data.frame(Z = c(1, 1, 1, 1, 0, 0, 0, 0),
                    W = c(1, 1, 0, 0, 1, 1, 0, 0),
                    T = c(1, 0, 1, 0, 1, 0, 1, 0),
                    n = c(70, 30, 100, 100, 80, 120, 50, 450),
                    accidents = c(21, 12, 10, 15, 4, 18, 1, 18))
# In base R, T means TRUE; inside dplyr verbs, T is the column
riders <- bind_rows(
  cells |> mutate(Y = 1, count = accidents),
  cells |> mutate(Y = 0, count = n - accidents)
) |>
  select(Z, W, T, Y, count) |>
  tidyr::uncount(count)
nrow(riders)
[1] 1000
  1. Compute the share with an accident among riders who took the course and among those who did not, from riders, and the difference.
  2. Compute the adjusted difference: the difference within each combination of \(Z\) and \(W\), averaged with weights \(P(Z = z, W = w)\). Then adjust for \(Z\) alone, and for \(W\) alone.
  3. A randomized version: give each of the 1,000 riders the course with probability 0.3, whatever their group, and draw whether they have an accident from the table’s rates. Compute the difference. Repeat 200 times. What is the average difference, and how much does it vary?

For 2: riders |> group_by(Z, W) |> summarize(...) gives one row for each group of riders; compute the difference and the group’s share of all riders there.

riders |> group_by(T) |> summarize(rate = mean(Y))
# A tibble: 2 x 2
      T  rate
  <dbl> <dbl>
1     0  0.09
2     1  0.12
adjusted <- function(...) {
  rows <- riders |>
    group_by(...) |>
    summarize(diff = mean(Y[T == 1]) - mean(Y[T == 0]),
              share = n() / nrow(riders), .groups = "drop")
  sum(rows$diff * rows$share)
}
round(c(both = adjusted(Z, W), Z_only = adjusted(Z), W_only = adjusted(W)), 3)
  both Z_only W_only 
-0.050 -0.025 -0.001 
rates <- riders |> group_by(Z, W, T) |> summarize(rate = mean(Y), .groups = "drop")
groups <- cells |> group_by(Z, W) |> summarize(n = sum(n), .groups = "drop")
one_trial <- function() {
  trial <- groups |>
    tidyr::uncount(n) |>
    mutate(T = rbinom(n(), 1, 0.3)) |>
    left_join(rates, by = c("Z", "W", "T")) |>
    mutate(Y = rbinom(n(), 1, rate))
  mean(trial$Y[trial$T == 1]) - mean(trial$Y[trial$T == 0])
}
trials <- replicate(200, one_trial())
c(mean = mean(trials), sd = sd(trials))
       mean          sd 
-0.04698651  0.01790002

  1. 0.12 and 0.09, a difference of 0.03.

  2. \(-0.05\) with both. With \(Z\) alone about \(-0.025\), and with \(W\) alone about \(-0.001\): each leaves one back door open.

  3. The average is close to \(-0.05\), and the differences vary from trial to trial with a standard deviation of about 0.02. Randomization removes the bias without measuring \(Z\) or \(W\), and the price is that a single trial can miss by a few points.

C. Intervening in a program

The job center from the lecture, as a program. As in the lecture, everyone referred attends and nobody else does. Each line generates one variable from the ones above it.

n <- 10000
Z <- rbinom(n, 1, 0.5)                        # worked last year
X <- rbinom(n, 1, ifelse(Z == 1, 0.8, 0.2))   # attended the course
Y <- rbinom(n, 1, 0.3 + 0.1 * X + 0.3 * Z)    # in work six months later
mean(Y[X == 1]) - mean(Y[X == 0])             # attenders against the others
  1. Run it. Compare the difference with the lecture’s first job-center table (64% against 36%).
  2. Compute the adjusted difference: the difference between attenders and non-attenders within \(Z = 1\) and within \(Z = 0\), averaged with weights mean(Z == 1) and mean(Z == 0).
  3. Intervene. Replace the line for X by X <- rep(1, n), run the line for Y again, and record mean(Y). Do the same with X <- rep(0, n). Compare the difference with your answer to 2.
  4. The center changes its referrals to 0.95 for \(Z = 1\) and 0.05 for \(Z = 0\). Before you run anything, write down what you expect for the differences in 1 and in 3. Then run it.

For step 3, replace the line for X and leave the line for Z alone. Then run the line for Y again, so that it uses the new X.

set.seed(313)    # the sheet's seed, so this output does not depend on the chunks above it
n <- 10000
Z <- rbinom(n, 1, 0.5)
X <- rbinom(n, 1, ifelse(Z == 1, 0.8, 0.2))
Y <- rbinom(n, 1, 0.3 + 0.1 * X + 0.3 * Z)
observed <- mean(Y[X == 1]) - mean(Y[X == 0])
d1 <- mean(Y[X == 1 & Z == 1]) - mean(Y[X == 0 & Z == 1])   # within Z = 1
d0 <- mean(Y[X == 1 & Z == 0]) - mean(Y[X == 0 & Z == 0])   # within Z = 0
adjusted <- d1 * mean(Z == 1) + d0 * mean(Z == 0)
Y1 <- rbinom(n, 1, 0.3 + 0.1 * 1 + 0.3 * Z)    # everyone attends
Y0 <- rbinom(n, 1, 0.3 + 0.1 * 0 + 0.3 * Z)    # nobody attends
c(observed = observed, adjusted = adjusted, intervened = mean(Y1) - mean(Y0))
  observed   adjusted intervened 
 0.2813623  0.1046528  0.1067000 
X95 <- rbinom(n, 1, ifelse(Z == 1, 0.95, 0.05))
Y95 <- rbinom(n, 1, 0.3 + 0.1 * X95 + 0.3 * Z)
mean(Y95[X95 == 1]) - mean(Y95[X95 == 0])      # observed, new referrals
[1] 0.3798091

  1. About \(0.28\), the table’s \(0.64 - 0.36\).

  2. About \(0.10\).

  3. About \(0.10\): intervening deletes the arrow \(Z \to X\), and the adjusted difference estimates the same quantity.

  4. The observed difference grows to about \((0.95 \times 0.7 + 0.05 \times 0.4) - (0.05 \times 0.6 + 0.95 \times 0.3) = 0.685 - 0.315 = 0.37\). The effect of the course stays at \(0.10\): the referral policy changes who attends, not what the course does. Each run differs from these numbers by a few hundredths.