WEBVTT 1 00:00:08.200 --> 00:00:13.379 David Stephens, Prof: Okay, so I'm picking up where we left off before the break. 2 00:00:13.580 --> 00:00:16.570 David Stephens, Prof: So we're looking at these, methods. 3 00:00:16.720 --> 00:00:24.279 David Stephens, Prof: that we've looked at this morning, regression and inverse probability weighting base methods. We saw how… how they could be applied to the 4 00:00:24.760 --> 00:00:30.539 David Stephens, Prof: by a new treatment case, and we're just now looking at extensions to the continuous case. So, as we saw for… 5 00:00:31.200 --> 00:00:41.289 David Stephens, Prof: the G-estimation calculations. Basically, things proceed in the same way. You need to be a little more artful in the continuous treatment case, where multiple 6 00:00:41.480 --> 00:00:52.769 David Stephens, Prof: functions of Z could be included, so Z, Z squared, and so on, you need to be a little bit more careful in the way that you perform the blocking, but essentially the principles are the same, and we need to 7 00:00:52.950 --> 00:00:55.440 David Stephens, Prof: Include the right terms in the regression model. 8 00:00:55.810 --> 00:01:01.950 David Stephens, Prof: Turning to inverse probability weighting, again, the basic calculation is unchanged. 9 00:01:02.750 --> 00:01:06.169 David Stephens, Prof: We can still use the inverse probability weighting mechanism. 10 00:01:07.510 --> 00:01:10.920 David Stephens, Prof: Weighting now by the inverse probability density. 11 00:01:11.800 --> 00:01:18.529 David Stephens, Prof: to carry out estimation of the average potential outcome. So, in principle, We can still use this. 12 00:01:18.710 --> 00:01:19.990 David Stephens, Prof: Estimator. 13 00:01:21.150 --> 00:01:28.230 David Stephens, Prof: Exactly the same form as before, except that now this term in the denominator is a… 14 00:01:29.430 --> 00:01:33.009 David Stephens, Prof: Density, rather than a individual probability. 15 00:01:34.720 --> 00:01:40.129 David Stephens, Prof: We can always use… we can also use a standardized version of the weighting estimate as well, that stays, 16 00:01:40.390 --> 00:01:41.500 David Stephens, Prof: Stays the same. 17 00:01:43.860 --> 00:01:51.469 David Stephens, Prof: And the reason this works here, and the reason that it works in the original case, is that the expected value of the weight is 1. 18 00:01:53.340 --> 00:01:54.570 David Stephens, Prof: That is… 19 00:01:56.700 --> 00:02:05.349 David Stephens, Prof: This term in the numerator is either 1 or 0, depending on whether the observed Z is equal to the target Z. 20 00:02:06.470 --> 00:02:14.019 David Stephens, Prof: And in the denominator, this is the probability of, or the density of… for receiving that Z for given X. 21 00:02:15.060 --> 00:02:21.310 David Stephens, Prof: So, inverse probability weighting Works, in part, because of this identity. 22 00:02:22.700 --> 00:02:28.320 David Stephens, Prof: And this identity holds whether we're continuous or discrete. In practice, though. 23 00:02:29.540 --> 00:02:37.670 David Stephens, Prof: The challenge in applying this estimator In the continuous treatment case, Here's the indicator. 24 00:02:38.560 --> 00:02:42.530 David Stephens, Prof: Alright, so this indicator says if we have a target level Z, 25 00:02:45.600 --> 00:02:47.440 David Stephens, Prof: This indicator is 1. 26 00:02:47.760 --> 00:02:53.560 David Stephens, Prof: If and only if, The observed treatment is equal to that target. 27 00:02:55.590 --> 00:02:58.769 David Stephens, Prof: Now, in the binary case, that was fine, because we had… 28 00:02:59.500 --> 00:03:07.949 David Stephens, Prof: a collection of individuals who were treated and who were not treated. We had enough samples for each case in order to perform the estimation. 29 00:03:08.590 --> 00:03:16.680 David Stephens, Prof: In the continuous case, however, We might have some target Zs. 30 00:03:16.800 --> 00:03:19.549 David Stephens, Prof: that don't correspond to any observed Z. 31 00:03:19.880 --> 00:03:24.070 David Stephens, Prof: Can you imagine tracing out a dose response type function? 32 00:03:24.580 --> 00:03:30.670 David Stephens, Prof: different targeted levels of Z, we would need… Repeated observations. 33 00:03:31.030 --> 00:03:36.110 David Stephens, Prof: of the observed Z at each of those targeted Zs. 34 00:03:36.400 --> 00:03:41.820 David Stephens, Prof: And in most cases, we're only going to get one. If the Zs are arising from a continuous distribution. 35 00:03:42.640 --> 00:03:45.260 David Stephens, Prof: Then, there'll be one and only one 36 00:03:47.000 --> 00:03:56.350 David Stephens, Prof: data point for each observed Z, and there would be many potential target Zs That have no data points. 37 00:03:57.860 --> 00:04:05.030 David Stephens, Prof: So, although in principle this estimator works okay, we need to do a bit more work in order to 38 00:04:06.730 --> 00:04:09.100 David Stephens, Prof: To actually be able to implement it. 39 00:04:10.200 --> 00:04:13.309 David Stephens, Prof: And… the fix is an obvious one. 40 00:04:14.340 --> 00:04:17.689 David Stephens, Prof: Good approximation could be obtained by saying, well, instead of 41 00:04:17.950 --> 00:04:25.250 David Stephens, Prof: choosing Zs that only match the target Z, will take… 42 00:04:25.950 --> 00:04:34.910 David Stephens, Prof: those observed Zs that lie in some small interval surrounding the target Z. So we have a window outside of the target little red Z, 43 00:04:35.030 --> 00:04:39.989 David Stephens, Prof: And any observation that is made inside that window, we take that data point. 44 00:04:44.000 --> 00:04:50.290 David Stephens, Prof: So it's a pragmatic, Strategy, and it leads to… Good estimation. 45 00:04:50.730 --> 00:04:59.920 David Stephens, Prof: We can sometimes… we sometimes use the stabilized weight version of this, where we account for the probability of lying in that interval appropriately in the 46 00:05:00.090 --> 00:05:01.510 David Stephens, Prof: In the numerator. 47 00:05:02.010 --> 00:05:06.050 David Stephens, Prof: Okay, so the stabilized weight version can also be… Helpful. 48 00:05:10.480 --> 00:05:18.369 David Stephens, Prof: So there, that was applying it in the inverse weighting, ordinary inverse weighting case. We can equally well go ahead and apply that in the, 49 00:05:19.440 --> 00:05:24.600 David Stephens, Prof: In the, augmented, IPW case. 50 00:05:25.340 --> 00:05:31.459 David Stephens, Prof: Just by changing the weight in the same way the estimator is modifying the weight in this way here. 51 00:05:31.580 --> 00:05:35.480 David Stephens, Prof: And, using the same form of estimating. 52 00:05:37.050 --> 00:05:41.739 David Stephens, Prof: Estimator func… estimating function to lead to this estimator. 53 00:05:42.650 --> 00:05:48.429 David Stephens, Prof: And, of course, we can also do it using augmented outcome regression, with that modified form of the clever covariate. 54 00:05:49.520 --> 00:05:54.480 David Stephens, Prof: So, for both G-estimation and inverse probability weighting, modulo, a few small changes. 55 00:05:54.800 --> 00:05:57.010 David Stephens, Prof: The methods go across perfectly fine. 56 00:06:00.430 --> 00:06:04.310 David Stephens, Prof: The other approach I want to mention is that 57 00:06:04.510 --> 00:06:08.410 David Stephens, Prof: If we are happy to make mean model assumptions. 58 00:06:08.600 --> 00:06:11.650 David Stephens, Prof: Then we could use weightedly squares. 59 00:06:13.290 --> 00:06:15.279 David Stephens, Prof: to do estimation. 60 00:06:16.240 --> 00:06:27.379 David Stephens, Prof: So if we're interested in Psi 0, one way of getting at that would be to use the weightedly squares criterion, so minimize this with respect to the parameters, where the weights 61 00:06:28.220 --> 00:06:30.080 David Stephens, Prof: Are the same weights as before. 62 00:06:32.300 --> 00:06:45.159 David Stephens, Prof: Okay, so those weights that we just defined in the inverse probability weighting case, we can just apply those weights here and minimize this. So this is, again, a good form of, 63 00:06:45.680 --> 00:06:51.049 David Stephens, Prof: causal estimation. The role of the weights is still to break the confounding between X and Z. 64 00:06:53.220 --> 00:07:02.220 David Stephens, Prof: This type of approach, though, which can be useful, does rely on Correct specification of this. 65 00:07:03.030 --> 00:07:04.800 David Stephens, Prof: Function here. 66 00:07:06.380 --> 00:07:09.419 David Stephens, Prof: In particular, the correct treatment effect model. 67 00:07:13.790 --> 00:07:26.689 David Stephens, Prof: And then the idea here is that even if we had multiple functions of Z, Z and Z squared, log Z, or whatever, we could include that in this model function, apply the same weights, and that would perform the… 68 00:07:26.960 --> 00:07:27.840 David Stephens, Prof: Correct. 69 00:07:28.780 --> 00:07:29.700 David Stephens, Prof: Estimation. 70 00:07:32.010 --> 00:07:37.739 David Stephens, Prof: So I… I didn't include yet on the website the… 71 00:07:39.130 --> 00:07:46.469 David Stephens, Prof: web folder that, some examples of this. I've got some examples of this worked up, and I'll upload that there, and I'll mention it on the, 72 00:07:49.260 --> 00:07:50.680 David Stephens, Prof: Slack channel as well. 73 00:07:55.360 --> 00:08:03.090 David Stephens, Prof: For the continuous treatment case, the issue of positivity also comes up, so we still need to make sure that every target treatment level 74 00:08:03.230 --> 00:08:06.930 David Stephens, Prof: It's obtainable for each individual patient. 75 00:08:08.870 --> 00:08:10.390 David Stephens, Prof: We can't have that… 76 00:08:10.530 --> 00:08:18.219 David Stephens, Prof: Therefore, X predicts Z too precisely. We can't have that X basically deterministically decides what Z is. 77 00:08:22.610 --> 00:08:28.220 David Stephens, Prof: And one way of checking that is to make sure that the weights are On average, near 1. 78 00:08:28.330 --> 00:08:31.889 David Stephens, Prof: Right? You shouldn't have extreme weights that give 79 00:08:33.470 --> 00:08:39.349 David Stephens, Prof: Exceptional predominance to the, to individual data points. 80 00:08:42.220 --> 00:08:46.370 David Stephens, Prof: Alright, so… We've now seen the… 81 00:08:47.460 --> 00:08:53.369 David Stephens, Prof: the core methods that rely on the Propensity score. Essentially, they break down into four classes. 82 00:08:53.710 --> 00:08:54.670 David Stephens, Prof: Matching? 83 00:08:55.030 --> 00:08:56.319 David Stephens, Prof: It's notification. 84 00:08:57.140 --> 00:08:58.140 David Stephens, Prof: Regression. 85 00:08:58.480 --> 00:09:00.109 David Stephens, Prof: Inverse probability weighting. 86 00:09:02.550 --> 00:09:07.119 David Stephens, Prof: And we've seen that there are different assumptions that go with each. Some are… 87 00:09:07.440 --> 00:09:09.590 David Stephens, Prof: Have fewer assumptions, some have more. 88 00:09:10.630 --> 00:09:14.970 David Stephens, Prof: We'll see in the upcoming practical studies the 89 00:09:15.400 --> 00:09:22.020 David Stephens, Prof: Benefits and downsides of, Of some of the methods. 90 00:09:26.460 --> 00:09:36.579 David Stephens, Prof: what I'll emphasize when we look at the real data analysis is that, we are… we are in a position and have an obligation to check assumptions when we make them. 91 00:09:36.780 --> 00:09:41.069 David Stephens, Prof: And, for some of the assumptions that we've made, actually. 92 00:09:41.240 --> 00:09:43.179 David Stephens, Prof: They are checkable in the data. 93 00:09:45.200 --> 00:09:53.760 David Stephens, Prof: So the next, short section is on implementation, and here I'm just gonna remind you of the key methods that are being used. 94 00:09:54.730 --> 00:09:58.130 David Stephens, Prof: So… As we've seen… 95 00:10:00.480 --> 00:10:09.319 David Stephens, Prof: There's not really too much… once we've seen the basic mechanisms for adjustment, for confounding, removal, and so on, for blocking, and 96 00:10:09.670 --> 00:10:11.830 David Stephens, Prof: Based on propensity scores and so on. 97 00:10:12.230 --> 00:10:16.450 David Stephens, Prof: The statistical tools that are used are really fairly standard. 98 00:10:17.130 --> 00:10:18.090 David Stephens, Prof: we make. 99 00:10:18.270 --> 00:10:21.480 David Stephens, Prof: We can make parametric assumptions about distribution, so… 100 00:10:22.030 --> 00:10:25.999 David Stephens, Prof: Normal, or binomial, or survival-type distributions. 101 00:10:26.470 --> 00:10:33.740 David Stephens, Prof: We use… Standard tools like regression, or… Generalized linear modeling. 102 00:10:34.380 --> 00:10:37.869 David Stephens, Prof: Survival modeling, if we have a survival outcome. 103 00:10:42.490 --> 00:10:44.859 David Stephens, Prof: The kinds of modeling that we have to do. 104 00:10:45.830 --> 00:10:49.800 David Stephens, Prof: Are basically focusing on the, at least in 105 00:10:50.340 --> 00:10:53.090 David Stephens, Prof: methods that we've seen. The outcome. 106 00:10:53.200 --> 00:10:56.170 David Stephens, Prof: Progression model? Or some proxy for it? 107 00:10:56.400 --> 00:10:58.689 David Stephens, Prof: And the Propensity or balancing model. 108 00:10:59.750 --> 00:11:05.160 David Stephens, Prof: So we have to propose, typically, a model for the outcome for Y given X and Z. 109 00:11:05.810 --> 00:11:13.250 David Stephens, Prof: a model for Z given X, either through its probability, or through its expectation. 110 00:11:16.120 --> 00:11:27.280 David Stephens, Prof: And when we've… when I've been referring to this up to this point, I've basically been talking about linear regression, or generalized linear modeling, logistic regression, and so on. 111 00:11:27.640 --> 00:11:34.790 David Stephens, Prof: But actually, there's a developing literature on more flexible models for this kind of component of the model. 112 00:11:37.730 --> 00:11:41.429 David Stephens, Prof: And there's a developing literature that you use 113 00:11:42.050 --> 00:11:45.690 David Stephens, Prof: Splines, or machine learning methods, generally. 114 00:11:46.040 --> 00:11:52.630 David Stephens, Prof: Deep neural networks and so on. Regression trees. CART-type, analyses. 115 00:11:53.830 --> 00:12:01.040 David Stephens, Prof: Either in the frequentist or the Bayesian world. Random forests, causal forests, becoming very popular. 116 00:12:01.360 --> 00:12:05.920 David Stephens, Prof: Or combinations of those methods through, model averaging type. 117 00:12:06.150 --> 00:12:09.339 David Stephens, Prof: approaches, or ensemble approaches. 118 00:12:10.120 --> 00:12:18.340 David Stephens, Prof: And… What people have been trying to do is to use these more sophisticated methods, To estimate these components. 119 00:12:19.030 --> 00:12:25.899 David Stephens, Prof: And indeed, sometimes other components of the model as well, like the treatment-free component, which is basically a part of this. 120 00:12:26.100 --> 00:12:28.030 David Stephens, Prof: As well as the propensity score. 121 00:12:29.170 --> 00:12:36.240 David Stephens, Prof: And… the… The methods can be effective. 122 00:12:36.670 --> 00:12:44.410 David Stephens, Prof: No doubt about that. And… There is a lot of ongoing research into… 123 00:12:44.550 --> 00:12:59.490 David Stephens, Prof: getting some theoretical guarantees about the performance of these more advanced methods. Of course, the complication in all of these cases, really, all of these advanced cases, is that we're using highly parameterized models, very, very flexible models. 124 00:13:01.740 --> 00:13:07.090 David Stephens, Prof: So what we want to make sure is that those flexible models are flexible enough 125 00:13:07.490 --> 00:13:09.779 David Stephens, Prof: To capture the true structure. 126 00:13:11.140 --> 00:13:16.830 David Stephens, Prof: But also not too flexible, that they require too much data to estimate them. 127 00:13:18.140 --> 00:13:21.570 David Stephens, Prof: And once you start working with 128 00:13:21.890 --> 00:13:26.359 David Stephens, Prof: Let's say that… let's take an example of the tree basement. It's a random forest, for example. 129 00:13:27.480 --> 00:13:29.259 David Stephens, Prof: The danger is that 130 00:13:29.440 --> 00:13:36.589 David Stephens, Prof: Those methods are so flexible that we need a massive dataset in order to be able to estimate them appropriately. 131 00:13:37.340 --> 00:13:44.870 David Stephens, Prof: And indeed, there are some theoretical issues with using models like Roundup Force, and indeed some of these other machine learning methods, in that 132 00:13:45.150 --> 00:13:46.090 David Stephens, Prof: we… 133 00:13:46.650 --> 00:13:58.240 David Stephens, Prof: can struggle to get the theoretical guarantees of consistent estimation and so on, that we would be able to obtain routinely using ordinary parametric models. 134 00:14:00.130 --> 00:14:09.870 David Stephens, Prof: So we've got this… Dilemma, really, which says that, if the… if the data-generated model is not simple. 135 00:14:11.990 --> 00:14:15.320 David Stephens, Prof: Simple regression-type models won't work. 136 00:14:15.800 --> 00:14:23.860 David Stephens, Prof: Whereas if… The model is, the degenerated model is too complex, and we try to use more 137 00:14:24.020 --> 00:14:32.390 David Stephens, Prof: Highly flexible methods, then… We don't have enough data to estimate those components correctly. 138 00:14:33.580 --> 00:14:40.130 David Stephens, Prof: So, there's ongoing theoretical work been going on, I guess, for the last 10 years or so, trying to figure out 139 00:14:40.360 --> 00:14:46.240 David Stephens, Prof: What we can get away with in terms of these… the use of these flexible modeling approaches. 140 00:14:46.820 --> 00:14:51.699 David Stephens, Prof: Okay, I've got… A couple of references in the reference list referring to this. 141 00:14:51.990 --> 00:14:58.670 David Stephens, Prof: There's this, popular piece of work that… 142 00:14:58.900 --> 00:15:01.000 David Stephens, Prof: came out in the econometrics literature. 143 00:15:01.520 --> 00:15:04.389 David Stephens, Prof: called, Double or… 144 00:15:04.490 --> 00:15:10.669 David Stephens, Prof: Or double and de-biased machine learning for treatment of structural parameters, which is basically causal estimation as we've seen it. 145 00:15:11.810 --> 00:15:16.199 David Stephens, Prof: And this paper lays out, in a very sophisticated way. 146 00:15:16.370 --> 00:15:23.940 David Stephens, Prof: The kinds of considerations that we need to put in place in order to get flexible methods. 147 00:15:24.450 --> 00:15:29.350 David Stephens, Prof: Including machine learning type methods, that give us the correct answer. 148 00:15:30.920 --> 00:15:38.940 David Stephens, Prof: Now, I would… so this… Work is, implementable… implementable now in several packages. 149 00:15:39.440 --> 00:15:46.250 David Stephens, Prof: And can be constructed from scratch using fairly, fairly simple R coding. 150 00:15:46.430 --> 00:15:54.890 David Stephens, Prof: But I would assess that the theory for… the theory supporting these more flexible methods is still not fully 151 00:15:55.510 --> 00:15:56.780 David Stephens, Prof: defined yet. 152 00:15:57.240 --> 00:16:00.019 David Stephens, Prof: So, although this is certainly a good start. 153 00:16:00.540 --> 00:16:03.990 David Stephens, Prof: The story is not complete yet, I would say. 154 00:16:04.810 --> 00:16:16.070 David Stephens, Prof: So that's not to say that these methods shouldn't be used in constructing things like the propensity score, things like the treatment-free model, and so on, it's just that you need to take care to check 155 00:16:16.940 --> 00:16:19.720 David Stephens, Prof: In the ways that we'll see in the practical analysis. 156 00:16:19.820 --> 00:16:21.810 David Stephens, Prof: That the methods are actually working. 157 00:16:28.050 --> 00:16:33.369 David Stephens, Prof: So the disadvantages of using flexible models like that, and why I would caution 158 00:16:34.000 --> 00:16:36.479 David Stephens, Prof: Anybody trying to use them in 159 00:16:36.690 --> 00:16:42.660 David Stephens, Prof: this sort of causal setting, is that… well, some of the reasons are that 160 00:16:45.010 --> 00:16:51.969 David Stephens, Prof: One has to be careful about quantifying uncertainty, so the role of any… the role of a statistician 161 00:16:52.110 --> 00:17:02.149 David Stephens, Prof: Or epidemiologists in this kind of analysis is to provide not only point estimates, but also uncertainty representation, uncertainty estimates, confidence interval, credible intervals. 162 00:17:02.470 --> 00:17:07.139 David Stephens, Prof: And… For these more advanced methods, this is not straightforward. 163 00:17:08.530 --> 00:17:15.699 David Stephens, Prof: You can use computational tools like the Bootstrap, but when you're fitting one of these massive models. 164 00:17:15.810 --> 00:17:20.339 David Stephens, Prof: Using the bootstrap is not, not very palatable. 165 00:17:23.579 --> 00:17:37.680 David Stephens, Prof: particularly to do with the propensity score, there's also a danger of positivity violations occurring. Remember, that means that, so you have a positivity violation if the relationship between Z and X is… 166 00:17:38.310 --> 00:17:41.069 David Stephens, Prof: Perceived to be deterministic, essentially. 167 00:17:41.250 --> 00:17:46.110 David Stephens, Prof: That is… X predicts Z precisely. 168 00:17:46.750 --> 00:17:51.430 David Stephens, Prof: So when fitting these more sophisticated, or these highly complex models. 169 00:17:51.840 --> 00:17:55.380 David Stephens, Prof: The issue of positivity violations can come up. 170 00:17:56.860 --> 00:18:01.599 David Stephens, Prof: And what we have to remember is that prediction of the treatment is not the fundamental goal. 171 00:18:01.700 --> 00:18:02.500 David Stephens, Prof: Right. 172 00:18:02.640 --> 00:18:10.630 David Stephens, Prof: Causal estimation is the goal, and what we need is a… confounding braking mechanism. 173 00:18:12.400 --> 00:18:13.720 David Stephens, Prof: To enable that. 174 00:18:18.140 --> 00:18:26.049 David Stephens, Prof: So, these prediction methods, these machine learning methods, may become very good at predicting the treatment on the basis of the 175 00:18:26.980 --> 00:18:30.970 David Stephens, Prof: predictors, But that's not the goal of the exercise. 176 00:18:32.480 --> 00:18:40.570 David Stephens, Prof: And what some methods have tried to do is to say, well, rather than predicting Z given X, try to come up with a balancing score. 177 00:18:40.900 --> 00:18:45.209 David Stephens, Prof: So… What we saw was that, 178 00:18:47.910 --> 00:18:50.680 David Stephens, Prof: The Propensity score itself is a balancing score. 179 00:18:50.950 --> 00:18:53.140 David Stephens, Prof: And in many ways, it's the optimal one. 180 00:18:54.020 --> 00:18:54.890 David Stephens, Prof: But… 181 00:18:55.180 --> 00:19:03.760 David Stephens, Prof: you need correct specification to do that, so maybe what you should be doing instead is coming up with a score that guarantees that you have balance. 182 00:19:05.880 --> 00:19:09.790 David Stephens, Prof: Because that would be good enough. It may not be optimal, but it will be good enough. 183 00:19:10.500 --> 00:19:14.389 David Stephens, Prof: To get consistent estimation of the causal parameters. 184 00:19:15.810 --> 00:19:18.710 David Stephens, Prof: Okay? So there are some methods out there that 185 00:19:19.030 --> 00:19:24.709 David Stephens, Prof: Look at balance rather than prediction of the treatment, and some of those methods can do… can do… can do quite well. 186 00:19:26.990 --> 00:19:31.920 David Stephens, Prof: One has to be careful about the theoretical guarantees that, 187 00:19:33.180 --> 00:19:46.040 David Stephens, Prof: one can obtain when using these flexible methods. So, in ordinary parametric models, the calculations are easy to perform, and you can get fairly, in a fairly simple fashion using standard methods, 188 00:19:46.480 --> 00:19:54.619 David Stephens, Prof: guarantees of performance. But once the model becomes more and more flexible, and you're using a much more large model. 189 00:19:54.830 --> 00:19:59.269 David Stephens, Prof: potentially when the dimension of X is… is high, 190 00:20:00.080 --> 00:20:03.620 David Stephens, Prof: You lose the capability of getting theoretical guarantees. 191 00:20:04.840 --> 00:20:14.130 David Stephens, Prof: So there are… some, methods, some theoretical, some computational methods that have been proposed to get around this. 192 00:20:14.640 --> 00:20:23.609 David Stephens, Prof: And in that paper, that Churner-Zhukov paper that I mentioned, there is the sample splitting and cross-fitting strategy that they've used there, which, 193 00:20:23.960 --> 00:20:27.980 David Stephens, Prof: Can solve some of the problems, but it doesn't, 194 00:20:28.420 --> 00:20:33.590 David Stephens, Prof: it, again, comes with some computational burden. It's not easy to implement. 195 00:20:33.790 --> 00:20:34.940 David Stephens, Prof: Necessarily. 196 00:20:38.360 --> 00:20:44.730 David Stephens, Prof: And I'll just give a… so this paper is a paper we published in Statistics of Medicine in 2019, 197 00:20:45.010 --> 00:20:56.350 David Stephens, Prof: Not really addressing the machine learning strategies, but some of those other flexible methods, so boosting, the method called Super Learner, for example. 198 00:20:56.450 --> 00:21:00.940 David Stephens, Prof: Hence the title here. So the method called Super Learner, was one of the original proposed 199 00:21:01.250 --> 00:21:09.450 David Stephens, Prof: machine learning, or actually ensemble methods for performing propensity score estimation. And what we show in this paper is that 200 00:21:09.810 --> 00:21:11.969 David Stephens, Prof: It's good performance. 201 00:21:12.210 --> 00:21:14.039 David Stephens, Prof: is not guaranteed. 202 00:21:15.970 --> 00:21:21.590 David Stephens, Prof: So I've included this on the list of papers that I would recommend people have a look at. 203 00:21:22.030 --> 00:21:24.019 David Stephens, Prof: Sorry for the self-citation there. 204 00:21:24.900 --> 00:21:27.280 David Stephens, Prof: Alright, so… 205 00:21:27.470 --> 00:21:38.090 David Stephens, Prof: So this, this paper is a bit old, but… so there's ongoing work to go beyond super learner to think about, other flexible, methods. 206 00:21:40.850 --> 00:21:45.400 David Stephens, Prof: So it turned into variance estimation, then, because as I mentioned, the, 207 00:21:46.800 --> 00:21:53.410 David Stephens, Prof: The role of the statistician is to provide not only point estimates of course or the facts, but also uncertainty estimates. 208 00:21:54.060 --> 00:21:59.630 David Stephens, Prof: So… As we saw, most of the methods 209 00:21:59.740 --> 00:22:05.979 David Stephens, Prof: that I, have described, well, several of the methods that I described, rely on 210 00:22:06.600 --> 00:22:20.829 David Stephens, Prof: Estimating equations, and we have a good statistical framework for thinking about variance estimation for estimating equations, which is, basically the sandwich estimator construction. 211 00:22:20.940 --> 00:22:26.149 David Stephens, Prof: So the sandwich estimator, or robust, Variance estimation procedure. 212 00:22:26.520 --> 00:22:36.070 David Stephens, Prof: Gives us a way of relaxing modeling assumptions, allows us to do, essentially, semi-parametric, or… Quasi-likelihood type. 213 00:22:36.250 --> 00:22:38.960 David Stephens, Prof: variance, representation. 214 00:22:39.750 --> 00:22:51.969 David Stephens, Prof: So… Because most of the methods rely on the semi-parametric or generalized estimating equation type, logic. 215 00:22:52.190 --> 00:22:59.949 David Stephens, Prof: We can use the sandwich estimator, so the sandwich package in R gives us, this sort of robust estimation of 216 00:23:00.260 --> 00:23:04.450 David Stephens, Prof: Standard errors and so on. 217 00:23:10.030 --> 00:23:15.359 David Stephens, Prof: So any method that's developed from a estimating equation perspective. 218 00:23:15.630 --> 00:23:21.600 David Stephens, Prof: Can use this package to get… Robust, perhaps better. 219 00:23:21.710 --> 00:23:27.969 David Stephens, Prof: estimation of the estimator variances. Okay, and that allows us to report confidence intervals and so on. 220 00:23:29.580 --> 00:23:35.009 David Stephens, Prof: There are some, packages that have been built That have, 221 00:23:35.350 --> 00:23:42.769 David Stephens, Prof: specific robust variance estimation, so the DRGEE package really implements doubly robust G-estimation. 222 00:23:43.590 --> 00:23:49.239 David Stephens, Prof: So we saw that in passing, and this package has got built-in robust variance estimation in it. 223 00:23:49.390 --> 00:23:51.680 David Stephens, Prof: And… 224 00:23:52.710 --> 00:23:58.070 David Stephens, Prof: What it does in part is to make sure that we correctly account for the nuisance parameters in the fit. 225 00:23:59.710 --> 00:24:14.090 David Stephens, Prof: Okay? It turns out that, actually, although we've viewed this as a two-stage estimation, so we've estimated parameters in the propensity model and in the outcome regression model, let's say, the Mead model. 226 00:24:16.530 --> 00:24:25.240 David Stephens, Prof: And you might think, therefore, that we need to reflect uncertainty and estimation in both components. It turns out, actually, that in these models, we, 227 00:24:25.490 --> 00:24:32.499 David Stephens, Prof: Can effectively disregard any uncertainty in the estimation of the propensity score parameters. 228 00:24:33.470 --> 00:24:39.890 David Stephens, Prof: Due to the structure of the model. And I'll talk a little bit about that in a second when I show this sandwich variance calculation. 229 00:24:40.510 --> 00:24:52.970 David Stephens, Prof: So, At least for estimating equation-based procedures, obtaining variance estimates and uncertainty representation is fairly straightforward. 230 00:24:53.200 --> 00:24:57.729 David Stephens, Prof: Adapting that to the… The more flexible setting. 231 00:24:58.780 --> 00:25:01.329 David Stephens, Prof: On the surface, looks a bit more complicated. 232 00:25:03.270 --> 00:25:11.270 David Stephens, Prof: But I think there are also results that tell us that this result that says estimating the propensity score, 233 00:25:11.460 --> 00:25:13.480 David Stephens, Prof: Estimating the Propensity score. 234 00:25:13.620 --> 00:25:20.150 David Stephens, Prof: And ignoring the variability of the estimation there, and only focusing on the variability in the outcome component. 235 00:25:20.760 --> 00:25:22.280 David Stephens, Prof: That will hold. 236 00:25:22.880 --> 00:25:25.980 David Stephens, Prof: Even if we use these more advanced methods. 237 00:25:26.290 --> 00:25:27.870 David Stephens, Prof: Okay, so there is some hope. 238 00:25:28.390 --> 00:25:31.670 David Stephens, Prof: Of course, in any… 239 00:25:32.130 --> 00:25:41.299 David Stephens, Prof: variance estimation situation, we've also got access to the bootstrap. And that's easy to apply for regression and IPW approaches. 240 00:25:42.150 --> 00:25:50.440 David Stephens, Prof: A bit more computationally challenging for advanced methods, so for machine learning, Random forest-type methods. 241 00:25:50.760 --> 00:25:56.159 David Stephens, Prof: And also not so suitable for, matching-type procedures. 242 00:26:01.070 --> 00:26:12.769 David Stephens, Prof: So, thinking about estimating equations, then just quickly go through the basic idea of sandwich estimation, because I've got some of the calculations laid out in the simulation example that I've got. 243 00:26:12.940 --> 00:26:21.379 David Stephens, Prof: So, for semi-parametric models based on estimating equations, we have a well-developed theory 244 00:26:21.720 --> 00:26:35.000 David Stephens, Prof: about the performance of the estimator, in particular, consistency and asymptotic normality, some statements on efficiency, that is, that they're the best estimators, and so on. And, 245 00:26:36.470 --> 00:26:39.550 David Stephens, Prof: Quickly remind us of the logic here. 246 00:26:39.670 --> 00:26:47.640 David Stephens, Prof: So, an estimating function u is a function that's used to estimate the parameter, generic parameter, theta. 247 00:26:48.600 --> 00:27:04.420 David Stephens, Prof: Through this form of, equation solution. So we say that U is the estimating function, it takes the data as inputs as a function of the parameter theta, and we end up sol… in order to estimate theta, we solve this equation. 248 00:27:04.950 --> 00:27:09.310 David Stephens, Prof: Here, I've written it in terms of the random variable, so that returns for us the estimator. 249 00:27:11.160 --> 00:27:21.500 David Stephens, Prof: The results of the semi-parametic theory tell us that such an estimator, under mild conditions, so if the expected value of U is 0, 250 00:27:22.590 --> 00:27:28.050 David Stephens, Prof: Taken with respect to the… Overall true data generating mechanism. 251 00:27:28.910 --> 00:27:36.770 David Stephens, Prof: Then solutions to this equation will be consistent and asymptotically normally distributed under particular conditions. 252 00:27:38.150 --> 00:27:45.170 David Stephens, Prof: And the variance of those estimators can be rewritten in this way, in terms of mu. 253 00:27:45.590 --> 00:27:52.680 David Stephens, Prof: In terms of these A and B matrices, so it's A inverse B, A inverse transpose. 254 00:27:53.080 --> 00:28:00.089 David Stephens, Prof: Where A inverse is minus the first derivative of U, evaluated at the true theta zero. 255 00:28:00.790 --> 00:28:03.699 David Stephens, Prof: And B is the outer product of the U's. 256 00:28:04.880 --> 00:28:13.350 David Stephens, Prof: So, this is the… essentially the standard theory of estimating equations, mimicking what goes on in the maximum likelihood. 257 00:28:13.530 --> 00:28:14.620 David Stephens, Prof: setting. 258 00:28:14.910 --> 00:28:18.230 David Stephens, Prof: So U dot is a matrix if, if, 259 00:28:18.630 --> 00:28:25.220 David Stephens, Prof: theta is a p-dimensional parameter, then this is a P byp matrix, this is a P byp matrix, and so on. 260 00:28:25.330 --> 00:28:29.370 David Stephens, Prof: So the sandwich… the sandwich package automatically 261 00:28:29.720 --> 00:28:35.479 David Stephens, Prof: computes these things for you, for any estimating equations, so your job is really only to 262 00:28:35.950 --> 00:28:39.369 David Stephens, Prof: Let… write down what the estimating function is. 263 00:28:41.140 --> 00:28:47.759 David Stephens, Prof: Theta 0 is the true degenerating parameter. U dot is the matrix of first partial derivatives of U. 264 00:28:48.700 --> 00:28:53.790 David Stephens, Prof: In this form, in general, A is not a symmetric matrix. 265 00:28:55.820 --> 00:29:01.670 David Stephens, Prof: In the case of parametric models, it is symmetric, but in… and therefore that leads you to the… 266 00:29:01.840 --> 00:29:07.790 David Stephens, Prof: to the, Sort of standard form of, sandwich estimator. 267 00:29:08.310 --> 00:29:09.839 David Stephens, Prof: variants. 268 00:29:14.500 --> 00:29:24.919 David Stephens, Prof: A and B, of course, we don't have access to theta zero, so are not computable, but these are things that are easily estimated from the data using the relevant sample-based averages. 269 00:29:25.310 --> 00:29:27.109 David Stephens, Prof: So this is the approach that… 270 00:29:28.230 --> 00:29:35.690 David Stephens, Prof: It's used in estimating equations, and most of our methods are best in estimating equations, so this is an approach we can use to get at that variance. 271 00:29:36.460 --> 00:29:44.779 David Stephens, Prof: If we have linear regression, this is how the calculations work. So this is the so-called robust version of the sandwich variance. 272 00:29:45.840 --> 00:29:53.240 David Stephens, Prof: sorry, the robust version of the variance, or sandwich variance, for the linear regression case. If we're estimating beta. 273 00:29:53.390 --> 00:30:03.989 David Stephens, Prof: then the typical estimating equation corresponding to ordinary least squares looks like this. U dot looks like this, and therefore A and B look like this, and the variance looks like this. 274 00:30:05.750 --> 00:30:20.249 David Stephens, Prof: Okay, so in some of the examples I'm going to show, I've actually gone through these calculations for the propensity score-based methods. Okay, the standard off-the-shelf stuff, but I decided to write it out to give some insight into those calculations. 275 00:30:24.080 --> 00:30:29.570 David Stephens, Prof: So the practical considerations that we're left with, then, are that, first of all, there's no measured confounders. 276 00:30:30.610 --> 00:30:31.310 David Stephens, Prof: Okay? 277 00:30:31.740 --> 00:30:36.899 David Stephens, Prof: As I mentioned yesterday, all of the causal adjustment procedures that we've discussed are 278 00:30:37.060 --> 00:30:47.180 David Stephens, Prof: predicated on the assumption of no measured confounders, that is, that all the confounders X in the system are observed and available to us for modeling. 279 00:30:52.290 --> 00:31:02.749 David Stephens, Prof: Now, as a statistician, there's nothing we can do about this. This is… this becomes a studied design issue, right? You need to make sure that you do measure all of these confounders. 280 00:31:03.290 --> 00:31:09.770 David Stephens, Prof: And then carry forward with the analysis on the assumption that you've done a good job in collecting those. 281 00:31:13.420 --> 00:31:19.269 David Stephens, Prof: So we've got two models to construct, basically, model construction for the outcome regression. 282 00:31:19.830 --> 00:31:29.680 David Stephens, Prof: So, we can get away with regression, provided we can guarantee correct specifications, so maybe through flexible modeling, we'll be able to do that. Maybe if we build a… 283 00:31:29.790 --> 00:31:38.360 David Stephens, Prof: complex enough regression model, we can correctly capture the relationship between YX and Z. That's a strategy that people have used in causal estimation, for sure. 284 00:31:38.690 --> 00:31:46.249 David Stephens, Prof: You're putting all your eggs in the basket of having correct specification, though. 285 00:31:46.560 --> 00:31:53.459 David Stephens, Prof: So… You need to do as much as you can to check that assumption. 286 00:31:54.800 --> 00:31:58.849 David Stephens, Prof: If you can do that, though, no causal adjustments are necessary. 287 00:31:59.290 --> 00:32:03.370 David Stephens, Prof: In passing there, you can think about… 288 00:32:03.560 --> 00:32:11.120 David Stephens, Prof: variable selection approaches, so you can decide which X's using formal methods like AIC, BIC, or whatever. 289 00:32:16.340 --> 00:32:23.730 David Stephens, Prof: And the critical thing about that is that if you can correctly select all of those variables that predict Y, 290 00:32:24.190 --> 00:32:29.459 David Stephens, Prof: Then you know you've got the collection of confounders in that set. 291 00:32:30.800 --> 00:32:35.679 David Stephens, Prof: Because to be a confounder, a variable has to be a predictor of Y. 292 00:32:37.040 --> 00:32:40.479 David Stephens, Prof: That's not to say that all predictors of Y are confounders. 293 00:32:41.020 --> 00:32:46.580 David Stephens, Prof: But you certainly know that… If something is a confander. 294 00:32:46.700 --> 00:32:48.629 David Stephens, Prof: It must be a predictor of Y. 295 00:32:49.740 --> 00:32:53.450 David Stephens, Prof: So you can do the confounder selection using those methods. 296 00:32:54.420 --> 00:32:55.090 David Stephens, Prof: Okay. 297 00:32:55.460 --> 00:33:00.679 David Stephens, Prof: Now, the challenge with that, though, is that you might omit some weak confounders. 298 00:33:00.960 --> 00:33:04.609 David Stephens, Prof: And therefore, leaving yourself open to… to bias. 299 00:33:06.370 --> 00:33:14.740 David Stephens, Prof: For the Propensity score, however, the story's a little bit different. The Propensity score… approach would encourage us to 300 00:33:14.900 --> 00:33:23.480 David Stephens, Prof: try to model the treatment assignment mechanism of Z given X. 301 00:33:26.120 --> 00:33:37.069 David Stephens, Prof: But, in this case, in this calculation, We have to remember that simply predicting Z, Is not the goal. 302 00:33:37.350 --> 00:33:40.349 David Stephens, Prof: Our goal is to select the confounders. 303 00:33:41.330 --> 00:33:52.230 David Stephens, Prof: We only need to adjust for the confounders. In particular, we can ignore instrumental variables. So instrumental variables, if you remember, are those variables that predict Z only. 304 00:33:52.380 --> 00:33:53.909 David Stephens, Prof: They don't predict why. 305 00:33:54.880 --> 00:34:02.679 David Stephens, Prof: And all of our calculations have told us that we do not need to adjust for instruments when constructing the propensity score. 306 00:34:02.820 --> 00:34:04.110 David Stephens, Prof: We can ignore them. 307 00:34:07.740 --> 00:34:12.309 David Stephens, Prof: The importance of Choosing the confounders is that… 308 00:34:17.300 --> 00:34:24.040 David Stephens, Prof: By conditioning on The Propensity score, constructed from the confounders, we will induce balance. 309 00:34:25.460 --> 00:34:31.870 David Stephens, Prof: So… If we include instruments in the… Propensity score. 310 00:34:33.050 --> 00:34:35.340 David Stephens, Prof: Through model selection methods or whatever. 311 00:34:35.679 --> 00:34:41.300 David Stephens, Prof: then… We will retain balance on those variables. 312 00:34:41.820 --> 00:34:46.400 David Stephens, Prof: But it won't be any use to us. It has no impact on bias of the estimator. 313 00:34:48.060 --> 00:34:52.909 David Stephens, Prof: So I would argue that for propensity score selection. 314 00:34:54.230 --> 00:34:57.760 David Stephens, Prof: You shouldn't really be thinking about using model selection. 315 00:34:57.900 --> 00:34:58.970 David Stephens, Prof: methods. 316 00:35:03.040 --> 00:35:07.789 David Stephens, Prof: If you have a strong predictor of treatment, That's an instrument. 317 00:35:08.440 --> 00:35:15.460 David Stephens, Prof: Your model selection method will select that as being something to go into the Propensity score model, but it's not going to help you. 318 00:35:17.720 --> 00:35:21.869 David Stephens, Prof: And of course, that's a little bit different from the outcome model. 319 00:35:21.980 --> 00:35:25.569 David Stephens, Prof: Because a strong predictor of outcome will always be useful. 320 00:35:28.480 --> 00:35:31.430 David Stephens, Prof: In the outcome model, because it's going to reduce variance. 321 00:35:34.720 --> 00:35:41.390 David Stephens, Prof: So including strong predictors of treatment that are not predictors of outcome, that's really no use to you. 322 00:35:44.090 --> 00:35:49.640 David Stephens, Prof: So that means you have to do the model selection with… with that… actual goal in mind. 323 00:35:51.130 --> 00:35:54.210 David Stephens, Prof: And once you've done the construction of the propensity score. 324 00:35:55.140 --> 00:36:06.139 David Stephens, Prof: In a sense, you're obligated to check that it's done its job, and created balance amongst the confounders. And we'll see, in the practical examples, the considerations for that. 325 00:36:09.140 --> 00:36:10.929 David Stephens, Prof: So again, this is the picture. 326 00:36:11.180 --> 00:36:21.490 David Stephens, Prof: We've got a confounding mechanism here, we've got instruments that predict Z only, we've got pure predictors of outcome XO, 327 00:36:22.080 --> 00:36:33.999 David Stephens, Prof: This is a typical situation. Of course, the data don't come nicely labeled like this. We don't know what's a instrument, we don't know what's a pure predictive outcome, we don't know what's a confounder. 328 00:36:34.220 --> 00:36:35.030 David Stephens, Prof: But… 329 00:36:38.580 --> 00:36:40.700 David Stephens, Prof: The theoretical message is clear. 330 00:36:40.820 --> 00:36:44.349 David Stephens, Prof: The theoretical message is that in order to 331 00:36:44.920 --> 00:36:48.010 David Stephens, Prof: Get optimal estimation of the average treatment effect. 332 00:36:48.350 --> 00:36:55.540 David Stephens, Prof: We need the confounders, Up here, and we need the pure predictors of outcome. 333 00:36:55.800 --> 00:36:57.920 David Stephens, Prof: And we don't need the instruments. 334 00:37:02.950 --> 00:37:08.829 David Stephens, Prof: So, as it says here, conventional model selection approaches For the Propensity score. 335 00:37:09.450 --> 00:37:15.170 David Stephens, Prof: I say there should not be used. Maybe I'll weaken that to say should be used only with great care. 336 00:37:16.140 --> 00:37:19.709 David Stephens, Prof: In that they can easily reproduce the, 337 00:37:20.920 --> 00:37:24.460 David Stephens, Prof: The problem of conditioning on instruments. 338 00:37:25.720 --> 00:37:31.109 David Stephens, Prof: And if you include strong predictors of the treatment, which are not confounders, in the… 339 00:37:31.470 --> 00:37:35.229 David Stephens, Prof: Propensity score model, it really gives you two problems. 340 00:37:35.590 --> 00:37:39.080 David Stephens, Prof: First of all, it uses up some of the data in estimation. 341 00:37:39.760 --> 00:37:43.530 David Stephens, Prof: Second of all, it can lead to positivity violations. 342 00:37:51.790 --> 00:37:54.620 David Stephens, Prof: So, having selected your propensity score model. 343 00:37:54.980 --> 00:37:58.009 David Stephens, Prof: We need to make sure that there are, 344 00:37:58.520 --> 00:38:02.330 David Stephens, Prof: There's no positivity violations, and we'll see some methods for checking that. 345 00:38:03.230 --> 00:38:09.890 David Stephens, Prof: And, we also need to ensure that there's balance. 346 00:38:15.250 --> 00:38:26.040 David Stephens, Prof: So one method that might be appealing initially is to say, well, let's be hyper… Careful. 347 00:38:26.810 --> 00:38:30.379 David Stephens, Prof: And throw all the variables into both models. 348 00:38:31.490 --> 00:38:36.320 David Stephens, Prof: Every variable that we've recorded, throw it into the outcome model, throw it into the propensity score model. 349 00:38:37.400 --> 00:38:38.600 David Stephens, Prof: Stick with that. 350 00:38:39.970 --> 00:38:44.720 David Stephens, Prof: So, that's gonna do the best possible job in terms of bias. 351 00:38:46.450 --> 00:38:47.439 David Stephens, Prof: For sure. 352 00:38:49.040 --> 00:38:53.000 David Stephens, Prof: But it's also gonna cut… that's gonna come at the cost of variance inflation. 353 00:38:55.030 --> 00:39:02.510 David Stephens, Prof: So… This, apparently conservative approach. 354 00:39:03.240 --> 00:39:13.309 David Stephens, Prof: Prioritises producing point estimates, over… Confidence intervals, in a sense. 355 00:39:13.790 --> 00:39:21.150 David Stephens, Prof: And, as I said before, part of the job of the statistician is to make sure that uncertainty in the inference is appropriately represented. 356 00:39:24.550 --> 00:39:27.979 David Stephens, Prof: So again, I would argue against this kind of strategy. 357 00:39:32.250 --> 00:39:35.899 David Stephens, Prof: Other practical considerations include, well, what's the measure of effect? 358 00:39:36.190 --> 00:39:38.100 David Stephens, Prof: Are we thinking about a… 359 00:39:38.250 --> 00:39:46.229 David Stephens, Prof: risk difference, risk ratio, average treatment effect, average treatment effect on the truth, average potential outcome. So again, that's a… 360 00:39:46.480 --> 00:39:49.790 David Stephens, Prof: Decision to be made, as you… 361 00:39:50.010 --> 00:40:00.240 David Stephens, Prof: begin the analysis. The methods that we've described can get at basically all of these things, but you need to be, sure about which ones you're, 362 00:40:01.190 --> 00:40:08.159 David Stephens, Prof: Which ones you're reporting. And also, whether we do this in a marginal way or a conditional way. 363 00:40:09.230 --> 00:40:11.569 David Stephens, Prof: So the causal effect of, 364 00:40:12.530 --> 00:40:19.489 David Stephens, Prof: A treatment conditional on some subsetting parameters, some stratifying parameters. 365 00:40:21.530 --> 00:40:26.680 David Stephens, Prof: Alright, so I think what I'll do now is to take a 5-minute break till 2.15, 366 00:40:26.840 --> 00:40:36.619 David Stephens, Prof: And then we'll, go into the two real data… well, the real data set and the simulation study that I've included in some of the other handouts. 367 00:40:36.730 --> 00:40:43.830 David Stephens, Prof: So I'll start again at 15 minutes past the hour, so 2.15 in my time, and we'll start off with… 368 00:40:45.620 --> 00:40:51.330 David Stephens, Prof: the, I think I'll start off with the simulation example. 369 00:40:58.040 --> 00:41:03.829 David Stephens, Prof: So this PDF you can download from the module webpage and also from my website. 370 00:45:10.400 --> 00:45:12.800 David Stephens, Prof: Okay, we'll start back in with this, 371 00:45:13.090 --> 00:45:20.179 David Stephens, Prof: Example, then, of a simulation study illustrating the methods that we've looked at in the module. 372 00:45:20.460 --> 00:45:31.930 David Stephens, Prof: So this is the handout that's called, SISCR CausalAdjustmentComparison.pdf. On the… Web folder. 373 00:45:32.210 --> 00:45:42.410 David Stephens, Prof: the one that I linked to on the first page of the slides, you'll find the knitter sheet there as well, the RNW sheet, which contains the R code in it. This is just a PDF that 374 00:45:42.630 --> 00:45:46.239 David Stephens, Prof: comes from the… from the PDF… from the, NetA sheet. 375 00:45:46.720 --> 00:45:48.170 David Stephens, Prof: Alright, so… 376 00:45:48.420 --> 00:45:54.149 David Stephens, Prof: What we're going to look at is a particular simulation example to illustrate the use of the methods. 377 00:45:54.300 --> 00:46:01.669 David Stephens, Prof: I'm gonna… Look at stratification, matching, regression, and inverse weighting, show you how they're implemented. 378 00:46:02.400 --> 00:46:10.370 David Stephens, Prof: And then, in the next example, we'll look at a real data… real data set, and look at the things that need to be checked. 379 00:46:11.370 --> 00:46:22.280 David Stephens, Prof: Alright, so… Here's the simulation setup. I've got a 3-dimensional X, Normally distributed with these parameters. 380 00:46:22.800 --> 00:46:26.319 David Stephens, Prof: And then for the treatment model, so Z is binary. 381 00:46:26.960 --> 00:46:39.369 David Stephens, Prof: And I'm going to model that as a Bernoulli random variable with the X-bit link on this collection of functions of X1 and X2. So in this case, X1 and X2 382 00:46:39.620 --> 00:46:41.929 David Stephens, Prof: Are gonna be confounders. 383 00:46:42.130 --> 00:46:45.150 David Stephens, Prof: And X3 is going to be a pure predictor of outcome. 384 00:46:45.350 --> 00:46:53.240 David Stephens, Prof: Right? X3 does not appear in the propensity model, it's not used to generate Z, therefore it's not a predictor of Z. 385 00:46:56.110 --> 00:47:06.149 David Stephens, Prof: Alright, so we see there the model for Z given X, and then finally the model for Y given X and Z given by this. So this is the data generating mechanism. 386 00:47:06.770 --> 00:47:12.759 David Stephens, Prof: with the parameters here dictating the treatment effect. So these parameters here. 387 00:47:13.210 --> 00:47:17.620 David Stephens, Prof: that encapsulate an interaction between the treatment, X1, 388 00:47:17.940 --> 00:47:24.789 David Stephens, Prof: So X is a main effect, but also interaction with X1, X2, and the interaction, X1, X2. 389 00:47:26.660 --> 00:47:31.200 David Stephens, Prof: And the coefficients that I'm going to try to recover here. 390 00:47:31.630 --> 00:47:34.369 David Stephens, Prof: They're all ones. The psi's are all ones. 391 00:47:40.820 --> 00:47:48.279 David Stephens, Prof: Oh, actually, so yeah, sorry, X3 is not a, not a predictor of outcome either, so this has only got X1 and X2 in both models. 392 00:47:48.600 --> 00:47:55.470 David Stephens, Prof: So, X does not predict Z, and it does not… X3 does not predict Z, and it doesn't predict Y. 393 00:48:00.000 --> 00:48:09.380 David Stephens, Prof: Now, the simulation that I'm doing, so this is just the R code to do that simulation, I'm getting a fairly large sample size of 2,000 in order to make the points clear. 394 00:48:10.720 --> 00:48:16.669 David Stephens, Prof: And then forming various, Functions of those data, that's fine. 395 00:48:17.020 --> 00:48:21.680 David Stephens, Prof: And then… The first thing I'm going to do is to compute the… 396 00:48:21.920 --> 00:48:32.109 David Stephens, Prof: the average treatment effect in this case. Now, in this model, the average treatment effect is not trivial to write down. It's no longer a single coefficient. 397 00:48:32.390 --> 00:48:41.169 David Stephens, Prof: Because it's the difference between the expected value here, when Z is 0, and the expected value here when Z is 1. 398 00:48:41.670 --> 00:48:47.670 David Stephens, Prof: And in order to compute that, you need the expectation of X1, X2, and the product X1, X2. 399 00:48:49.810 --> 00:48:55.399 David Stephens, Prof: So that's what the next little bit of theory does here, does all those calculations for you. 400 00:48:56.610 --> 00:48:57.989 David Stephens, Prof: Come to the end. 401 00:48:58.400 --> 00:49:05.920 David Stephens, Prof: The two average potential outcomes under the two cases are under no treatment, 4.4675. 402 00:49:06.320 --> 00:49:09.300 David Stephens, Prof: Under Treatment, 2.58. 403 00:49:09.550 --> 00:49:20.770 David Stephens, Prof: And therefore, the causal effect, the average treatment effect that we're after, that we're trying to estimate all the way through, is minus 1.8875. That's the true value of the parameter. 404 00:49:23.930 --> 00:49:29.660 David Stephens, Prof: Okay, now, in this dataset, We can go through a series of… 405 00:49:30.160 --> 00:49:37.949 David Stephens, Prof: analyses. So the unadjusted analysis just says, look at the difference in means between the treatment group and the no treatment group. 406 00:49:38.800 --> 00:49:41.249 David Stephens, Prof: Now, we know that that's going to give us the wrong answer. 407 00:49:43.630 --> 00:49:53.000 David Stephens, Prof: I can certainly estimate the means in the two groups fairly well. These are the untreated, these are the treated. Box plot of the observed of the individual observations. 408 00:49:54.160 --> 00:49:57.149 David Stephens, Prof: Red dot is the, sample mean. 409 00:49:58.370 --> 00:49:59.730 David Stephens, Prof: In that case… 410 00:50:00.130 --> 00:50:05.909 David Stephens, Prof: We end up with an estimated causal effect, just taking the two difference… the difference between the two of… 411 00:50:06.560 --> 00:50:10.699 David Stephens, Prof: Minus 1.518, so that looks to be a bit incorrect. 412 00:50:10.930 --> 00:50:13.440 David Stephens, Prof: We don't have a standard error attached to that. 413 00:50:13.640 --> 00:50:20.519 David Stephens, Prof: But we can get that standard error using the LM function just by regressing Y on Z. That gives us the same answer. 414 00:50:22.010 --> 00:50:27.400 David Stephens, Prof: So we see from this analysis that the estimate… 415 00:50:28.400 --> 00:50:36.980 David Stephens, Prof: Of the… in the unadjusted analysis is 1.518, with a standard error of 0.04, so clearly, we're getting biased estimation. 416 00:50:38.940 --> 00:50:44.159 David Stephens, Prof: Remember, the answer that I'm after is minus 1.8875. 417 00:50:46.240 --> 00:50:47.500 David Stephens, Prof: Okay, good. 418 00:50:47.820 --> 00:50:52.829 David Stephens, Prof: So, let's now go through the… oh yeah, and there, I just explained the, 419 00:50:53.200 --> 00:50:56.939 David Stephens, Prof: the sandwich-based estimation as well. So this is the model 420 00:50:58.100 --> 00:51:13.460 David Stephens, Prof: model-assisted standard error being computed here. I can also compute the same thing using a robust or sandwich approach, using the estimating functions that I wrote down in the slides. If I do that, here's the calculation here, the sandwich 421 00:51:13.890 --> 00:51:20.790 David Stephens, Prof: function gives me the, sorry, the sandwich library gives me the function sandwich, and I can get the robust variance from that. 422 00:51:21.140 --> 00:51:22.570 David Stephens, Prof: leaves us here. 423 00:51:23.180 --> 00:51:28.419 David Stephens, Prof: And I can get the same answer by direct calculation using the theoretical results I had. 424 00:51:29.030 --> 00:51:33.539 David Stephens, Prof: Alright, so… That tells us that the… 425 00:51:33.820 --> 00:51:42.580 David Stephens, Prof: confidence interval, after I've computed these robust standard Robust standard errors. 426 00:51:42.860 --> 00:51:48.550 David Stephens, Prof: And using the… Correct formula, I can tell that the… 427 00:51:48.920 --> 00:52:00.489 David Stephens, Prof: Confidence interval, approximate 95% confidence interval based on the robust standard error, goes from minus 1.588 to minus 1.447, so that's biased. 428 00:52:00.620 --> 00:52:05.020 David Stephens, Prof: The true value, minus 1.88, does not lie in the confidence interval. 429 00:52:05.250 --> 00:52:09.010 David Stephens, Prof: Right, and in this large sample size, I can detect that as bias. 430 00:52:10.330 --> 00:52:18.100 David Stephens, Prof: Alright, so, the unadjusted analysis gives me the wrong answer. That's not a huge surprise. So now let's try some corrected analyses. 431 00:52:19.660 --> 00:52:22.320 David Stephens, Prof: If I use the correctly specified model. 432 00:52:23.400 --> 00:52:27.070 David Stephens, Prof: Doing only the outcome regression analysis. 433 00:52:27.710 --> 00:52:40.810 David Stephens, Prof: I use LM to estimate the coefficients, and then use this formula that I described in the slides in order to estimate the average potential outcome for treated and untreated individuals. 434 00:52:41.090 --> 00:52:44.510 David Stephens, Prof: I use the predict function in order to get me that. 435 00:52:44.740 --> 00:52:55.060 David Stephens, Prof: And when I plug in the final results to get the average potential outcome under no treatment and under treatment, they're both correct. 436 00:52:55.370 --> 00:52:56.790 David Stephens, Prof: And the average… 437 00:52:56.980 --> 00:53:11.830 David Stephens, Prof: treatment effect is also estimated in a fairly accurate fashion, okay? So the true answer is minus 1.88, here I get minus 1.89, a little bit of random variability due to the data variation, the measurement errors. 438 00:53:12.400 --> 00:53:13.100 David Stephens, Prof: Okay? 439 00:53:14.690 --> 00:53:17.559 David Stephens, Prof: I can compute the sandwich variance. 440 00:53:19.570 --> 00:53:29.210 David Stephens, Prof: This is what this calculation does here, all the way through. And when I do that, either using the sandwich form, or using a direct, analytical-based calculation. 441 00:53:29.490 --> 00:53:41.150 David Stephens, Prof: I can get… Competence intervals, and it's clear that the competence interval takes me to the… 442 00:53:41.590 --> 00:53:43.189 David Stephens, Prof: To the right sort of region. 443 00:53:45.520 --> 00:53:53.280 David Stephens, Prof: Okay, so… the important thing to note is here that when I create these predictions, Here. 444 00:53:53.530 --> 00:54:01.989 David Stephens, Prof: I artificially create a dataset, With no treatment, and treatment respectively, in order to plug into this formula. 445 00:54:02.220 --> 00:54:09.030 David Stephens, Prof: What I don't do is use the estimate… sorry, is use the estimated model and the observed Zs. 446 00:54:09.390 --> 00:54:15.820 David Stephens, Prof: If I do that, I get back the original answer, the biased, unadjusted analysis answer. 447 00:54:18.160 --> 00:54:19.550 David Stephens, Prof: And that's due to confounding. 448 00:54:21.760 --> 00:54:25.749 David Stephens, Prof: If I use an incorrectly specified model, so I get the regression model 449 00:54:26.160 --> 00:54:35.230 David Stephens, Prof: wrong at the first stage, then again, I get the wrong answer. Here, it's not too far off, because I've done a moderate job in getting a good specification. 450 00:54:35.520 --> 00:54:42.879 David Stephens, Prof: But, following through the confidence interval calculations, I still get the wrong answer. I've still got a little bit of bias. 451 00:54:44.930 --> 00:54:45.700 David Stephens, Prof: Alright? 452 00:54:47.690 --> 00:54:48.640 David Stephens, Prof: Now… 453 00:54:48.880 --> 00:54:56.139 David Stephens, Prof: What I've done a good job in here, though, is including X1 and X2 in the model. The only thing I'm really missing is the X1-X2 interaction. 454 00:54:57.130 --> 00:55:01.469 David Stephens, Prof: So, although this is an incorrect specification, it's actually not a bad specification. 455 00:55:02.230 --> 00:55:10.429 David Stephens, Prof: So… if I omit One of those variables, though, Everything goes completely wrong. 456 00:55:12.330 --> 00:55:17.490 David Stephens, Prof: Alright, so now we're going to turn on to propensity score methods, and in this case. 457 00:55:17.720 --> 00:55:21.670 David Stephens, Prof: I'm gonna display the propensity score histogram. 458 00:55:22.800 --> 00:55:27.240 David Stephens, Prof: So this is the, essentially, the observed 459 00:55:27.540 --> 00:55:33.970 David Stephens, Prof: values of EX, the probability that Z is equal to 1 given X, 460 00:55:34.250 --> 00:55:39.980 David Stephens, Prof: Falls on a range from 0 to 1, and it looks a bit like this. So this is a nicely configured propensity score. 461 00:55:40.790 --> 00:55:48.370 David Stephens, Prof: We don't have any positivity violations, at least on the surface. We're not too near the zero end, we're not too near the one end. This is a nice, amenable 462 00:55:48.660 --> 00:55:52.500 David Stephens, Prof: Propensity score distribution to apply our methods to. 463 00:55:53.760 --> 00:56:00.780 David Stephens, Prof: Okay? These are the… the true propensity scores, in a second I'm going to estimate them from the data. 464 00:56:02.250 --> 00:56:02.950 David Stephens, Prof: Okay? 465 00:56:04.260 --> 00:56:17.329 David Stephens, Prof: Before I, do that, I'm just going to show that this stratification idea, so stratification by the propensity score, remember that means breaking up the data into subsets identified by 466 00:56:17.790 --> 00:56:27.170 David Stephens, Prof: approximately equal propensity score values. If I take quintiles of the propensity score, so divide the Propensity scope into 5 467 00:56:27.800 --> 00:56:30.990 David Stephens, Prof: Strata. 468 00:56:31.750 --> 00:56:36.390 David Stephens, Prof: And I use those to, 469 00:56:36.700 --> 00:56:51.020 David Stephens, Prof: perform estimation, in a sense, non-parametric estimation based on the stratification. First of all, I note that if I do that, I get good representation of the untreated and treated within each stratum. Remember, I said that that was important yesterday. 470 00:56:51.170 --> 00:56:53.550 David Stephens, Prof: There are a large number of individuals 471 00:56:54.560 --> 00:56:58.009 David Stephens, Prof: Who are both treated and untreated within each stratum. 472 00:56:58.340 --> 00:57:05.180 David Stephens, Prof: So that means I can perform the causal estimation within each stratum, and then combine across strata at the end. 473 00:57:06.070 --> 00:57:08.369 David Stephens, Prof: That's what this little bit of code does here. 474 00:57:08.960 --> 00:57:12.340 David Stephens, Prof: And it returns for me an estimate, which is really not too bad. 475 00:57:15.330 --> 00:57:21.899 David Stephens, Prof: I can implement that using, LM as well, using stratum as a factor predictor. 476 00:57:24.270 --> 00:57:26.370 David Stephens, Prof: And then, ultimately get… 477 00:57:26.530 --> 00:57:33.859 David Stephens, Prof: predictions from the method, so the prediction is the same. The advantage of doing it this way is that I can also get standard errors out easily. 478 00:57:35.050 --> 00:57:40.360 David Stephens, Prof: Right, so if I use LM and the sandwich formula, then I can get… 479 00:57:41.070 --> 00:57:49.330 David Stephens, Prof: Easily standard errors, whereas previously stratification estimators were only giving me point estimates. 480 00:57:49.770 --> 00:57:57.119 David Stephens, Prof: So I'm implementing the stratification estimator using LM, and that ultimately returns to me a method that works pretty well. 481 00:57:57.630 --> 00:58:04.759 David Stephens, Prof: Here, I'm getting slightly wider confidence intervals compared to before, but the true value is included in the confidence interval. 482 00:58:05.230 --> 00:58:05.930 David Stephens, Prof: Right. 483 00:58:07.130 --> 00:58:11.519 David Stephens, Prof: Well, the second one narrowly excludes it, but it's still not too bad. 484 00:58:12.910 --> 00:58:26.090 David Stephens, Prof: That's with the known propensity score. If I repeat the same thing with the fitted propensity score, though, that means I use GLM to fit Z as a model of the X, based on the model of the X's, correctly specified. 485 00:58:26.350 --> 00:58:33.049 David Stephens, Prof: to estimate the propensity score values, and then I use the propensity score values in the propensity score regression. 486 00:58:34.070 --> 00:58:35.740 David Stephens, Prof: Sorry, in the stratification. 487 00:58:36.270 --> 00:58:38.610 David Stephens, Prof: So I do that, repeat that the same way. 488 00:58:39.160 --> 00:58:41.200 David Stephens, Prof: I end up with a similar estimate. 489 00:58:41.560 --> 00:58:44.480 David Stephens, Prof: And… similar confidence intervals. 490 00:58:44.860 --> 00:58:50.440 David Stephens, Prof: So in this large sample size and the correct specification, with this nice propensity score distribution. 491 00:58:50.670 --> 00:59:02.229 David Stephens, Prof: really, it makes no difference whether I use the true propensity scores or estimated propensity scores. In practice, of course, I'm always going to have to use the estimated one, but this shows that you can get good recovery of that from 492 00:59:02.800 --> 00:59:05.210 David Stephens, Prof: From the data in this case. 493 00:59:07.340 --> 00:59:16.100 David Stephens, Prof: And, well, I've glossed over it, the methods that we're using all the way down are really very standard methods. They're using lm function, we're using predict. 494 00:59:17.020 --> 00:59:24.339 David Stephens, Prof: we're using matrix manipulation to get the sandwich estimate. Everything is very, very straightforward in our… 495 00:59:28.880 --> 00:59:32.470 David Stephens, Prof: If the Propensity score model is misspecified, however. 496 00:59:32.760 --> 00:59:35.469 David Stephens, Prof: I can still go through the steps of the calculation. 497 00:59:35.570 --> 00:59:46.219 David Stephens, Prof: If I only include X1, for example, remember, actually, I should be including X2, and the interaction between X1 and X2 in the propensity score. But if I only include X1, 498 00:59:46.610 --> 00:59:53.860 David Stephens, Prof: Then… going through the same calculations, I end up with, again, a… Poor estimate. 499 00:59:54.200 --> 00:59:58.639 David Stephens, Prof: Right, so that tells me that correct specification of the propensity score is necessary. 500 01:00:00.000 --> 01:00:09.690 David Stephens, Prof: And, I can… Assess whether the propensity score model is poor, By looking at balance. 501 01:00:10.610 --> 01:00:11.580 David Stephens, Prof: So… 502 01:00:12.010 --> 01:00:21.890 David Stephens, Prof: What this next little bit of code does here is to look at, again, in the quintiles of the propensity score, whether I get the same sample mean of the X's. 503 01:00:23.320 --> 01:00:26.429 David Stephens, Prof: For treated and untreated individuals. 504 01:00:27.240 --> 01:00:33.569 David Stephens, Prof: Okay? So what this shows, so this is where I fit the… the, the true model. 505 01:00:34.130 --> 01:00:36.429 David Stephens, Prof: Or, yeah, fit the correct model. 506 01:00:36.570 --> 01:00:46.600 David Stephens, Prof: And I'm getting good balance. Whereas if I fit the incorrect model, I'm getting… Appreciable lack of balance. 507 01:00:49.750 --> 01:00:54.269 David Stephens, Prof: This is summary for X1, this is summary for X2, this is summary for the interaction. 508 01:00:55.450 --> 01:01:02.379 David Stephens, Prof: Okay? So the Propensity score mechanism will provide balance for those variables that are included. 509 01:01:03.940 --> 01:01:11.199 David Stephens, Prof: But it won't provide balance for those variables that are not included. Remember, that's an essential part of the propensity score construction. 510 01:01:15.120 --> 01:01:20.400 David Stephens, Prof: To observe lack of balance here, and we'll see other representations of this in the real data analysis. 511 01:01:20.750 --> 01:01:27.180 David Stephens, Prof: Then I go to matching, so matching on the propensity score. This is using the matching package. 512 01:01:27.220 --> 01:01:42.030 David Stephens, Prof: And, there's some summary output here. And the good thing about this package is that you can ask it to perform various different sorts of estimation. There are lots of different options here. I'm going to estimate the ATE using one-to-one matching. 513 01:01:43.340 --> 01:01:54.589 David Stephens, Prof: Okay? The good thing about this package is it returns the estimate, but it also returns the standard error that allows me to construct a confidence interval, and you can see here that the matching estimation does a good job. 514 01:01:56.210 --> 01:01:56.920 David Stephens, Prof: Okay? 515 01:01:58.090 --> 01:02:01.840 David Stephens, Prof: If I use a 1 to M matching. 516 01:02:02.060 --> 01:02:03.980 David Stephens, Prof: 1 to 5, or 1 to 10. 517 01:02:04.140 --> 01:02:08.790 David Stephens, Prof: then… It changes a little, but not too much. 518 01:02:11.260 --> 01:02:22.480 David Stephens, Prof: Then I introduce propensity score regression. The only additional step here is that I need to fit a regression model for the outcome, given the treatment and the propensity score. 519 01:02:24.490 --> 01:02:25.160 David Stephens, Prof: Okay? 520 01:02:25.750 --> 01:02:27.340 David Stephens, Prof: So when I do that. 521 01:02:27.470 --> 01:02:36.399 David Stephens, Prof: I end up with a model-based, sorry, an estimate which is, minus 2.101, so not too far off. 522 01:02:36.510 --> 01:02:39.660 David Stephens, Prof: And a standard error of .029. 523 01:02:40.280 --> 01:02:46.650 David Stephens, Prof: Sandwiched standard error of .021. So that means that the confidence interval is, 524 01:02:46.760 --> 01:02:51.190 David Stephens, Prof: So it changes in the second decimal place, so we still have a… 525 01:02:53.070 --> 01:03:00.090 David Stephens, Prof: Oh.03 here, so the… we have a little bit of bias, but the… but the, 526 01:03:04.550 --> 01:03:08.599 David Stephens, Prof: But the estimation, the adjustment, is doing a reasonable job. 527 01:03:09.690 --> 01:03:18.659 David Stephens, Prof: The first propensity score regression model I fitted here, though, was… 528 01:03:20.200 --> 01:03:23.260 David Stephens, Prof: As we've seen from the theory, not the correct one. 529 01:03:23.490 --> 01:03:29.599 David Stephens, Prof: Because in the data generating model, We've got interactions with Z, 530 01:03:29.850 --> 01:03:36.830 David Stephens, Prof: an X, and that means that we have to modify this function, and modify the propensity score term. 531 01:03:37.230 --> 01:03:40.200 David Stephens, Prof: So, although this does a reasonable job. 532 01:03:40.370 --> 01:03:45.420 David Stephens, Prof: with only a little bit of bias, it's actually an incorrect specification to do the job. 533 01:03:46.460 --> 01:03:50.570 David Stephens, Prof: If I move to the correct specification of the treatment effect model. 534 01:03:51.830 --> 01:03:54.919 David Stephens, Prof: But leave the propensity score model 535 01:03:56.380 --> 01:03:58.980 David Stephens, Prof: Only in this single additive term. 536 01:03:59.480 --> 01:04:04.380 David Stephens, Prof: then… I end up with a reasonable… Estimate. 537 01:04:08.430 --> 01:04:11.509 David Stephens, Prof: If I fit the thing that I'm supposed to fit. 538 01:04:11.670 --> 01:04:15.370 David Stephens, Prof: correctly, I end up with an even better 539 01:04:15.590 --> 01:04:18.890 David Stephens, Prof: fit. So this is the model that I should be fitting. 540 01:04:19.430 --> 01:04:22.079 David Stephens, Prof: and… That gets for me. 541 01:04:22.320 --> 01:04:23.700 David Stephens, Prof: Good estimation. 542 01:04:24.090 --> 01:04:29.450 David Stephens, Prof: with a… Very precise estimation, in terms of the standard error. 543 01:04:30.650 --> 01:04:40.069 David Stephens, Prof: So these different propensity score regression methods show a couple of models that get approximately the correct answer, but don't have the theoretical guarantee. 544 01:04:40.460 --> 01:04:44.559 David Stephens, Prof: And then finally, the fully correct model, which shows the… 545 01:04:45.280 --> 01:04:47.559 David Stephens, Prof: The, the, gives the right answer. 546 01:04:48.840 --> 01:04:55.010 David Stephens, Prof: Okay? In this Propensity score regression, we can also think about estimating the individual coefficients of the terms. 547 01:04:55.520 --> 01:05:02.160 David Stephens, Prof: So the size that appear in this model, and this little table here shows that we get a good answer for those. 548 01:05:04.200 --> 01:05:06.270 David Stephens, Prof: So again, all of the methods 549 01:05:06.950 --> 01:05:10.619 David Stephens, Prof: Based on regression, just use LM and GLM. 550 01:05:13.560 --> 01:05:18.900 David Stephens, Prof: sandwich estimation done by hand again. So, next inverse probability weighting. 551 01:05:22.810 --> 01:05:28.329 David Stephens, Prof: easily implemented through standard approaches. We only need the Propensity score model fit. 552 01:05:28.890 --> 01:05:34.959 David Stephens, Prof: That returns for me an estimate of the causal effect, which is approximately correct, again. 553 01:05:35.560 --> 01:05:37.909 David Stephens, Prof: We can use weighted least squares. 554 01:05:38.430 --> 01:05:45.540 David Stephens, Prof: Using LM, but with the weights argument. Again, the weights are the… The inverse probability weighting weights. 555 01:05:47.170 --> 01:05:51.250 David Stephens, Prof: Again, that gives me a reasonable answer, but again, a slightly higher standard deviation. 556 01:05:54.800 --> 01:06:02.909 David Stephens, Prof: With augmentation, that requires the modeling of another mean function, so I do that, include that in the estimation. 557 01:06:03.420 --> 01:06:09.180 David Stephens, Prof: Use ordinary weights or standardized weights, and again, we get a reasonable answer. 558 01:06:09.500 --> 01:06:16.059 David Stephens, Prof: Okay? So we're able to solve this problem of confounding using any number of these approaches. 559 01:06:16.210 --> 01:06:21.550 David Stephens, Prof: And provided we apply the method correctly, we can recover the unconfounded estimate. 560 01:06:24.820 --> 01:06:28.189 David Stephens, Prof: There's a bit more of the theoret logic, which I'll skip over now. 561 01:06:28.440 --> 01:06:34.149 David Stephens, Prof: And then some consequences of misspecification. 562 01:06:34.280 --> 01:06:37.230 David Stephens, Prof: I've also got a little bit of, 563 01:06:38.190 --> 01:06:41.769 David Stephens, Prof: Illustration of the double robustness in this. 564 01:06:42.710 --> 01:06:47.679 David Stephens, Prof: Okay. There are packages that implement AIPW, so RCAL does that. 565 01:06:48.120 --> 01:06:49.420 David Stephens, Prof: For example. 566 01:06:49.540 --> 01:06:55.940 David Stephens, Prof: And, just at the end here, I go through a little bit of the calculation of the… 567 01:06:57.030 --> 01:07:00.789 David Stephens, Prof: Sandwich variance for this weighted estimating setting. 568 01:07:01.980 --> 01:07:20.019 David Stephens, Prof: Okay, so there's the theoretical calculations, and I demonstrate where that comes from. So I find this useful, when I teach this in, courses, because it allows you to go through, rather than use the sandwich package, you can go through and see the exact nature of the calculation at each step. 569 01:07:20.140 --> 01:07:22.659 David Stephens, Prof: computed using standard R tools. 570 01:07:24.810 --> 01:07:32.210 David Stephens, Prof: Okay? And then, augmented outcome regression, again, essentially implementing an AIPW method. 571 01:07:32.660 --> 01:07:37.849 David Stephens, Prof: Again, using standard tools, LM and PREDICT, in order to get 572 01:07:38.250 --> 01:07:41.889 David Stephens, Prof: What are reasonably good estimates of the true value? 573 01:07:45.140 --> 01:07:49.890 David Stephens, Prof: Some standard error calculations. And then finally, G-estimation. 574 01:07:50.150 --> 01:07:52.550 David Stephens, Prof: Again, I explained the logic there. 575 01:07:52.930 --> 01:08:01.510 David Stephens, Prof: And, I put it by hand, first of all, and then I used the DRGEE package to get this… to get the results. 576 01:08:01.820 --> 01:08:05.330 David Stephens, Prof: Okay, and then this goes through. Explaining a little bit about that. 577 01:08:05.850 --> 01:08:15.070 David Stephens, Prof: So… with the theoretical calculations at the end. So, this handout is meant to lay out, 578 01:08:15.600 --> 01:08:24.380 David Stephens, Prof: the taxonomy of methods, and how you would implement them in R. And really, there is very little of it which is using specialized packages. 579 01:08:24.810 --> 01:08:34.190 David Stephens, Prof: So the matching calculation I didn't do from hand, but all the other ones didn't do by, from first principles, but all the other ones you can do, 580 01:08:34.979 --> 01:08:37.700 David Stephens, Prof: You can use… you can do using standard tools. 581 01:08:37.920 --> 01:08:41.950 David Stephens, Prof: So the code is there for you to… to look at those, 582 01:08:42.290 --> 01:08:52.159 David Stephens, Prof: results, and you can take the RNW file from the website to, download and run and manipulate yourself if you want to change the values. 583 01:08:53.689 --> 01:09:03.420 David Stephens, Prof: Alright, so… the… Other handout that I want to talk about is the NHANES. 584 01:09:04.000 --> 01:09:06.440 David Stephens, Prof: And, real data analysis. 585 01:09:07.000 --> 01:09:13.750 David Stephens, Prof: So, here we're not simulating data, so there are no… True answers. 586 01:09:15.050 --> 01:09:17.450 David Stephens, Prof: they're only… 587 01:09:18.100 --> 01:09:27.650 David Stephens, Prof: what we can learn from the data. So the NHANES dataset is where we have, it's a large ongoing cohort, survey, actually. 588 01:09:27.960 --> 01:09:33.130 David Stephens, Prof: Where we, for our purposes, we're going to simplify it to look at 589 01:09:33.240 --> 01:09:38.579 David Stephens, Prof: the effect of systolic… sorry, the effect of smoking on systolic blood pressure. So, smoking… 590 01:09:38.740 --> 01:09:46.110 David Stephens, Prof: Current smoking status, no or yes, the impact of that on systolic blood pressure. 591 01:09:46.439 --> 01:09:49.029 David Stephens, Prof: In the presence of some other covariates. 592 01:09:51.029 --> 01:09:52.479 David Stephens, Prof: So… 593 01:09:53.649 --> 01:10:02.060 David Stephens, Prof: you can look through the analysis here. The things I want to illustrate, mainly with respect to this real analysis, are the ways that we check 594 01:10:02.200 --> 01:10:03.930 David Stephens, Prof: Whether the models are working. 595 01:10:05.770 --> 01:10:11.010 David Stephens, Prof: So… One way I'm going to do that is to look at 596 01:10:11.580 --> 01:10:16.979 David Stephens, Prof: Balanced, through some formal measures, through what's called the standardized mean difference. 597 01:10:17.680 --> 01:10:21.170 David Stephens, Prof: Which will either be… the, 598 01:10:21.480 --> 01:10:32.189 David Stephens, Prof: Standardized mean difference, or weighted standardized mean difference, between the covariate value Covariate by covariate. 599 01:10:32.300 --> 01:10:35.239 David Stephens, Prof: In the treated and untreated individuals. 600 01:10:36.370 --> 01:10:40.820 David Stephens, Prof: Okay, so I'm going to be able to compute that using some standard R technology. 601 01:10:41.080 --> 01:10:45.309 David Stephens, Prof: I'm going to look at those numerical values to assess balance. 602 01:10:46.110 --> 01:10:55.849 David Stephens, Prof: So if there are no differences between the treatment group and the treated and non-treated groups, the smokers and the non-smokers, then I'll be able to proceed with my analysis. 603 01:10:56.650 --> 01:11:04.490 David Stephens, Prof: I'm also going to look at things like empirical CDFs or weighted versions of those to see whether there's any difference between the covariate profiles. 604 01:11:05.340 --> 01:11:18.220 David Stephens, Prof: Okay? So the Table 1 calculation for this, stratified by smoking, tells us that there are rather large standardized mean differences, so we're looking for this value to be less than 0.1. 605 01:11:18.740 --> 01:11:29.170 David Stephens, Prof: In order to indicate some balance. In all of the variables, all of the predictors, there is a systematic difference between the smokers and the smokers and the non-smokers. 606 01:11:29.950 --> 01:11:33.289 David Stephens, Prof: There is a standardized mean difference, which is too large. 607 01:11:33.730 --> 01:11:36.159 David Stephens, Prof: To regard these to be balanced. 608 01:11:37.720 --> 01:11:47.010 David Stephens, Prof: And if we look at the empirical CDF for age, for example, we can clearly see that the smokers are fundamentally different in terms of age from the non-smokers. 609 01:11:48.250 --> 01:11:52.260 David Stephens, Prof: So, if smoking… It's also a predictor of the outcome. 610 01:11:52.610 --> 01:11:55.260 David Stephens, Prof: this is a problem. Smoking is then a… 611 01:11:55.940 --> 01:11:58.880 David Stephens, Prof: Sorry, age is therefore a… 612 01:11:59.460 --> 01:12:03.950 David Stephens, Prof: So, sorry, if age is the predictor of outcome, then age is a confounder. 613 01:12:09.970 --> 01:12:16.470 David Stephens, Prof: So clearly, age is not balanced, the profile for age is different in the two groups, and therefore, we need to adjust for that somehow. 614 01:12:17.020 --> 01:12:22.139 David Stephens, Prof: So I'm going to build a propensity score model based on the full collection of predictors. 615 01:12:24.030 --> 01:12:30.590 David Stephens, Prof: I'm gonna stratify about that first of all, and see that I get reasonable representation of each of the strata. 616 01:12:32.600 --> 01:12:39.500 David Stephens, Prof: If I look at the plots of the propensity score for the two groups separately, I see that we have… 617 01:12:40.650 --> 01:12:45.490 David Stephens, Prof: no positivity violations, essentially. What I see here, so the dark blue. 618 01:12:45.870 --> 01:12:53.280 David Stephens, Prof: is the smoking group, or the dark blue. The blue, the purple, sorry, this blue here, that's the… 619 01:12:53.440 --> 01:12:56.859 David Stephens, Prof: smoking group. The red is the non-smoking group. 620 01:12:58.240 --> 01:13:05.430 David Stephens, Prof: No, sorry. The blue is the non-smoking group, the red is the smoking group, and this purple color is where those two histograms overlap. 621 01:13:05.780 --> 01:13:06.510 David Stephens, Prof: Right? 622 01:13:06.650 --> 01:13:14.549 David Stephens, Prof: So we see good overlap over the whole range of the propensity score. That means that, backed up by these figures here, there are 623 01:13:15.040 --> 01:13:21.889 David Stephens, Prof: matchable people. That is, people that have the same propensity score value, but have different treatment status. 624 01:13:23.060 --> 01:13:26.789 David Stephens, Prof: Those people are critical, those people facilitate the causal comparison. 625 01:13:28.690 --> 01:13:33.019 David Stephens, Prof: This is the Propensity score overlap picture. This is the one thing that we check. 626 01:13:33.210 --> 01:13:34.550 David Stephens, Prof: First thing that we check. 627 01:13:34.780 --> 01:13:39.760 David Stephens, Prof: So, this propensity score model is… a good one, for… 628 01:13:39.960 --> 01:13:46.850 David Stephens, Prof: doing the comparison, we don't have any extreme propensity score values. We have some overlap between the values. 629 01:13:49.360 --> 01:13:55.130 David Stephens, Prof: Very quickly, I'm gonna go through the, the… 630 01:13:56.720 --> 01:13:59.359 David Stephens, Prof: A way that we would approach doing this analysis. 631 01:14:00.100 --> 01:14:10.889 David Stephens, Prof: So, the first thing we look at is standardized mean difference. These are the original standardized mean differences, and these are the standardized mean differences in quintiles of the propensity score. 632 01:14:11.010 --> 01:14:13.539 David Stephens, Prof: So imagine we were just doing stratification. 633 01:14:13.860 --> 01:14:18.570 David Stephens, Prof: And what the overall message is, is that these two… these numbers are too big. 634 01:14:18.950 --> 01:14:22.030 David Stephens, Prof: The standardized mean difference in each of the quintiles. 635 01:14:22.390 --> 01:14:25.679 David Stephens, Prof: Is too large on average for all of the covariates. 636 01:14:26.620 --> 01:14:28.489 David Stephens, Prof: So we don't have balance. 637 01:14:31.760 --> 01:14:35.889 David Stephens, Prof: So that tells us that… on the… 638 01:14:36.300 --> 01:14:44.510 David Stephens, Prof: empirical CDS back this up, we still have too large a difference, even if we stratify by the Propensity score, even if we look at these strata. 639 01:14:44.780 --> 01:14:50.559 David Stephens, Prof: The Propensity score is not… A sufficient balancing mechanism. 640 01:14:51.430 --> 01:14:53.699 David Stephens, Prof: For, for, for use in stratification. 641 01:14:57.460 --> 01:15:01.220 David Stephens, Prof: If we take a matching approach, however, matching on the propensity score. 642 01:15:01.770 --> 01:15:07.670 David Stephens, Prof: We get, actually, good agreement between the… the… 643 01:15:08.200 --> 01:15:12.349 David Stephens, Prof: treated and untreated groups. If I look at the age CDF, 644 01:15:12.620 --> 01:15:18.160 David Stephens, Prof: if I've matched one… one smoker To one non-smoker. 645 01:15:18.740 --> 01:15:21.240 David Stephens, Prof: On the basis of their propensity score value. 646 01:15:21.520 --> 01:15:40.900 David Stephens, Prof: then this is the empirical CDF of the match sample, and the standardized mean differences become a lot smaller, as we would expect. So what that tells us is that stratification is too broad a tool, we need to be more precise, and to stratify at an even finer level based on individual pairing. 647 01:15:41.280 --> 01:15:47.550 David Stephens, Prof: Of individuals. And once we've got this matched or paired analysis, we can estimate the causal effect. 648 01:15:51.030 --> 01:15:55.730 David Stephens, Prof: We can achieve the same thing with weighting, actually. In fact, weighting does a slightly better job. 649 01:15:57.920 --> 01:16:03.379 David Stephens, Prof: Right, the standardized mean difference, or the weighted sample, they're much smaller. 650 01:16:04.400 --> 01:16:10.840 David Stephens, Prof: So what that tells us is that this propensity score model could not be used for stratification analysis. 651 01:16:11.030 --> 01:16:16.070 David Stephens, Prof: But the same model could be used for matching, and it could be used for… Waiting. 652 01:16:19.150 --> 01:16:33.280 David Stephens, Prof: The rest of this handout really goes through the fitting of the Propensity score models through stratification, matching, regression, and inverse weighting. So the codes are basically the same. I've really… 653 01:16:33.500 --> 01:16:39.620 David Stephens, Prof: just repeated everything from that previous analysis. The important thing for this, what this handout shows, is that 654 01:16:39.730 --> 01:16:44.109 David Stephens, Prof: We need to do this initial first step of checking for overlap. 655 01:16:44.720 --> 01:16:49.959 David Stephens, Prof: And then checking that balance has been occurred. Balance has been obtained. 656 01:16:53.180 --> 01:16:58.340 David Stephens, Prof: So what I do at the end is to, so then we're getting estimates 657 01:16:58.490 --> 01:17:02.680 David Stephens, Prof: Without standard errors recorded here, of… 658 01:17:04.110 --> 01:17:09.930 David Stephens, Prof: Which basically come up at the 1.99 solar level, so this is the real 659 01:17:10.440 --> 01:17:12.160 David Stephens, Prof: An estimate of the real effect. 660 01:17:12.420 --> 01:17:27.010 David Stephens, Prof: Here I, in this bit of the analysis, I didn't bother doing the sandwich estimation, but you can easily implement that. I didn't do it. Instead, I did a bootstrap analysis, which is what's contained here. So this is the bootstrap comparison of, estimation 661 01:17:27.010 --> 01:17:32.539 David Stephens, Prof: for the different methods. So I've got 3 different regression methods, so one very naive one. 662 01:17:32.540 --> 01:17:35.230 David Stephens, Prof: And then 2 more sophisticated ones. 663 01:17:35.550 --> 01:17:44.090 David Stephens, Prof: Stratification, matching, and then through propensity score regression models, and then the inverse weighting and augmented inverse weighting approaches. 664 01:17:45.920 --> 01:17:53.380 David Stephens, Prof: So as you'll see when you look at the handout, only really the matching estimator requires special technology. 665 01:17:54.480 --> 01:17:59.320 David Stephens, Prof: Everything else can be done through… LM and GLM type calculations. 666 01:17:59.520 --> 01:18:07.450 David Stephens, Prof: Right, and what we're getting is that there's a consistent view of the… Effect at about this level. 667 01:18:10.650 --> 01:18:25.400 David Stephens, Prof: Okay, and these are the numerical values. So those methods return estimates of this sort of magnitude with these sorts of standard errors. So there's pretty much consistency of everything from the stratified estimator matching. 668 01:18:25.510 --> 01:18:28.800 David Stephens, Prof: Propensity score 3 IPW and A IPW. 669 01:18:29.030 --> 01:18:33.370 David Stephens, Prof: So that's… Giving an idea of what the correct answer is in this case. 670 01:18:34.400 --> 01:18:36.459 David Stephens, Prof: Okay? So, 671 01:18:37.420 --> 01:18:52.960 David Stephens, Prof: I think it's better if… that's… and having shown you that analysis, I think that's something that you can replicate in your own analyses. You can work through these examples, and if you've got any questions on that, you can post them to me on the Slack channel, or… 672 01:18:53.130 --> 01:18:58.029 David Stephens, Prof: By email directly to my email address. 673 01:19:02.070 --> 01:19:05.309 David Stephens, Prof: Okay, so, back to the slides for the last 10 minutes. 674 01:19:05.870 --> 01:19:12.380 David Stephens, Prof: And let's think about some further extensions. So… 675 01:19:12.490 --> 01:19:20.990 David Stephens, Prof: One extension I mentioned yesterday is moving from the cross-sectional study to the multiple time point study. 676 01:19:21.870 --> 01:19:26.040 David Stephens, Prof: This is where we have… Longitudinal measurements. 677 01:19:26.710 --> 01:19:29.379 David Stephens, Prof: Multiple treatments at different time points. 678 01:19:29.700 --> 01:19:36.490 David Stephens, Prof: Maybe intermediate outcomes, certainly… intermediate predictor measurements. 679 01:19:36.720 --> 01:19:39.680 David Stephens, Prof: And then maybe a terminal outcome that we're interested in. 680 01:19:41.810 --> 01:19:44.399 David Stephens, Prof: So, in this kind of setting, we've got… 681 01:19:44.860 --> 01:19:57.089 David Stephens, Prof: more complicated calculations to worry about. We can think about the direct effect of exposure or treatment, or we can think about a cumulative effect, a total effect of treatment over the length of the series. 682 01:19:58.460 --> 01:20:02.270 David Stephens, Prof: We might have a complicated DAG structure that looks a bit like this. 683 01:20:02.450 --> 01:20:08.959 David Stephens, Prof: Right, we've got multiple Xs, multiple predictors, multiple stages of treatment. 684 01:20:09.090 --> 01:20:13.289 David Stephens, Prof: Multiple outcomes, intermediate outcomes, and then a terminal outcome. 685 01:20:14.050 --> 01:20:14.740 David Stephens, Prof: Right? 686 01:20:17.100 --> 01:20:25.769 David Stephens, Prof: And the difficulty with a DAG like this, or the difficulty with a DAG like this points out, is that it's much harder to interpret what the effect of Z1 is. 687 01:20:26.380 --> 01:20:34.409 David Stephens, Prof: Right? Z1 has a direct effect on Y1, but it also has an indirect effect on Y2, and on Y3. 688 01:20:37.250 --> 01:20:43.210 David Stephens, Prof: The effect of Z2, say, is there's a direct effect, but there's also a confounding effect. 689 01:20:45.940 --> 01:20:47.870 David Stephens, Prof: Confounded a factor X2. 690 01:20:48.140 --> 01:20:53.040 David Stephens, Prof: So there's multiple, now more complicated paths. So we might have 691 01:20:53.210 --> 01:20:57.749 David Stephens, Prof: The problem of time-varying confounding. We might have the problem of mediation. 692 01:20:59.140 --> 01:21:04.009 David Stephens, Prof: And it turns out to be not quite so easy to decide what paths to block. 693 01:21:04.280 --> 01:21:07.679 David Stephens, Prof: Or how to block those paths in that kind of graph. 694 01:21:12.350 --> 01:21:17.769 David Stephens, Prof: Now, there are multivariate versions of the Propensity score that exist to allow you to study direct effects. 695 01:21:18.020 --> 01:21:22.029 David Stephens, Prof: And they come down to essentially the same kind of modeling that we had previously. 696 01:21:22.960 --> 01:21:31.799 David Stephens, Prof: Essentially, what we have to do is imagine a series of treatments, so basically this is just the notation that I'm using, so a tilde means history up to that point. 697 01:21:32.200 --> 01:21:39.939 David Stephens, Prof: We have a… A series of treatments, and a corresponding series of potential outcomes. 698 01:21:41.180 --> 01:21:44.300 David Stephens, Prof: That account for the different possible treatment patterns. 699 01:21:48.240 --> 01:21:52.549 David Stephens, Prof: The elementary way of constructing a balance in school, the multivariate 700 01:21:52.710 --> 01:21:55.260 David Stephens, Prof: Generalized Propensity score is to look at 701 01:21:56.650 --> 01:22:01.929 David Stephens, Prof: A series of models for the treatment, given the history up to that point. 702 01:22:02.380 --> 01:22:05.850 David Stephens, Prof: So the conditional model for treatment at time 703 01:22:06.940 --> 01:22:14.829 David Stephens, Prof: J, say, given history of treatment and other variables up to that point, up to J-1. 704 01:22:15.920 --> 01:22:22.780 David Stephens, Prof: And correcting using this balancing score, this, generalized Propensity score, 705 01:22:24.380 --> 01:22:26.560 David Stephens, Prof: Kind of allows to get a direct effect. 706 01:22:27.980 --> 01:22:44.619 David Stephens, Prof: Again, the same assumptions that we required before are still present here. We need known measured confounders, we need positivity, and this allows us to study the direct effect of treatment based on adjustment using this propensity score. So really, for the direct effect, nothing changes compared to the previous analysis. 707 01:22:46.410 --> 01:22:50.289 David Stephens, Prof: We can also use mixed model type strategies to account for 708 01:22:50.550 --> 01:22:55.839 David Stephens, Prof: The within-person dependence, so there's some work that's been done on this. 709 01:22:59.340 --> 01:23:11.059 David Stephens, Prof: And essentially what we're driving at, of course, is building some model for the entire collection of observations. So a mixed model, or a random effect model, is a natural way 710 01:23:11.800 --> 01:23:14.280 David Stephens, Prof: A common way that's used to model 711 01:23:14.710 --> 01:23:18.150 David Stephens, Prof: Longitudinal observations on the same individual. 712 01:23:18.980 --> 01:23:21.850 David Stephens, Prof: So we can deploy those methods in the causal setting. 713 01:23:22.320 --> 01:23:29.730 David Stephens, Prof: And do, adjustment using that, essentially internal balancing strategy. 714 01:23:31.340 --> 01:23:40.729 David Stephens, Prof: If we're looking at total effects, there's been a model developed called the marginal Structural Model, which is basically a longitudinal version of inverse probability weighting. 715 01:23:41.180 --> 01:23:54.990 David Stephens, Prof: So because we've only got a few minutes left, I won't go into a huge amount of detail on this, but this is just demonstrating that all the previous ideas of regression adjustment, propensity score adjustment, and inverse weighting can go across to this more complicated setting. 716 01:23:55.750 --> 01:24:08.049 David Stephens, Prof: So the marginal structural model is one where we have a series of X's, a series of Zs, and then a terminal outcome that we're interested in. We might have intermediate outcomes, but we'll fold those into the history. 717 01:24:09.770 --> 01:24:10.530 David Stephens, Prof: Okay? 718 01:24:11.500 --> 01:24:21.190 David Stephens, Prof: So the way that we think about this model is to say, well, okay, in light of… so our potential treatment pattern is going to give rise to an outcome. 719 01:24:23.120 --> 01:24:28.630 David Stephens, Prof: Let's model the outcome as some function of a summary of the treatments. 720 01:24:30.660 --> 01:24:33.960 David Stephens, Prof: Okay? So our so-called marginal model 721 01:24:34.630 --> 01:24:44.250 David Stephens, Prof: imagines a direct impact of that sum on the outcome. Here, I'm using a binary response, so I'm using a logistic-type structure, right? 722 01:24:44.510 --> 01:24:47.499 David Stephens, Prof: Our estimate of the causal effect 723 01:24:48.300 --> 01:24:50.860 David Stephens, Prof: will correspond to an estimation of psi. 724 01:24:51.900 --> 01:24:57.419 David Stephens, Prof: There are other versions of this model that have been used. This is the first one that was designed. 725 01:25:01.140 --> 01:25:08.899 David Stephens, Prof: So… We're faced with the same problem as before. We have data from an observational study. 726 01:25:09.280 --> 01:25:12.119 David Stephens, Prof: But our target is in the experimental world. 727 01:25:13.910 --> 01:25:20.230 David Stephens, Prof: So, inverse weighting is the tool we use to get around that calculation in this more complicated case. 728 01:25:20.720 --> 01:25:30.910 David Stephens, Prof: Okay, so there are the assumptions. If I take this, simple, 2 time point graph. 729 01:25:31.130 --> 01:25:46.050 David Stephens, Prof: just for illustration, then our task is to identify the impact of intervened upon Z1 and Z2 on Y, when in fact we have this time-varying confounding and mediation setting. 730 01:25:48.890 --> 01:25:51.929 David Stephens, Prof: Okay? So we consider treatment patterns. 731 01:25:52.250 --> 01:25:56.889 David Stephens, Prof: We're interested in getting at this kind of marginal effect. 732 01:25:57.300 --> 01:26:06.520 David Stephens, Prof: or marginal model. What is the expected response given a hypothetical treatment pattern? When we recognize that the data arrived 733 01:26:07.250 --> 01:26:12.080 David Stephens, Prof: Arrived with us, from the observational 734 01:26:12.530 --> 01:26:16.970 David Stephens, Prof: data generating distribution. So, the data arise from this. 735 01:26:17.330 --> 01:26:24.149 David Stephens, Prof: But actually, we want to consider what happens when all the arrows from the X's to the Zs have been broken. 736 01:26:25.700 --> 01:26:28.300 David Stephens, Prof: So, the marginal structure model 737 01:26:28.620 --> 01:26:39.449 David Stephens, Prof: So this explains why this is complicated. I will skip over this, leave you to read it in the interest of time, but basically it's saying that, 738 01:26:40.490 --> 01:26:46.229 David Stephens, Prof: Correction of the confounding by conditioning, is not trivial. 739 01:26:47.520 --> 01:26:49.770 David Stephens, Prof: Right? We've got to block the same paths. 740 01:26:50.160 --> 01:26:52.840 David Stephens, Prof: Or just confounding in a particular way. 741 01:26:53.990 --> 01:26:57.719 David Stephens, Prof: That goes beyond the simplest case… simpler cases that we saw before. 742 01:26:59.660 --> 01:27:06.529 David Stephens, Prof: So the solution of the marginal structural model is really pretty simple. It just, appeals to inverse probability weighting. 743 01:27:08.080 --> 01:27:16.070 David Stephens, Prof: of the outcomes, by… probabilities of treatment patterns. So rather than have 744 01:27:17.080 --> 01:27:31.550 David Stephens, Prof: a single inverse probability in the denominator here. We imagine all of those individuals that followed the target treatment pattern, and we re-weight those by the probability that they received those treatments. 745 01:27:33.000 --> 01:27:40.380 David Stephens, Prof: Okay? So this is a fairly simple extension to the ordinary IPW idea. The… 746 01:27:40.840 --> 01:27:48.560 David Stephens, Prof: The presence of the inverse weighting is again there to create this pseudosample and pseudopulation where the confounding is broken. 747 01:27:51.480 --> 01:27:56.479 David Stephens, Prof: I think I'll skip over the other Possible approaches here, 748 01:27:57.110 --> 01:27:59.430 David Stephens, Prof: You can use inverse weighting as, 749 01:27:59.560 --> 01:28:02.959 David Stephens, Prof: Through, weighted least squares and so on, that works pretty well. 750 01:28:03.110 --> 01:28:13.569 David Stephens, Prof: And then there are other… some other niceties, but I… in the interest of time, we've only got a couple of minutes left, so I'll leave you to read through those details. There's also an appendix of a real example 751 01:28:13.850 --> 01:28:17.340 David Stephens, Prof: Where I show the use of these methods in practice. 752 01:28:17.590 --> 01:28:22.660 David Stephens, Prof: The message I want to get across from that final part is that… 753 01:28:24.550 --> 01:28:30.959 David Stephens, Prof: In particular, inverse probability weighting can be extended to be used in this longitudinal case. 754 01:28:33.990 --> 01:28:37.610 David Stephens, Prof: To break the confounding in a similar way. 755 01:28:37.870 --> 01:28:39.579 David Stephens, Prof: For this longitudinal setting. 756 01:28:42.000 --> 01:28:56.030 David Stephens, Prof: So, very quickly to finish off, practical challenges. So, the classic challenges of statistical analysis are still present for all causal methods, and this includes, then, the estimation of the two components of the Propensity score adjustment. 757 01:28:59.870 --> 01:29:17.449 David Stephens, Prof: models, the outcome model and the propensity model. We might have missing data, we might have, truncation or, censoring by, censoring present in the data. We might have incomplete data through noncompliance. All of these practical challenges are going to come up. 758 01:29:17.550 --> 01:29:28.980 David Stephens, Prof: It turns out that we can really handle them in the same way that we would in the non-propensity score setting, we just have to account for informative missingness in the right way, for example. 759 01:29:30.890 --> 01:29:36.159 David Stephens, Prof: Datasets are large and complicated, getting larger and more complicated. 760 01:29:36.280 --> 01:29:39.039 David Stephens, Prof: As we go through the years, 761 01:29:40.840 --> 01:29:44.970 David Stephens, Prof: So more data is always good in statistics. 762 01:29:46.170 --> 01:29:48.939 David Stephens, Prof: It allows us to use more flexible models. 763 01:29:50.890 --> 01:29:58.559 David Stephens, Prof: But of course, what comes with, bigger data, larger data sets, larger dimensional data, is the difficulty of, 764 01:29:59.750 --> 01:30:00.810 David Stephens, Prof: computation. 765 01:30:03.020 --> 01:30:05.189 David Stephens, Prof: There are some examples at the bottom there. 766 01:30:06.540 --> 01:30:08.980 David Stephens, Prof: We've, 767 01:30:09.340 --> 01:30:17.490 David Stephens, Prof: throughout, stressed the importance of correct uncertainty representation. That becomes more complicated when we're bringing in data from different sources. 768 01:30:17.700 --> 01:30:28.330 David Stephens, Prof: So there's lots of ongoing work to do with causal estimation in the federated learning sense, so we get data coming in from different sources. Data has to be combined in an effective way. 769 01:30:28.480 --> 01:30:35.350 David Stephens, Prof: That's a challenge, again, of statistical methods generally, rather than particularly of causal inference, but there are some 770 01:30:35.640 --> 01:30:48.370 David Stephens, Prof: particular issues to do with covariate shifts and so on that we might have, combining databases from different sources that needs attention. There is some work ongoing in that field. 771 01:30:49.910 --> 01:30:53.109 David Stephens, Prof: And then finally some, 772 01:30:53.290 --> 01:30:58.310 David Stephens, Prof: where else might we use this kind of approach? Well, in the, 773 01:30:58.460 --> 01:31:10.509 David Stephens, Prof: construction of optimal treatment strategies. So I've worked quite a lot in the construction of optimal treatment strategies using propensity score, adjustment to perform, this… these constructions of 774 01:31:11.360 --> 01:31:16.079 David Stephens, Prof: Optimal strategies, single time point, or dynamic across time. 775 01:31:16.420 --> 01:31:23.550 David Stephens, Prof: Accounting for confounding in the observational data, using propensity score approaches. 776 01:31:23.940 --> 01:31:29.209 David Stephens, Prof: To define optimal strategies for future hypothetical experimental world. 777 01:31:30.040 --> 01:31:30.770 David Stephens, Prof: Okay. 778 01:31:31.060 --> 01:31:32.520 David Stephens, Prof: And, 779 01:31:32.990 --> 01:31:39.399 David Stephens, Prof: So there's also a lot of work on this. This is a big area of current research ongoing for the last… 780 01:31:41.440 --> 01:31:47.529 David Stephens, Prof: I guess 25 years, but still… still providing lots of interesting applications for the statisticians. 781 01:31:48.010 --> 01:31:51.660 David Stephens, Prof: Okay, so… Finally, conclusions. 782 01:31:52.500 --> 01:32:02.190 David Stephens, Prof: Well… I would make… basically these points. So, we need to do causal inference to answer important questions 783 01:32:03.250 --> 01:32:06.720 David Stephens, Prof: Concerning the impact of hypothetical exposures. 784 01:32:06.890 --> 01:32:11.039 David Stephens, Prof: And to do that, we can use causal graphs as an important tool. 785 01:32:15.000 --> 01:32:18.380 David Stephens, Prof: Sometimes we can get away with using standard methods like regression. 786 01:32:19.120 --> 01:32:21.649 David Stephens, Prof: But we have to be careful of the issue of confounding. 787 01:32:23.400 --> 01:32:29.400 David Stephens, Prof: So if confounding is present, and we have model misspecification, then ordinary regression methods will not give us the right answer. 788 01:32:31.090 --> 01:32:43.410 David Stephens, Prof: Balance is the key in the consideration of confounding, and the Propensity score is a tool to achieve balance. Now, we can deploy the Propensity score in a number of different ways. We can use matching, weighting. 789 01:32:43.930 --> 01:32:47.830 David Stephens, Prof: Or regression methods, and indeed combinations of those. 790 01:32:50.650 --> 01:32:54.640 David Stephens, Prof: We can think about flexible representations of the model components. 791 01:32:57.070 --> 01:33:01.059 David Stephens, Prof: We can use model selection with some care? 792 01:33:02.140 --> 01:33:08.520 David Stephens, Prof: And then… What we can hope to reflect is the magnitude and complexity of the observational data. 793 01:33:11.470 --> 01:33:17.810 David Stephens, Prof: So these are my collaborators that have helped me with this work. This is a field I've been working on over the last 20 years. 794 01:33:18.310 --> 01:33:21.290 David Stephens, Prof: The, 795 01:33:21.850 --> 01:33:35.430 David Stephens, Prof: My funding for my research is by these Canadian research councils. Just very quickly at the end, I put together a reading list of some key papers on different topics. You'll find all of these articles on the website. You can download them from there, or indeed. 796 01:33:35.790 --> 01:33:39.929 David Stephens, Prof: In a more honest fashion from the, publisher websites. 797 01:33:40.210 --> 01:33:45.389 David Stephens, Prof: There's the reading list, with all the… with some key, 798 01:33:45.710 --> 01:34:00.599 David Stephens, Prof: publications. If you've got any questions on this module, hopefully it's been useful to you. If you've got any more questions, there was a quick run-through there right at the end of the practical stuff, but I think that's stuff that you can go away yourself and look at the code for. If you've got any questions on that, after the 799 01:34:00.710 --> 01:34:01.770 David Stephens, Prof: module. 800 01:34:02.020 --> 01:34:05.289 David Stephens, Prof: please feel free to get in touch, and I'll answer what I can. 801 01:34:06.630 --> 01:34:07.710 David Stephens, Prof: Thank you very much.