Evaluating the impact of conservation programs is a challenge, as researchers and policy makers must estimate the difference between the post-program outcome we can observe and the unobservable, counterfactual outcome that would have existed in the absence of the program. In a randomized controlled trial, the counterfactual outcome can be estimated by measuring the outcome in the control group that is not exposed to the program. But when the program is not implemented as a randomized controlled trial, researchers must make an untestable assumption that the counterfactual outcome can be estimated by looking at what has happened in some unexposed areas. That assumption is often not very credible because the areas where conservation programs are placed are typically different, often in unobservable ways, from areas where the programs are not placed.
In an article in Conservation Biology, McConnachie, Romero, Ferraro and van Wilgen discuss this challenge and argue that a partial identification approach, rather than the conventional point-estimate approach, will improve the credibility and transparency of conservation impact evaluations. In the past, the standard approach has been to generate a single estimate of program impact with a confidence interval. In contrast, a partial identification approach estimates upper and lower bounds on the impact, starting with the weakest assumptions. These bounds are then narrowed using increasingly stronger, yet plausible, assumptions based upon which areas were ultimately selected for the program. This approach takes into account why each unit was selected for the intervention, and also how that particular unit would respond to the intervention.
The partial identification approach, compared to the standard point-estimate approach, encourages better understanding and accounting for treatment selection bias and provides a less controversial and more transparent estimate for stakeholders to agree upon. McConnachie et al. apply the approach in a case study of a program which removed invasive trees in the Cape Floristic Region in South Africa.
In conservation programs in which the program was not randomly assigned and in which not enough data were collected to isolate the causal impacts of the program using statistical methods, a partial identification approach allows policy-makers to still develop evidence-based programs based upon lessons learned from past conservation initiatives.