Air pollution causes millions of premature deaths annually, driving widespread implementation of clean air interventions. Quantitative evaluation of the efficacy of such interventions is critical in air quality management. Machine learning-based weather normalization (ML-WN) has been employed to isolate meteorological influences from emission-drive changes; however, it has its own limitations, particularly when abrupt emission shifts occur, e.g., after an intervention. Here we developed a logical evaluation framework, based on paired observational datasets and a test of “ML algebra” (i.e., the “commutation” of a normalisation step), to show that ML-WN significantly underestimates the immediate effects of short-term interventions on nitrogen oxides (NOx), with discrepancies reaching up to 42 % for 1 week interventions. This finding challenges assumptions about the robustness of ML-WN for evaluating short-term policies, such as emergency traffic controls or episodic pollution events. We propose a refined approach (MacLeWN) that can reduce such underestimation biases by > 90 % in idealised but plausible cases studies. We applied both approaches to evaluate the impact of COVID-19 lockdown on NOx as measured at Marylebone Road, London. For the 1 week period after the lockdown, ML-WN estimates approximately 17 % smaller NOx reductions compared to MacLeWN, and such underestimation diminishes as policy duration extends, decreasing to ∼ 10 % for 1 month and becoming insignificant after 3 months. Our findings indicate the importance of carefully selecting evaluation methodologies for air quality interventions, suggesting that ML-WN should be complemented or adjusted when assessing short-term policies. Increasing model interpretability is also crucial for generating trustworthy assessments and improving policy evaluations.
CORE (COnnecting REpositories)
CORE (COnnecting REpositories)