Since model selection methods choose the "best" model in some sense, significance tests for variables in that model will tend to be anti-conservative, and goodness of fit tests will tend to be conservative. This is troubling, as it implies these tests in practice do not actually provide evidence in favor of the chosen variables or model. We demonstrate methods of post-selection inference to obtain conditionally valid significance tests and conditionally unbiased goodness of fit tests and show how these outperform unadjusted tests.
Joshua Loftus is an Assistant Professor of Statistics at New York University and an affiliate of the NYU Center for Data Science. Before joining NYU, he was a Research Fellow at the Alan Turing Institute and Postdoctoral Researcher at the University of Cambridge. He earned a Ph.D. in Statistics at Stanford University.