On methods for correcting for the look-elsewhere effect in searches for new physics

Sara Algeri, D. A. Van Dyk, J. Conrad, B. Anderson

Research output: Contribution to journalArticle

3 Citations (Scopus)

Abstract

The search for new significant peaks over a energy spectrum often involves a statistical multiple hypothesis testing problem. Separate tests of hypothesis are conducted at different locations over a fine grid producing an ensemble of local p-values, the smallest of which is reported as evidence for the new resonance. Unfortunately, controlling the false detection rate (type I error rate) of such procedures may lead to excessively stringent acceptance criteria. In the recent physics literature, two promising statistical tools have been proposed to overcome these limitations. In 2005, a method to "find needles in haystacks" was introduced by Pilla et al. [1], and a second method was later proposed by Gross and Vitells [2] in the context of the "look-elsewhere effect" and trial factors. We show that, although the two methods exhibit similar performance for large sample sizes, for relatively small sample sizes, the method of Pilla et al. leads to an artificial inflation of statistical power that stems from an increase in the false detection rate. This method, on the other hand, becomes particularly useful in multidimensional searches, where the Monte Carlo simulations required by Gross and Vitells are often unfeasible. We apply the methods to realistic simulations of the Fermi Large Area Telescope data, in particular the search for dark matter annihilation lines. Further, we discuss the counter-intuitive scenario where the look-elsewhere corrections are more conservative than much more computationally efficient corrections for multiple hypothesis testing. Finally, we provide general guidelines for navigating the tradeoffs between statistical and computational efficiency when selecting a statistical procedure for signal detection.

Original languageEnglish (US)
Article numberP12010
JournalJournal of Instrumentation
Volume11
Issue number12
DOIs
StatePublished - Dec 14 2016
Externally publishedYes

Fingerprint

Physics
physics
Signal detection
Testing
Computational efficiency
Telescopes
Needles
Multiple Hypothesis Testing
signal detection
tradeoffs
Gross
stems
needles
acceptability
dark matter
counters
energy spectra
simulation
grids
telescopes

Cite this

On methods for correcting for the look-elsewhere effect in searches for new physics. / Algeri, Sara; Van Dyk, D. A.; Conrad, J.; Anderson, B.

In: Journal of Instrumentation, Vol. 11, No. 12, P12010, 14.12.2016.

Research output: Contribution to journalArticle

@article{9705f46879d7457da3a95909edb02e97,
title = "On methods for correcting for the look-elsewhere effect in searches for new physics",
abstract = "The search for new significant peaks over a energy spectrum often involves a statistical multiple hypothesis testing problem. Separate tests of hypothesis are conducted at different locations over a fine grid producing an ensemble of local p-values, the smallest of which is reported as evidence for the new resonance. Unfortunately, controlling the false detection rate (type I error rate) of such procedures may lead to excessively stringent acceptance criteria. In the recent physics literature, two promising statistical tools have been proposed to overcome these limitations. In 2005, a method to {"}find needles in haystacks{"} was introduced by Pilla et al. [1], and a second method was later proposed by Gross and Vitells [2] in the context of the {"}look-elsewhere effect{"} and trial factors. We show that, although the two methods exhibit similar performance for large sample sizes, for relatively small sample sizes, the method of Pilla et al. leads to an artificial inflation of statistical power that stems from an increase in the false detection rate. This method, on the other hand, becomes particularly useful in multidimensional searches, where the Monte Carlo simulations required by Gross and Vitells are often unfeasible. We apply the methods to realistic simulations of the Fermi Large Area Telescope data, in particular the search for dark matter annihilation lines. Further, we discuss the counter-intuitive scenario where the look-elsewhere corrections are more conservative than much more computationally efficient corrections for multiple hypothesis testing. Finally, we provide general guidelines for navigating the tradeoffs between statistical and computational efficiency when selecting a statistical procedure for signal detection.",
author = "Sara Algeri and {Van Dyk}, {D. A.} and J. Conrad and B. Anderson",
year = "2016",
month = "12",
day = "14",
doi = "10.1088/1748-0221/11/12/P12010",
language = "English (US)",
volume = "11",
journal = "Journal of Instrumentation",
issn = "1748-0221",
publisher = "IOP Publishing Ltd.",
number = "12",

}

TY - JOUR

T1 - On methods for correcting for the look-elsewhere effect in searches for new physics

AU - Algeri, Sara

AU - Van Dyk, D. A.

AU - Conrad, J.

AU - Anderson, B.

PY - 2016/12/14

Y1 - 2016/12/14

N2 - The search for new significant peaks over a energy spectrum often involves a statistical multiple hypothesis testing problem. Separate tests of hypothesis are conducted at different locations over a fine grid producing an ensemble of local p-values, the smallest of which is reported as evidence for the new resonance. Unfortunately, controlling the false detection rate (type I error rate) of such procedures may lead to excessively stringent acceptance criteria. In the recent physics literature, two promising statistical tools have been proposed to overcome these limitations. In 2005, a method to "find needles in haystacks" was introduced by Pilla et al. [1], and a second method was later proposed by Gross and Vitells [2] in the context of the "look-elsewhere effect" and trial factors. We show that, although the two methods exhibit similar performance for large sample sizes, for relatively small sample sizes, the method of Pilla et al. leads to an artificial inflation of statistical power that stems from an increase in the false detection rate. This method, on the other hand, becomes particularly useful in multidimensional searches, where the Monte Carlo simulations required by Gross and Vitells are often unfeasible. We apply the methods to realistic simulations of the Fermi Large Area Telescope data, in particular the search for dark matter annihilation lines. Further, we discuss the counter-intuitive scenario where the look-elsewhere corrections are more conservative than much more computationally efficient corrections for multiple hypothesis testing. Finally, we provide general guidelines for navigating the tradeoffs between statistical and computational efficiency when selecting a statistical procedure for signal detection.

AB - The search for new significant peaks over a energy spectrum often involves a statistical multiple hypothesis testing problem. Separate tests of hypothesis are conducted at different locations over a fine grid producing an ensemble of local p-values, the smallest of which is reported as evidence for the new resonance. Unfortunately, controlling the false detection rate (type I error rate) of such procedures may lead to excessively stringent acceptance criteria. In the recent physics literature, two promising statistical tools have been proposed to overcome these limitations. In 2005, a method to "find needles in haystacks" was introduced by Pilla et al. [1], and a second method was later proposed by Gross and Vitells [2] in the context of the "look-elsewhere effect" and trial factors. We show that, although the two methods exhibit similar performance for large sample sizes, for relatively small sample sizes, the method of Pilla et al. leads to an artificial inflation of statistical power that stems from an increase in the false detection rate. This method, on the other hand, becomes particularly useful in multidimensional searches, where the Monte Carlo simulations required by Gross and Vitells are often unfeasible. We apply the methods to realistic simulations of the Fermi Large Area Telescope data, in particular the search for dark matter annihilation lines. Further, we discuss the counter-intuitive scenario where the look-elsewhere corrections are more conservative than much more computationally efficient corrections for multiple hypothesis testing. Finally, we provide general guidelines for navigating the tradeoffs between statistical and computational efficiency when selecting a statistical procedure for signal detection.

UR - http://www.scopus.com/inward/record.url?scp=85008163164&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85008163164&partnerID=8YFLogxK

U2 - 10.1088/1748-0221/11/12/P12010

DO - 10.1088/1748-0221/11/12/P12010

M3 - Article

VL - 11

JO - Journal of Instrumentation

JF - Journal of Instrumentation

SN - 1748-0221

IS - 12

M1 - P12010

ER -