A tutorial on statistically sound pattern discovery

Wilhelmiina Hämäläinen*, Geoffrey I. Webb

*Corresponding author for this work

Research output: Contribution to journalArticleScientificpeer-review

8 Citations (Scopus)
62 Downloads (Pure)

Abstract

Statistically sound pattern discovery harnesses the rigour of statistical hypothesis testing to overcome many of the issues that have hampered standard data mining approaches to pattern discovery. Most importantly, application of appropriate statistical tests allows precise control over the risk of false discoveries—patterns that are found in the sample data but do not hold in the wider population from which the sample was drawn. Statistical tests can also be applied to filter out patterns that are unlikely to be useful, removing uninformative variations of the key patterns in the data. This tutorial introduces the key statistical and data mining theory and techniques that underpin this fast developing field. We concentrate on two general classes of patterns: dependency rules that express statistical dependencies between condition and consequent parts and dependency sets that express mutual dependence between set elements. We clarify alternative interpretations of statistical dependence and introduce appropriate tests for evaluating statistical significance of patterns in different situations. We also introduce special techniques for controlling the likelihood of spurious discoveries when multitudes of patterns are evaluated. The paper is aimed at a wide variety of audiences. It provides the necessary statistical background and summary of the state-of-the-art for any data mining researcher or practitioner wishing to enter or understand statistically sound pattern discovery research or practice. It can serve as a general introduction to the field of statistically sound pattern discovery for any reader with a general background in data sciences.

Original languageEnglish
Pages (from-to)325–377
Number of pages53
JournalData Mining and Knowledge Discovery
Volume33
Issue number2
DOIs
Publication statusPublished - 2019
MoE publication typeA1 Journal article-refereed

Keywords

  • Association rule
  • Dependency rule
  • Dependency set
  • Hypothesis testing
  • Pattern discovery
  • Statistical significance

Fingerprint Dive into the research topics of 'A tutorial on statistically sound pattern discovery'. Together they form a unique fingerprint.

Cite this