Specious rules: An efficient and effective unifying method for removing misleading and uninformative patterns in association rule mining

Wilhelmiina Hämäläinen, Geoffrey I. Webb

Research output: Chapter in Book/Report/Conference proceedingConference contributionScientificpeer-review

3 Citations (Scopus)
19 Downloads (Pure)

Abstract

We present theoretical analysis and a suite of tests and procedures for addressing a broad class of redundant and misleading association rules we call specious rules. Specious dependencies, also known as spurious, apparent, or illusory associations, refer to a well-known phenomenon where marginal dependencies are merely products of interactions with other variables and disappear when conditioned on those variables. The most extreme example is Yule-Simpson's paradox where two variables present positive dependence in the marginal contingency table but negative in all partial tables defined by different levels of a confounding factor. It is accepted wisdom that in data of any nontrivial dimensionality it is infeasible to control for all of the exponentially many possible confounds of this nature. In this paper, we consider the problem of specious dependencies in the context of statistical association rule mining. We define specious rules and show they offer a unifying framework which covers many types of previously proposed redundant or misleading association rules. After theoretical analysis, we introduce practical algorithms for detecting and pruning out specious association rules efficiently under many key goodness measures, including Mutual information and exact hypergeometric probabilities. We demonstrate that the procedure greatly reduces the number of associations discovered, providing an elegant and effective solution to the problem of association mining discovering large numbers of misleading and redundant rules.

Original languageEnglish
Title of host publicationProceedings of the 17th SIAM International Conference on Data Mining, SDM 2017
Pages309-317
Number of pages9
ISBN (Electronic)9781611974874
DOIs
Publication statusPublished - 2017
MoE publication typeA4 Article in a conference publication
EventSIAM International Conference on Data Mining - Houston, United States
Duration: 27 Apr 201729 Apr 2017
Conference number: 17

Conference

ConferenceSIAM International Conference on Data Mining
Abbreviated titleSDM
CountryUnited States
CityHouston
Period27/04/201729/04/2017

Keywords

  • Association rule
  • Birch's test
  • Mutual information
  • Specious dependency
  • Yule-Simpson's paradox

Fingerprint Dive into the research topics of 'Specious rules: An efficient and effective unifying method for removing misleading and uninformative patterns in association rule mining'. Together they form a unique fingerprint.

  • Cite this