Smart Monitoring: Exploiting History in Continuous Monitoring Systems
---MOIRAE



Monitoring applications enable users to continuously observe the current state of a system and receive alerts when interesting events occur. For example, an administrator can monitor a cluster of computers, a computer network, the car traffic in some area, etc. In many situations, historical information about current events may help users address ongoing or imminent problems in the monitored system. However, providing timely historical information for real-time events is challenging because of the large volume of historical data.

In this project, we are building a new type of continuous monitoring system called Moirae. The goal of Moirae is to complement newly detected events with useful historical information in near-real-time. To achieve this goal, Moirae allows users to describe what constitutes the interesting context of an event. The system then delivers, for each new event, a set of k results derived from the most similar (in terms of given context) recent events.

Challenges

In the Moirae project, we are addressing the following challenges related to history-enhanced monitoring.

System Architecture

Moirae is a framework which tightly integrates a stream processing engine (e.g., Borealis), for continuous monitoring, and an RDBMS, for archiving historical information.

The main insight behind the design of Moirae is that users will be more interested in receiving a few relevant results soon after each new event (especially if these events are recent), rather than a complete set of results or the best results with higher latency. We thus proposed a system architecture based on hierarchical log partitioning and hierarchical query execution, where the recent past is stored at a higher cost, but can be queried faster than older data.

Project Members

Project Alumni

Publications

  1. Shengliang Xu and Magdalena Balazinska: Sensor Data Stream Exploration for Monitoring Applications DMSN 2011 [PDF]
  2. Prasang Upadhyaya, YongChul Kwon, and Magdalena Bamlazinska: A Latency and Fault-Tolerance Optimizer for Online Parallel Data Processing SIGMOD 2011 [PDF]
  3. YongChul Kwon, Magdalena Balazinska, and Albert Greenberg: Fault-tolerant Stream Processing using a Distributed, Replicated File System VLDB 2008 [PDF]
  4. YongChul Kwon, Wing Yee Lee, Magdalena Balazinska, and Guiping Xu: Clustering Events on Streams using Complex Context Information Mining Complex Data 2008 [PDF]
  5. Magdalena Balazinska, YongChul Kwon, Nathan Kuchta, and Dennis Lee: Moirae: History-Enhanced Monitoring. CIDR 2007 [PDF]

Acknowledgments

The Moirae project is partially supported by NSF grant IIS-0713123, NSF CRI grant CNS-0454425, a gift from Cisco Systems Inc., a Mitre contract, and Balazinska's Microsoft Research New Faculty Fellowship.

Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

Last Modified at $Id: index.htm 4456 2008-09-08 03:08:43Z yongchul $