Using Relational Databases in Network Forensic Analysis
--- FlowDB
Today, network intrusion detection systems (NIDSs) use custom solutions to log historical network flows and support forensic analysis by network administrators. The custom databases are usually based on log files, sometimes enhanced with a fixed number of indexes to speed up certain queries. These solutions are expensive to build and lack flexibility.
Relational database management systems (RDBMS), on the other hand, offer many features that are desirable in forensic analysis: a flexible and standard query lannguage (SQL), a powerful query optimizer, good support for indexes, etc.
The main reason why NIDS vendors are reluctant to use relational databases is that they consider them too general and bulky to be able to handle their workloads.
In FlowDB, we challenge this assumption and investigate how relational databases can support interactive network forensic analysis. Through benchmarking, we found that an "out-of-the-box" RDBMS can support flow rates from medium-size NIDS deployments in a manner that ensures high query performance. To enable support for significantly larger networks, we seek new techniques that would enable the database to handle both high data insert rates and good forensic query performance.
On-Demand View Materialization and Indexing (OVMI)
On-Demand View Materialization and Indexing (OVMI) is the first technique we developed. In our approach, when an event occurs, the system proactively extracts relevant historical data and indexes it in preparation for forensic queries over that data. The Figure below shows the setup.
Figure 1. On-demand view materialization for forensic analysis. The NIDS aggregates network traffic summaries (called flows) from routers within the network and stores them into a relational database. When suspicious activity is detected on the network, the NIDS fires an alarm to a network administrator. In OVMI, all alarms are also sent to the OVMI engine, that prepares the relevant data for the upcoming forensic queries. Data preparation involves first materializing the relevant data and then indexing it heavily.
We show that our approach significantly improves response times for a large class of queries, while maintaining high insert throughput.
Project Members
Graduate Students
Undergraduate Student
Faculty
Publications
- Roxana Geambasu, Tanya Bragin, Jaeyeon Jung. On-Demand View Materialization and Indexing for Network Forensic Analysis. NetDB '07, April 2007. [PDF, HTML]