Thursday, November 08, 2007

Snort 3.0 Architecture Series Part 1: Overview

Snort 3.0 is the next generation Snort engine that is currently under active development at Sourcefire. I have been acting as lead architect as well as a contributing developer on the project for many months now. As one of the people who's driving development of the system I thought it would be worthwhile to start talking about what we're building because I know a lot of people are interested in learning more about this next generation Snort engine.

Snort is 9 years old this month and has a lot of miles under its belt. It's one of the most widely deployed network security technologies in the world and is therefore one of the most highly exposed (in terms of live network packets processed) and well tested IDP code bases available today. Snort 3.0 is a huge undertaking but I feel its a worthwhile effort to achieve some of the long term goals that we have for the engine. I believe that ultimately our users will benefit tremendously from the design of the new engine and that it will be a platform that will work well for the at least the next 9 years.

There are several goals associated with Snort 3.0's development:

0) Rewrite the core frameworks for Snort from the ground up to clean out code base cruft and leverage external libraries where possible to contain the scope of the rewrite. This will also allow us to flush unused features and effectively reduce the size and complexity of the code base making it easier to extend and ultimately lending the security benefits of a smaller code base.

1) Build an "contextually aware engine", one that has the ability to understand what it's defending built around the concept of network context. Network context is essentially data about the environment that is being defended by Snort, the composition of the hosts in the network as well as the local network composition. This is important in Snort 3 in order to:
  • Reduce/simplify/eliminate tuning as much as possible leveraging network context.

  • Generate event priorities based on network context.

  • Address network and transport layer evasion by leveraging network context.

2) Abstract and compartmentalize Snort's subsystems to make components "separable".
  • Compartmentalize common functionality that any network traffic analysis application would need to enable Snort to be a more effective platform for building arbitrary network traffic analyzers.

3) Improve support for protocol encapsulation within the overall engine architecture to make handling things like enterprise/WAN protocols and IPv6 support more natural.

4) Add an interactive shell to the system so that it may be more fully orchestrated at runtime.

5) Multithread the engine to take better advantage of multi-core platforms that are standard today.
  • Make the engine parallalizable so that multiple analytics threads may run simultaneously on the same traffic.

6) Normalize Snort's language so that it's easier to read and write.


As a result of these goals the engine architecture has a number of major discrete software components.

1) Data Source. The Data Source component encapsulates common functionality required by any network traffic analyzer, functions that will have to be performed prior to running almost any analysis task. The data source incorporates a number of components including:
  • Data Acquisition (DAQ) - The DAQ provides an interface between the rest of the engine and the host OS packet facilities. This is where we get packets from the underlying hardware and where we talk to that hardware regarding the disposition of those packets. The DAQ subsystem allows Snort 3.0 to incorporate arbitrary external packet interfaces including things like libpcap, IPQ and divert sockets.

  • Decoder - The Decoder performs basically the same tasks it did in Snort 2.x. Validate the packets, detect protocol anomalies and provide a referential structure for the rest of the program to operate upon.

  • Flow Manager - The Flow Manager provides services for tracking conversations between endpoints on the network. In Snort 3.0 it also contains features for "fastpathing" traffic, allowing it to pass straight through the engine in the event that the analytics modules have decided that they're not interested in a particular flow any longer. Snort 3.0 also includes a mechanism called "flow slots" that subsystems can use to store stateful "flow local" information. This is the place that things like flowbits will get stored in Snort 3.0.

  • IP Defragmenter - This module provides services for putting IPv4 and IPv6 packets back together and will include mechanisms to allow for target-based fragment reassembly.

  • TCP Stream Reassembler - As with the IP Defragmenter, provides target-based services for reassembling TCP segments into normalized streams and presenting them to the underlying analytics.

  • Data Source API - An abstraction API between the facilities provided by the data source and the rest of the Snort 3.0 software framework. This API exists to that the rest of Snort 3.0 can work without caring whether the Data Source is implemented as hardware or software.

2) Action System. The Action System handles event queuing, notification and logging when the system fires events. The supported output types in Snort 3.0 will be text (console), syslog and Unified 2, a serialized binary stream format.

3) Attribute Management System (AMS). The AMS will store network contextual data about the operational environment being defended by a particular Snort instance. This subsystem will be addressable continuously at runtime and provide interactive interfaces to the command shell as well as analytics modules that can leverage its data. The inclusion of the AMS will make all the goals in section 1 of the goals section attainable.

4) Analytics System. The Analytics System is where Snort detection engine threads will be located. The idea in Snort 3.0 is to put all detection logic in analytics modules that run as separate threads, all the other code exists to support the functions of the Analytics System. Multiple threads may operate on the data coming from a dispatcher instance simultaneously. The Analytics System is structured so that all interaction between the analytics modules and the rest of the Snort 3.0 framework is brokered by an API called the "Snort Abstraction Layer" (SAL). Note that arbitrary functions may be performed by analytics modules in a given runtime instance. For example the Sourcefire edition of this engine is going to include RNA's functionality as an analytics module running side-by-side with Snort 3.0.

5) Dispatcher. The Dispatcher exists to coordinate information flow between the different components of Snort 3.0 and to manage traffic queuing and disposition across analytics threads. It also ties together all of the objects in a runtime instance of a Snort engine, uniting data source, analytics, action system and the attribute manager into a single manageable entity for the purposes of process and threat management from the command shell.

6) Snortd and the command shell. Snortd is the daemon process that provides marshaling services for the objects that are instantiated in a particular framework instance. The command shell also runs attached within a thread to snortd. The command shell provides interactive object management services for different software modules, runtime management of the process and threads, health management and a full scripting language for Snort 3.0. The command shell is running the Lua scripting language, a lightweight embeddable scripting language that is fast and portable as well as being very nice for implementing Domain Specific Languages. If Snort's parser wasn't one of your favorite features in the past you should definitely like this change! For those of you wondering if Snort 3.0 will handle Snort's existing rules language, of course it will. We're not planning on throwing out 9 years of accumulated detection functionality!

The analytics modules that are under development right now (that I can discuss) include a Snort 2.x detection engine implementation, RNA (for Sourcefire implementations) and a Lua traffic analysis module for users who are in environments where a scripting interface to traffic analysis would be very useful.

Over the coming days and weeks I am planning to post a subsystem by subsystem design overview of the engine components so that users of the system may familiarize themselves with the system as we prepare to release additional alpha code snapshots on our way to a Snort 3.0 beta!

Technorati Tags:
, , , ,

Labels: , , ,


At 3:34 PM, Anonymous Anonymous said...

Sweet. Any ideas on when you plan on releasing the final version? Also, will it allow users to more finely select the IPs instead of resorting to subnet addresses in rules and in the config file? Thanks

At 8:44 PM, Blogger Martin Roesch said...

We're hoping to get a beta out around the end of the year or perhaps early in 2008. Quite a bit of progress has been made over the past several months as more people from Sourcefire have been working on the code. The anticipated initial GA release is currently slated for the first half of 2008, we're just not narrowing it down more than that right now.

At 12:07 AM, Blogger geek00l said...

Hi Roesch,

That's what I like to read about, more snort 3.0 coverage ;]


At 10:04 PM, Blogger Joe said...

I know it's early to ask, but are there going to be any improvements or changes to snort's output methods? Is barnyard getting an update or replacement?

At 10:51 AM, Blogger Martin Roesch said...

Joe: Part 7 or 8 (or whatever) of this series will cover output architecture. The planned output methods for 3.0 currently stand at text/console, syslog and Unified 2 (as seen in Snort 2.8). Barnyard will either be updated or replaced to process its output.

At 1:15 PM, Blogger Raul Siles said...

Hi Martin,
Although you're covering the architecture here, and this is a more specific question, any plans to provide native and full support for 802.11 frames (mgmt., control & data) in Snort 3.0? Additionally, what about VoIP support (I know some SIP signatures were added to Snort 2.8)?

Thanks, and keep up the great work!

At 11:53 AM, Blogger Martin Roesch said...

Hi Raul,

The decoder infrastructure in 3.0 is more modular and somewhat easier to extend and there's no reason we can't do 802.11 management frames or SIP protocol decoders. We don't have any immediate plans to implement them but I know we'll at least get to SIP in the nearer term as that's something a lot of people are very interested in these days.

At 12:29 PM, Anonymous Anonymous said...

Most people really want a UTM, which has IPS, A/V, Firewall, etc. Linux kernel (netfilter) provides a very powerful tool for Firewall. It also has the flow concept including the TCP state machine. I wonder why Snort does not integrate into Linux kernel. If so, it will provide a complete UTM solution. It is not that difficult.

At 2:18 PM, Blogger Martin Roesch said...

Snort is a cross-platform engine, integrating tightly with the Linux kernel doesn't allow us to remain that way very well. Additionally, while a lot of people are interested in UTM, there are still lots that aren't and don't feel the concept has a lot of value for them. The platform that is being built for Snort 3.0 is actually pretty well suited to being a major component of a UTM if the appropriate analysis engines are written. I think eventually that'll be a natural part of what Snort can do but my goal here hasn't been to set out to write a UTM, it's to write an excellent platform for network traffic analysis applications and then to populate that platform with great analysis apps.

At 6:26 PM, Blogger David said...

Snort 3.0 looks really cool, I was about to start working on an output plugin for snort 2.8 until I found out about the 3.0.. who doesn't want to be on the bleeding edge? :)

Anyway, any chance we could get a part two to the architecture series?

The main thing I'm curious about is how to get the raw packet data inside of an output plugin. Any hints/tips would be greatly appreciated.


At 4:17 AM, Anonymous Anonymous said...

Good idea.
I have seen the 3.0beta .
It seems less document for the code :)

At 7:21 AM, Anonymous Anonymous said...

Hi Martin,
I have some problems to understand concept-to-code of Snort 3.0.

It will be nice if you put an example of the data flow diagram.

For example, I'm reviewing the code and I see this flow.

network interface --> pcap daq --> decoder --> analysys func

┬┐Where acts dispatcher? I see a structures like traffic queue in dispatcher, but when is this filled?. The flow is always throught dispatcher?

network interface -> pcap_daq --> decoder --> dispatcher queue --> analysis func ?
if there is an alert the flow is..

network interface -> pcap_daq --> decoder --> dispatcher queue --> analysis func --> alert to dispatcher --> Action System

Thanks in advance

At 4:03 AM, Anonymous Anonymous said...

I recently came across your blog and have been reading along. I thought I would leave my first comment. I don't know what to say except that I have enjoyed reading. Nice blog. I will keep visiting this blog very often.


At 10:51 PM, Anonymous Intrusion Prevention System said...

Sourcefire is now transforming into a complete company especially with their purchase of the opensource antivirus ClamAV.


Post a Comment

<< Home