Snort 3.0 Architecture Series Part 1: Overview

Snort 3.0 is the next generation Snort engine that is currently under active development at Sourcefire. I have been acting as lead architect as well as a contributing developer on the project for many months now. As one of the people who's driving development of the system I thought it would be worthwhile to start talking about what we're building because I know a lot of people are interested in learning more about this next generation Snort engine.

Snort is 9 years old this month and has a lot of miles under its belt. It's one of the most widely deployed network security technologies in the world and is therefore one of the most highly exposed (in terms of live network packets processed) and well tested IDP code bases available today. Snort 3.0 is a huge undertaking but I feel its a worthwhile effort to achieve some of the long term goals that we have for the engine. I believe that ultimately our users will benefit tremendously from the design of the new engine and that it will be a platform that will work well for the at least the next 9 years.

There are several goals associated with Snort 3.0's development:

0) Rewrite the core frameworks for Snort from the ground up to clean out code base cruft and leverage external libraries where possible to contain the scope of the rewrite. This will also allow us to flush unused features and effectively reduce the size and complexity of the code base making it easier to extend and ultimately lending the security benefits of a smaller code base.

1) Build an "contextually aware engine", one that has the ability to understand what it's defending built around the concept of network context. Network context is essentially data about the environment that is being defended by Snort, the composition of the hosts in the network as well as the local network composition. This is important in Snort 3 in order to:

• Reduce/simplify/eliminate tuning as much as possible leveraging network context.


• Generate event priorities based on network context.


• Address network and transport layer evasion by leveraging network context.

2) Abstract and compartmentalize Snort's subsystems to make components "separable".

• Compartmentalize common functionality that any network traffic analysis application would need to enable Snort to be a more effective platform for building arbitrary network traffic analyzers.

3) Improve support for protocol encapsulation within the overall engine architecture to make handling things like enterprise/WAN protocols and IPv6 support more natural.

4) Add an interactive shell to the system so that it may be more fully orchestrated at runtime.

5) Multithread the engine to take better advantage of multi-core platforms that are standard today.

• Make the engine parallalizable so that multiple analytics threads may run simultaneously on the same traffic.

6) Normalize Snort's language so that it's easier to read and write.

As a result of these goals the engine architecture has a number of major discrete software components.

1) Data Source. The Data Source component encapsulates common functionality required by any network traffic analyzer, functions that will have to be performed prior to running almost any analysis task. The data source incorporates a number of components including:

• Data Acquisition (DAQ) - The DAQ provides an interface between the rest of the engine and the host OS packet facilities. This is where we get packets from the underlying hardware and where we talk to that hardware regarding the disposition of those packets. The DAQ subsystem allows Snort 3.0 to incorporate arbitrary external packet interfaces including things like libpcap, IPQ and divert sockets.


• Decoder - The Decoder performs basically the same tasks it did in Snort 2.x. Validate the packets, detect protocol anomalies and provide a referential structure for the rest of the program to operate upon.


• Flow Manager - The Flow Manager provides services for tracking conversations between endpoints on the network. In Snort 3.0 it also contains features for "fastpathing" traffic, allowing it to pass straight through the engine in the event that the analytics modules have decided that they're not interested in a particular flow any longer. Snort 3.0 also includes a mechanism called "flow slots" that subsystems can use to store stateful "flow local" information. This is the place that things like flowbits will get stored in Snort 3.0.


• IP Defragmenter - This module provides services for putting IPv4 and IPv6 packets back together and will include mechanisms to allow for target-based fragment reassembly.


• TCP Stream Reassembler - As with the IP Defragmenter, provides target-based services for reassembling TCP segments into normalized streams and presenting them to the underlying analytics.


• Data Source API - An abstraction API between the facilities provided by the data source and the rest of the Snort 3.0 software framework. This API exists to that the rest of Snort 3.0 can work without caring whether the Data Source is implemented as hardware or software.

2) Action System. The Action System handles event queuing, notification and logging when the system fires events. The supported output types in Snort 3.0 will be text (console), syslog and Unified 2, a serialized binary stream format.

3) Attribute Management System (AMS). The AMS will store network contextual data about the operational environment being defended by a particular Snort instance. This subsystem will be addressable continuously at runtime and provide interactive interfaces to the command shell as well as analytics modules that can leverage its data. The inclusion of the AMS will make all the goals in section 1 of the goals section attainable.

4) Analytics System. The Analytics System is where Snort detection engine threads will be located. The idea in Snort 3.0 is to put all detection logic in analytics modules that run as separate threads, all the other code exists to support the functions of the Analytics System. Multiple threads may operate on the data coming from a dispatcher instance simultaneously. The Analytics System is structured so that all interaction between the analytics modules and the rest of the Snort 3.0 framework is brokered by an API called the "Snort Abstraction Layer" (SAL). Note that arbitrary functions may be performed by analytics modules in a given runtime instance. For example the Sourcefire edition of this engine is going to include RNA's functionality as an analytics module running side-by-side with Snort 3.0.

5) Dispatcher. The Dispatcher exists to coordinate information flow between the different components of Snort 3.0 and to manage traffic queuing and disposition across analytics threads. It also ties together all of the objects in a runtime instance of a Snort engine, uniting data source, analytics, action system and the attribute manager into a single manageable entity for the purposes of process and threat management from the command shell.

6) Snortd and the command shell. Snortd is the daemon process that provides marshaling services for the objects that are instantiated in a particular framework instance. The command shell also runs attached within a thread to snortd. The command shell provides interactive object management services for different software modules, runtime management of the process and threads, health management and a full scripting language for Snort 3.0. The command shell is running the Lua scripting language, a lightweight embeddable scripting language that is fast and portable as well as being very nice for implementing Domain Specific Languages. If Snort's parser wasn't one of your favorite features in the past you should definitely like this change! For those of you wondering if Snort 3.0 will handle Snort's existing rules language, of course it will. We're not planning on throwing out 9 years of accumulated detection functionality!

The analytics modules that are under development right now (that I can discuss) include a Snort 2.x detection engine implementation, RNA (for Sourcefire implementations) and a Lua traffic analysis module for users who are in environments where a scripting interface to traffic analysis would be very useful.

Over the coming days and weeks I am planning to post a subsystem by subsystem design overview of the engine components so that users of the system may familiarize themselves with the system as we prepare to release additional alpha code snapshots on our way to a Snort 3.0 beta!

Technorati Tags:
intrusion detection, intrusion prevention, Open Source, programming, Snort 3.0

Daemonlogger 1.0 released

Daemonlogger 1.0 is available on my user page on snort.org. It's got a couple new features but nothing major, if you're a Daemonlogger fan it's definitely worth a download!