Anomaly detection Engine for Linux Logs (ADE)

Overview

ADE detects anomalous time slices and messages in Linux logs (either RFC3164 or RFC5424 format) using statistical learning.

To predict anomalous behavior ADE processes the Linux logs to create a model of expected behavior and compares the expected behavior with the behavior of the time periods of interest. It does not require that either messages or time slices be labelled. ADE uses unsupervised statistical learning algorithms that depend on the behavior of enterprise IT solutions running on Linux being stable and predictable.

The ADE analysis results are written to files in XML format, which can be viewed using a web browser or used in other processing to support the enterprise Linux IT solution that is generating the logs.

For each time slice (interval), ADE measures how unusual the interval is by

ADE creates a summary file with this information for all of the time slices (interval) within a day (period). Here is an example of the summary file for one day.

Example of ADE analysis for a day (period)

For each message, ADE determines if the message strings or similar strings are unusual by calculating a consolidated anomaly score based on

For each interval ADE creates a file with a detailed description of the time slice (interval) with this information. Here is an example of one the details provided about an interval

Example of ADE analysis for a time slice (interval) - finding an anomalous message

The statistical algorithms used by ADE to detect unexpected behavior requires that


Content

The ADE repository is made available under a GPL V3 license. GPL V3 logo

ADE repository contains

The ADE repository and maven control statements do not provide the JDBC compliant database which is needed to run ADE. The ADE code delivery has been tested using Apache Derby.


How to participate

ADE is a project supported by the Open MainFrame Project. To contribute to ADE requires a Corporate Contributor License Agreement. Use the following link to apply for a Corporate Contributor License Agreement which is needed to contribute to ADE Open MainFrame Project ADE project signup.

To report problems and for the status of problems reported please use GitHub issue support for the ADE repository.


Additional details on

Installing ADE

Tailoring ADE to your environment

Installing an ADE instance

Configuring an ADE instance

Running ADE

How to run ADE in your environment

ADE Command Summary

Creating a database for ADE

Priming ADE

Defining a model groups

Verifying that the amount of data is sufficient

Generating a model of expected behavior

Analyzing the Linux Log to check for unusual behavior

Results

Using output from ADE to answer questions about the behavior of Linux systems

How the ADE output is organization - Directory Structure

Details description of the content of period summary file in index.xml

Details description of the content of an interval file in interval_nnn.xml

How ADE detects unusual behavior of Linux systems

Examples

Example of ADE analysis for a day

Example of ADE analysis for a time slice

Example of ADE analysis for another time slice