Anomaly detection Engine for Linux Logs (ADE)

Defines the relationship between systems and model groups - updategroups

updategroups is a bash script which invokes the correct Java class after setting up the environment needed. updategroups loads the ADE tables which define the relationship between Linux systems and model groups based on the name of the system. train uses the model group to group one or more Linux systems when it creates a model of normal behavior.

Usage

updategroups updates the ADE tables which defines how train groups Linux systems.

There are two ADE tables which are updated by updategroups:

Groups table
Rules table

The rules stored in the rules table use the name of the system to assign a Linux system to a model group.

Notes

updategroups replaces the tables in the database with the contents of the json file.

If verify determines that there are too few message ids (message keys) to create a high quality model, one solution is to increase the number of Linux systems containing in the model groups. This can be done by reducing the number of model groups and reloading the model groups using updategroups.

Command options

options	action
-j “name_of_json_file”	Updates the ADE database tables with data from json file

Description of fields in json file

field	description
groups
name	Name of group
dataType	Must be “syslog”
evaluationOrder	Determines the order in which the rules are evaluated
ruleName	Name of the rule to use to create the group

rules
name	Name of rule should match a ruleName entry in groups
description	Text description of rule may contain any characters supported by ADE database
membershipRule	Rule that is used to assign a Linux system to a group

Details:

The name field is a required field for a model group definition. The name can contain alphanumeric characters (A through Z, a through z, and 0 through 9), underscores (_), and blanks.

Rules for membershipRule. The text string is a full or partial Linux system name, which can be a fully qualified domain name, a hostname, or an IP address. The text string can contain alphanumeric characters (A through Z, a through z, and 0 through 9), periods (.), colons (:), dashes (–), and forward slashes (/)

When specifying an evaluation order, make sure that more specific membership rules are evaluated before more generic rules; otherwise, a Linux system might be assigned to the wrong group. For example, suppose you have several systems with names ranging from LNXVM50 to LNXVM59. If you define a group for them with a rule of LNXVM5, that rule has to be moved higher in the evaluation order than a more general rule, such as LNXVM.

Example of model group json file

{
    "groups":
        {
            "modelgroups":
                [
                    {"name" : "mg1", "dataType": "syslog", "evaluationOrder" : 1, "ruleName" : "prefixRule" },
                    {"name" : "mg2", "dataType": "syslog", "evaluationOrder" : 2, "ruleName" : "postfixRule"},
                    {"name" : "default", "dataType": "syslog", "evaluationOrder" : 3, "ruleName" : "defaultRule"}
                ]
        },
    "rules":
        [
            {"name" : "prefixRule", "description" : "Matches systems that start with PREFIX", "membershipRule" : "PREFIX*" },
            {"name" : "postfixRule", "description" : "Matches systems that end with POSTFIX", "membershipRule" : "*POSTFIX" },
            {"name" : "defaultRule", "description" : "Matches systems that do not fit another rule", "membershipRule" : "*" }
        ]
}

Restrictions

updategroups does not check for the internal consistency of the json file. There could be groups without rules and rules without ruleNames. The only failures detected are from parsing the json file.

ADE