Anomaly detection Engine for Linux Logs (ADE)

Configuration - setup.props

Use the property keys in the setup.props file to tailor ADE to your environment. ADE contains a working, default configuration that is defined in the conf/setup.props file. If the default configuration does not meet your needs it can be changed by editing the setup.props file or by using a different setup.props file that you created.

The following information may need to be changed:

Where data and control files are stored
What the time zone is
How to connection to a JDBC compliant database

The setup.props file contains the minimum set of properties that are required by ADE.

Examine the source code carefully before changing any of these properties.

The location of the setup.props file is passed to each command using the Java command line argument -Dade.setUpFilePath. The env.sh bash script creates an environment variable ADE_SETUP_FILE which points to the location of this file.

Definition of how data is laid out in the file system

ade.outputPath

The property ade.outputPath defines where ADE writes the files it generates, including directories for trace information and models.

ade.analysisOutputPath

The property ade.analysisOutputPath defines where ADE writes the files that contains the results.

The results are organized using the following directory structure

system name
- date of results
  - index.xml, which contains the summary of results for the period; See Period description for description of the content of this type of file.
  - intervals
    - interval_nnn.xml, which contains the details for each interval; the flowlayout.xml and outputter provided creates one file for every 10 minutes, so nnn ranges from 1 to 144. See Interval description for description of the content of this type of file.
    - interval_nnn.debug provides, for each interval additional information which can be used to review the results of the analytics; the file has been compressed with gzip.

ade.flowLayoutFile

The property ade.flowLayoutFile defines where ADE finds the information that controls how to process the Linux logs. See flowlayout description for additional details about content of file.

ade.xml.xsltDir

The property ade.xml.xsltDir defines where ADE finds the XML schema and style sheets.

ade.criticalWords.file

The property ade.criticalWords.file defines where ADE finds the list of critical words that are used by the CriticalWord scorer.

Time

ade.inputTimeZone

The property ade.inputTimeZone defines the time zone associated with input log data.

ade.outputTimeZone

The property ade.outputTimeZone defines the time zone to be used to store the output from the analysis.

JDBC Database connection

ade.databaseUrl

The property ade.databaseUrl defines how to access the networked database. ADE does not currently support an embedded database.

The statement in the example uses a networked database Derby on the local machine localhost listening on port 1527. The database is created in the directory database/db1.

ade.databaseUser

The property ade.databaseUser defines the user id that ADE uses to accesses the database.

ade.databasePassword

The property ade.databasePassword defines the password used to access the database.

ade.database.keepOnlyAscii

The property ade.database.keepOnlyAscii defines where only Ascii characters are allowed in database.

ade.dataStoreType

The property ade.dataStoreType defines what type of database can be supported. ADE only supports SQL.

ade.databaseDriver

The property ade.databaseDriver defines the name of the JDBC driver that is used to access the JDBC compliant database.

Period Length

ade.periodMode

The property ade.periodMode defines the length of the period. The default flowlayout shipped with ADE assumes that the ade.periodMode is DAILY (24 hours long).

Defining Java class to use

ade.analysisGroupToFlowNameMapperClass

The property ade.analysisGroupToFlowNameMapperClass specifies the JAVA class that maps Linux system names to the model group.

ade.outputFilenameGenerator

The property ade.outputFilenameGenerator specifies the JAVA class that creates the file name used for ADE results.

Statistics

adeext.parseErrorToKeep

The property adeext.parseErrorToKeep defines how many parsing errors to record. If more than the specified number of parsing errors are found, then they are not recorded

adeext.parseErrorTrackNullComponent

The property adeext.parseErrorTrackNullComponent defines whether the absence of a component name is defined as an error which needs to be tracked.

adeext.msgRateReportFreq

The property adeext.msgRateReportFreq specifies how often the message rate report is to be created. This is used when upload or analyze are receiving data from std in.

adeext.msgRateMsgToKeep

The property adeext.msgRateMsgToKeep specifies how many unique messages are needed for each line in the message rate report. This is used when upload or analyze are receiving data from std in.

adeext.msgRate10MinSlotsToKeep

The property adeext.msgRate10MinSlotsToKeep specifies the number of ten minute slots to keep for the message rate report. This is used when upload and analyze are receiving data from std id.

adeext.msgRateMergeSource

The property adeext.msgRateMergeSource specifies if message rates should be keep by individual systems or merged together. This is used when upload and analyze are receiving data from std id.

Example

# --------------------------------------------------------------------
# Paths
# --------------------------------------------------------------------

ade.outputPath=output/
ade.analysisOutputPath=output/continuous

ade.flowLayoutFile=conf/xml/FlowLayout.xml
ade.xml.xsltDir=conf/xml
ade.criticalWords.file=conf/criticalWords.txt


# ----------------------------------------------------------------------
# Time Zone definition
# -----------------------------------------------------------------------

ade.inputTimeZone=GMT+00:00
ade.outputTimeZone=GMT

# --------------------------------------------------------------------
# Database settings
# --------------------------------------------------------------------

ade.databaseUrl=jdbc:derby://localhost:1527/databases/db1
ade.databaseUser=dbuser
ade.databasePassword=passw0rd

ade.database.keepOnlyAscii=true
ade.dataStoreType=SQL
ade.databaseDriver=org.apache.derby.jdbc.ClientDriver

# --------------------------------------------------------------------
# Interval/Period settings
# --------------------------------------------------------------------

# Available period modes: HOURLY, DAILY, WEEKLY, MONTHLY
ade.periodMode=DAILY

# --------------------------------------------------------------------
# Defining class to use for advanced functions
# --------------------------------------------------------------------

ade.analysisGroupToFlowNameMapperClass=com.ibm.ade.ext.os.LinuxAnalysisGroupToFlowNameConstantMapper
ade.outputFilenameGenerator=com.ibm.ade.ext.output.ZOutputFilenameGenerator

# --------------------------------------------------------------------
# Control gathering of statistics 
# --------------------------------------------------------------------

adeext.parseErrorToKeep=100
adeext.parseErrorTrackNullComponent=false

adeext.msgRateReportFreq=5
adeext.msgRateMsgToKeep=1000
adeext.msgRate10MinSlotsToKeep=24
adeext.msgRateMergeSource=true

ADE