Anomaly detection Engine for Linux Logs (ADE)
Checking logs for anomalies - analyze
analyze is a bash script which invokes the correct Java class after setting up the environment needed. analyze parses the log, loads the information into the JDBC compliant database for use by verify and train, and uses the model created by train to detect anomalies, unusual behavior, in the Linux logs.
Usage
Use analyze to detect anomalies in Linux logs after a model has been created by train. analyze can process Linux logs in RFC 3164 or RFC 5424 format. The logs can be in either files stored in the file system or in “stdin”.
Notes
- If a model has not been created or the model representation can not be loaded from the file system, then analyze is the equivalent of upload. In this case, no analysis results will be created.
- If the -d option is specified ADE will attempt to load all files in the directory
- ADE supports both files which are compressed with gzip and files which are not compressed
- Never seen before messages IDs(message keys) are detected during analyze
Options
| Option | Required | Details |
|---|---|---|
| -d | either -f or -d are required | Name of directory to be loaded into ADE |
| -f | either -f or -d are required | Name of file to be loaded into ADE or “stdin” |
| -g | no | GMT offset of the system which created the log |
| -h | no | Print out help information |
| -o | no | Name of operating system that generated log, defaults to Linux |
| -s | no | System name |
| -years | no | Year of the message log, the default is that it contains the current day |
| -dump_parse_report | no | Creates a report of parsing failures |
Files created by analyze
If you are using the defaults shipped with ADE, the following directories and files are created:
- directory system_name 1
- directory yearMonthDay
- index.xml (summary of intervals within the period)
- directory intervals
- interval_nnn.xml ( details of messages issued during this interval)
- interval_nnn_debug.xml.gz ( information to debug problems with scorers (gzipped))
- directory yearMonthDay
- index.xml (summary of intervals within the period)
- directory intervals
- interval_nnn.xml ( details of messages issued during this interval)
- interval_nnn_debug.xml.gz ( information to debug problems with scorers (gzipped))
- directory yearMonthDay
- index.xml (summary of intervals within the period)
- directory intervals
- interval_nnn.xml ( details of messages issued during this interval)
- interval_nnn_debug.xml.gz ( information to debug problems with scorers (gzipped))
- directory yearMonthDay
- index.xml (summary of intervals within the period)
- directory intervals
- interval_nnn.xml ( details of messages issued during this interval)
- interval_nnn_debug.xml.gz ( information to debug problems with scorers (gzipped))
- directory yearMonthDay
- directory system_name 2
- directory yearMonthDay
- index.xml (summary of intervals within the period)
- directory intervals
- interval_nnn.xml ( details of messages issued during this interval)
- interval_nnn_debug.xml.gz ( information to debug problems with scorers (gzipped))
- directory yearMonthDay
- index.xml (summary of intervals within the period)
- directory intervals
- interval_nnn.xml ( details of messages issued during this interval)
- interval_nnn_debug.xml.gz ( information to debug problems with scorers (gzipped))
- directory yearMonthDay
- index.xml (summary of intervals within the period)
- directory intervals
- interval_nnn.xml ( details of messages issued during this interval)
- interval_nnn_debug.xml.gz ( information to debug problems with scorers (gzipped))
- directory yearMonthDay
- index.xml (summary of intervals within the period)
- directory intervals
- interval_nnn.xml ( details of messages issued during this interval)
- interval_nnn_debug.xml.gz ( information to debug problems with scorers (gzipped))
- directory yearMonthDay
Restrictions
analyze does not support merging of logs. If you analyze data for an interval which already exists, the existing data will be replaced.