Configuring the Custom Datasources

This section details the necessary steps required to configure custom datasources, once they have been selected.

Click next to the custom datasource that you want to configure.

From the drop-down menu, click Settings.

Each custom datasource has different configuration parameters that can be applied.

Log Reader Custom Datasource

The Log Reader custom datasource can be used to extract audit data from any application or device that retains information in a log file. You can define a unique name and description for the Log Reader custom datasource but the real configuration is achieved using the remaining parameters on this page.

DataSource Settings

The datasource settings specify the location and detail of the log file from which the data is parsed.

Data Collection
  • Folder Path: Enter the name of the target path to be monitored, including the absolute name of a folder.
  • File: Enter the regular expression for the name of the log file, for example: '.*\.log$'.
  • Process Historical Data: If this option is enabled, the log is processed from the beginning. Otherwise, only new entries from when monitoring is started are taken into account.
Data Parsing
  • Lines Inclusion Filter: Enter the regular expression that is used to parse and filter the lines. The groups that can be used are the fields defined on the Variables selection and Mapping section.
  • Is Inclusion Filter Multiline?: Enable this option to specify whether or not the regular expression fits a multiline result.
  • Time Format: This attribute must be used when the timestamp retrieved from the datasource is not in a native datetime format but is a string.

Variables selection and Mapping

The variables selection and mapping section is used to configure how the original event fields (named groups retrieved by the Regular Expression) are mapped into the Event normalized variables. Multiple variables and mapped selections are permitted. Each time a variable is added another variable line is made available for input.

  • Variable: From the drop-down menu list select the Event Manager variable into which you want to map the regular expression from the log file.
  • Regular Expression Group: Type the regular expression entry exactly as it appears in the log file.
  • Default Value: Enter the default value for what should be returned if the regular expression is missing from the file. For example, Unknown.
Retrieved Audited Actions

The parameters in this section define the actions that are audited whenever the mapping criteria is met. They are split into three definitive sections:

  • System Management: Audits actions relating to any change in system values, such as audit log changes, object or configuration changes.
  • User Activity: Audits action relating to any form of user activity such as successful and unsuccessful system logons, switch of user and object access.
  • Users’ Management: Audits actions relating to any change to user authority, group and role profile activity, reset and modified passwords, inactivity and enabling / disabling of users.

For each of these three sections you specify the Action, Subaction and Subaction Regular Expression Filter to define the audit criteria.

  • Action: From the drop-down list select the general action for which the audit criteria applies. The list of available options is dependent on the section from where the choice is made.
  • Subaction: Use the drop-down list to select the specific action relating to the choice made in the Action list. The available subactions are dependent on the selected Action.
  • Subaction Regular Expression Filter: Enter the actual text in the logfile that you want the audit control to report on. See the following examples.

Example Retrieved Audited Actions

The following two examples show typical entries for monitoring successful and unsuccessful user logins.

NOTE: The entries in the Subaction Regular Expression Filter must match the entries found in the actual log file which you defined in the Data Collection settings.

Microsoft 365 Custom Datasource

The Microsoft 365 custom datasource can be used to retrieve information from Microsoft 365 products.

Data Collection

  • Content Type: From the drop-down choice menu, select the Microsoft 365 content from which you want to collect audit data.

  • Time zone: From the drop-down choice menu select the time zone in which the datasource is located. Note that daylight saving time is automatically adjusted.

Click Show Advanced Settings for additional Data Collection settings for this datasource:

Automatically retrieved actions limit: Enter a value that represents the maximum number of actions that are automatically retrieved from this datasource.

  • Publisher Id: The Publisher Id is a unique identifier reference code to identify Event Manager on API calls. Microosft 365 allows to you limit (or increase) the number of requests by minute by Publisher ID.

  • Refresh Time: This is used to specify the interval at which the assets are checked. The default setting is 300 seconds (5 minutes). Enter a new value if required.

  • Back date hours: Enter the number of hours back from the current point from which you want to retrieve audit data from this Microsoft365 custom datasource.

  • Enable Events Cache: Click this option to enable events caching for this datasource.

  • Events Cache Keep Time Minutes: Enter the number of minutes for which events are held in the cache. The default setting is 1440 minutes (1 day)

  • Events Cache Max Load: Enter the maximum number of events that can be held in the cache at any one time.

NOTE: The cache will hold the maximum of whichever of the Events Cache Keep Time Minutes or Event Cache Max Load is reached first. See also Microsoft Office 365 Event Latency.

  • Enable Trace: Check this option to enable a trace within the datasource which can then be used to help support with the issue diagnosis.

  • Trace Level: Use the drop-down menu to specify the level at the which the trace will be conducted. Select from:

    • Debug

    • Error

    • Info

    • Warning

Variables Selection & Mapping:

The variables selection and mapping section is used to configure how the original event fields (named groups retrieved by the Regular Expression) are mapped into the Event normalized variables. Multiple variables and mapped selections are permitted. Each time a variable is added another variable line is made available for input.

The following are the default settings for Microsoft 365 custom datasource:

Variable Value
Event Time (Source Timezone) [MS365.CretionTime]
Category [MS365.ClientIP]
Operator Name [MS365.UserId]
Object Name [MS365.ObjectID]
Application [MS365.Workload]
Action result [MS365.ResultStatus]
Additional Information 1 [MS365.Id]
Additional Information 2 [MS365.Operation]
Complete Message [MS365.RAWEVENT]
Variable 01 [MS365.OrganizationId]
Variable 02 [MS365.RecordType]
Variable 03 [MS365.UserType]
Variable 04 [MS365.UserKey]

Microsoft Office 365 Event Latency

Because the Event Manager Microsoft 365 datasource data queries must rely on information as provided by the Microsoft Office 365 Management Activity API, you may see non-sequential events as well as delayed timestamps for retrieved events and generated alarms. This is beyond the control of Event Manager.

The Office 365 Management Activity API aggregates actions and events into tenant-specific content binary large objects (BLOBs). It creates these BLOBs by collecting and aggregating actions and events across multiple servers and data centers. Because of this distributed process, the actions and events contained in the BLOBs do not necessarily appear in the order in which they occur. Also, the timestamp for logs stored in these BLOBs are based on the BLOB creation, not the events. For detailed information about log collection and aggregation by the Microsoft Activity API, refer to this Microsoft article.

Additionally, the Management Activity API incorporates mechanisms designed to ensure that customers have access to logs through service interruptions. This can result in a time delay of up to 30 minutes, and sometimes 24 hours or more, after an event occurs for the corresponding audit log entry to be collected and provided by the API. For a table listing the time delays of different services in Office 365, refer to this Microsoft article. However, if you observe delays to be more than 5 days, it could indicate a potential issue. Microsoft advises to check the Service Health Dashboard or open a ticket with Microsoft support.

Windows Event Log Custom Datasource

The Windows Event Log custom datasource can be used to extract audit data from any application or device that records information in a Windows Event Log.

The Windows Event Log custom datasource uses the same settings as the Log Reader Custom Datasource (see Log Reader Custom Datasource for more information) except for the Data Collection settings.

Data Collection
  • Event Log Name: Enter the name of the Windows Event Log from where the audit information is retrieved.
  • Event Log Source: The event source is the name of the software that logs the event. It is often the name of the application or the name of a sub-component of the application if the application is large. The entry in this field must match the Provider Name Field in the XML detailed view of the Event Log Event.
  • Event IDs List: Enter the list of Event IDs on which you want to retrieve audit data that should be filtered by the datasource. Separate multiple entries with a comma.

SNMP Traps Custom Datasource

The SNMP Traps custom datasource is used to extract audit data from any application or device that sends alerts or events as SNMP traps. SNMP traps forwarding is typically found in network devices and monitoring applications. The SNMP traps custom datasource uses the same settings as the Log Reader Custom Datasource (see Log Reader Custom Datasource).

Syslog Custom Datasource

The Syslog custom datasource is used to extract audit data from any application or device that records information in a Syslog file. Syslog files are typically found in systems based upon the UNIX operating platform and common devices such as routers.

The Syslog custom datasource uses the same settings as the Log Reader Custom Datasource (see Log Reader Custom Datasource).

Syslog CEF Custom Datasource

Common Event Format (CEF) is an extensible, text-based format designed to support multiple device types by offering the most relevant information. Message syntaxes are reduced to work with ESM normalization. Specifically, CEF defines a syntax for log records comprised of a standard header and a variable extension, formatted as key-value pairs. CEF is an industry-standard method of sending event information from devices to Event Management software such as Event Manager.

The Syslog CEF custom datasource uses the same settings as the Log Reader Custom Datasource (see Log Reader Custom Datasource).

Syslog CEF variables selection and mapping

The variables selection and mapping section is used to configure how the original event fields (named groups retrieved by the Regular Expression) are mapped into the Event Manager normalized variables. Multiple variables and mapped selections are permitted. Each time a variable is added another variable line is made available for input. See Adding specific mapping by Event, CEF and Regex.

Mapping
Event Manager Normalized Variable CEF key name
Event Time (Source Timezone) rt
Category CEF Integrated
Subcategory CEF Integrated
Action DeviceEventClassid if sTRING else if CAT != " ELSE Name
Subaction act IF != " ELSE same value as Action
Event ID DeviceEventClassID IF Int
Source Machine Name shost
Source Machine IP src
Destination Machine Name dhost
Destination Machine IP dst
Operator Name IF suser != " else suid
User Name duser IF != " else duid
Object Name fname IF != " else Name
Object type fileType
Application dproc
Action Result outcome
Severity Severity
Session ID suid
Net Service app IF != " else dpt
Protocol proto
Additional Information 1 msg
Additional Information 2 reason
Previous Value oldFileName
Current Value fname
User Group/Role dpriv
Complete Message Full_Msg_Syslog
TIP: In the Variable Mapping table, any defined event is shown in blue highlighted text, any defined CEF event in pink highlighted text. If Regex variables are added using specific mapping, they are shown in gray highlighted text.

Adding specific mapping by Event, CEF, and Regex

In addition to the default Syslog CEF mapping detailed above, it is possible to define further mapping by Event, CEF or Regular Expression (Regex).

The Event variables are the same as those that are already defined in the mapping, the CEF variables contain additional variables that are defined in the protocol and the Regular Expression variables are taken from any definition in the Data Parsing section,

  1. Click on the next available line after the default mapping has ended and select a new variable.
  2. Click in the associated field in the Value column to display a list of Events, CEF and Regex options which can be selected for the new mapping.
  3. Select a value to map to the new variable.

Adding a Custom CEF definition for variable mapping

With CEF definitions, you are not limited to the pre-defined variables.

You can add a custom field using the following CEF format:

[CEF.MyCustomKey] (where MyCustomKey is the name of the CEF definition that you want to map to the selected variable).

TIP: Custom CEF definition mapping is shown in green highlighted text in the Variable Mapping table.
NOTE: See Syslog CEF Custom Datasource Action and SubAction Auto-discovery for information on automatically retrieving action information from this custom datasource.

Database Reader Custom Datasource

The Database Custom Reader datasource is unlike any of the other three available custom datasources in that it requires you to interrogate a named database in order to retrieve the required information.

NOTE: The settings for the actual database from where the audit information is retrieved are specified in the Host and Credentials section of the custom datasource.

Variables selection and mapping

The variables selection and mapping section is used to configure how the original event fields (named groups retrieved by the Regular Expression) are mapped into the Event Manager normalized variables. Multiple variables and mapped selections are permitted. Each time a variable is added another variable line is made available for input.

  • Variable: From the drop-down menu list select the Event Manager variable into which you want to map the regular expression from the log file.
  • Query Field: Enter the required field on which the query is based as taken from the SQL query line within the SQL Query. The entry here may be the exact entry as it is found in the SQL Query or may be the result of an alias used in the SQL Query.
  • Default Value: Enter the default value for what should be returned if the regular expression is missing from the file. For example, Unknown.

DataSource Settings

The datasource settings specify the nature of the SQL Query and the specific criteria of the data that is parsed.

Data Collection
  • SQL Query: The entry here specifies the query to be executed. The query (aliases can be used here) should return the fields defined on the Variables selection and Mapping section (see above). For example, Select DBDate as EventTime, DBAction as Action, DBMessage as CompleteMsg from DBTable).Use the $IncrementalCondition variable after the 'where' clause to use the incremental read settings. For example, Select DBDate as EventTime, DBAction as Action, DBMessage as CompleteMsg from DBTable where $IncrementalCondition)
  • Incremental Policy: This field states how the an incremental condition is defined. An incremental condition means that secondary criteria must also be met in order for The Incremental policy can be one of: Disabled, DateAndTime, Datetime or Indexed. Note that If an incremental policy is chosen, it is necessary to include $IncrementalCondition in the where query clause.
  • Incremental Field: This field states the entry to be used as Incremental field in the incremental condition. If the field is not native datetime type, the actual datetime format used has to be specified between {}.
Data Parsing
  • Time Format: This attribute must be used when the timestamp retrieved from the datasource is not in a native datetime format but is a string.
  • Subaction Filter field: This is the query field that is used in the subactions filter in the Actions Mapping section. The entry in this field is very important as it specifies the field in the SQL query on which the action is performed.
For example, your SQL Query entry may contain the line:
 
select message, operator
 
From this, we want to perform the subaction filter on the message element so in the Subaction Filter field we enter:
 
message

We now need to define the Subaction Regular Expression Filter so that it exactly matches the wording of the message. So for example, in the Retrieved Audited Action section for User Activity we can define the following:

Action Subaction Subaction Regular Expression Filter
Successful Login Successful Login .*login ok.*

It is then possible to return to the Variables selection and mapping section of this display to complete the Variable and Query fields with the appropriate criteria that matches the entry in this field. So in this instance:

Variable Query Field
Complete Message CompleteMessage

Event Message Customization

The Event Message Customization section allows you to customize the message that is created by the entry in the Data Parsing field so that it can be made more meaningful and personalized to the end user that views it.

The event message customization can be compiled from a mixture of free-text and the normalized variables that are retrieved during the data collection.

To view the variables that can be used within the customization, click . This displays a list of events for this custom datasource.

Scroll through the list of variables to view those that are available for inclusion in the customized message. Alternatively, start typing text into the Search field to display any variables that match the entered text.

At run time, the entered variables are replaced by the definition for each variable. For example, if Action is entered as the variable and the definition of the variable is ‘Success’, at run time, the word ‘Success’ is displayed in the customized event message.

Retrieved Audited Actions

The Retrieved Audit Actions section allows you to specify the Actions, Subactions and Subaction Regular Expression Filters for each category that can be audited within the custom datasource.

Click next to the auditing category that you want to add action, subaction and regular expression filter information to expand and display the available fields. In the example above, the User Activity category has been expanded.

Action

In the Action column, select the action that you want to audit for this custom datasource. All auditable actions for the selected datasource and category are available in the drop-down menu. In the example above, shown in the screen shot, Logon Failure has been selected as the first action to be audited.

SubAction

Based upon the selection in the Action column, use the drop-down menu in the SubAction column to select one of the available subactions that are associated with the selected action. For example, if the selected action is Logon Failure, subsequent subactions could include AAA Logon Failure, Interactive Logon Failure, Logon Failure or VPN Logon Failure.

There is also the option to add a new subaction by selecting Add New and entering the new subaction name in the dialog that is displayed.

SubAction Regular Expression Filter

The SubAction Regular Expression Filter is used to determine when the audited action is triggered. For example, entering .*Failure.* in this field would mean that any Logon Failure that includes the word Failure in the selected SubAction would trigger an audited action.

Syslog and Syslog CEF Custom Datasource Action and SubAction Auto-discovery

For Syslog and Syslog CEF custom datasources, actions and subactions can be automatically discovered and created upon receipt of the first event message sent by the asset. Multiple and subactions are discovered simultaneously, so if the first event message has a 100 actions defined, the auto-discovery routine will process each action and sub-action separately and define them in the table against the default configuration. This can then be edited as needed to suit your operational requirements.

NOTE: It is also possible to manually enter the actions and subactions for Syslog and Syslog CEF custom datasources as you would with other datasources if required.

Complete Message Parsing

This feature can be used to parse the "Complete message" retrieved for a specific subaction. The groups retrieved using this regular expression overwrite the ones retrieved by the datasource and append the new ones. Using this functionality provides a greater level of detail.

NOTE: This feature requires a full understanding of Regular Expressions and how they operate in order to achieve the desired results.

To activate complete message parsing click the settings icon to the right of the Subaction Regular Expression Filter field.

Click Complete Message Parsing to enable this feature for the selected subaction.

A line becomes available for the entry of the regular expression required for the complete message.

TIP: It is recommended that you use a specialist regex test and debugger such as www.regex101.com which can be used to validate and test and regular expression that you plan to use for the entry in this field.

Complete Message Parsing Example

This example uses a typical entry in a log file, that records any Login Failure from any Operator on a named Source Name, that can be then be read by the Log Reader custom datasource and translated into normal Event Manager variables. A customized event message that can be created to be displayed whenever the audited action is triggered.

As a base for this example, we will use the original audited action of Login Failure by a named user, and add in the details of the Source Machine on which the login failure occurred.

For this example, the original variables and mapping covered the complete message and the operator name.

Using a Regular Expression test program, or by typing directly into the Complete Message Parsing field, build the regex for the complete message by adding the required components to the existing Variables & Mapping Selection fields. For example:

.*from \"(?P<SourceMachine>[^\"]+)\"-.*

This identifies the Complete Message from the specified Source Machine for the required Operator Name. At present there is no Event Manager mapping for the Source Machine component of this message so it needs to be added.

Use the Variables Selection & Mapping section within the Complete Message Parsing window to map the Source Machine group from the Regular Expression into the Event Manager variables.

In the Variable column, select the Event Manager variable to which the Regular Expression is matched, in this case Source Machine Name. Enter the Regular Expression Group as taken from the Complete Message Parsing example. If you leave the "Regular Expression Group" text box empty, the configured "Default Value" will be used instead.

The final requirement is to create a customized event message. This is the message received if the complete message parsing entry triggers an event. You can use all the normalized variables that are being retrieved from during the data collection.

Use the symbol to display a list of variables from which variable selections can be made to compose the event message.

Click OK to save the Complete Message Parsing entry for this subaction.

Once the Complete Message Parsing has been saved for the subaction entry, the Settings icon changes to red to indicate that Complete Message Parsing is active.

Click Apply Changes to save the configuration.