Implementing JSON

These instructions describe how to configure SIEM Agent to output Events in JSON format. JSON is entirely dependent on using Extensions.

JSON support was added in version 4.2 of SIEM Agent, primarily to support event transfer to Apache Kafka, which requires events to be sent in JSON format.

Support for JSON and Kafka allows you to integrate your IBM i systems into a multi-system, Kafka-based event processing workflow.

The JSON Format

JSON, pronounced “Jason,” is a data format used for data exchange. If you are familiar with XML, you can think of it as a lightweight version of XML. The acronym stands for “JavaScript Object Notation.” JSON is an open-source format that is used by many applications.

The JSON format is flexible. In contrast to formats like Syslog or LEEF, there is no fixed header, and there are no fixed attributes. In the context of integration with SIEM Agent, sending JSON to Kafka requires additional information about the structure of the data being received.

The central concept in JSON is the object. An object is an unordered collection of key–value pairs, the attributes.

Values can have different data types, most importantly string, number, and object.

In JSON, objects are surrounded by curly brackets. An object can be empty, and would be represented simply like this:

{ }

In an attribute, the key and value are separated from each other by colons. Keys are surrounded by quotation marks. Values are surrounded by quotation marks if they are string (text) values.

This is a JSON object containing a single attribute:

{"EventID":"TCP0001" }

The key of the attribute is "EventID". Its value is "TCP0001".

Attributes are separated from each other by commas.

{"EventID":"TCP0001","EventText":"User profile FRITZ was created"}

As a value can be of type object, objects can be nested—that is—an attribute value (and hence the object that it resides in) can in-turn contain an object. The following is an example of a JSON object with a nested object:

{ "EventID" : "TCP0001" ,
"EventText" : "User profile FRITZ was created" ,
"FullyQualifiedJobName" : {
	"JobNumber" : "123456" ,
	"JobUser" : "QTCP" ,
	"JobName" : "QTFTP0001" } ,
"CurrentUser" : "QSECOFR" }

This example object is intended to represent information about TCP0001 (User profile created/modified). Events in an *AUDIT Event Source.

The object contains the attributes named "EventID", "EventText", "FullyQualifiedJobName", and "CurrentUser". Most of those attributes have text values, but the value of the "FullyQualifiedJobName" consists of a nested object, which contains the attributes named "JobNumber", "JobUser", and "JobName".

The object's structure can be charted as:

A schema is a generalized version of a JSON object that describes what JSON structure is expected (similar to an XML schema).

NOTE: Whitespace (blanks, tabs, line ends) is ignored during processing (athough, if it is contained in text values, it is preserved).
Aside from nested objects, the order of the attributes is irrelevant.

To use SIEM Agent to create JSON output

1) Use the JSON format for Output, and,

2) You must define Extensions for all Events and/or Subtypes for which you want to output Events.

Using the JSON Format in SIEM Agent

A preset Format named JSON is included with SIEM Agent. It is based on the *JSON Message Style.

For Outputs of Type *KAFKA, always specify JSON as the Format.

You can also use JSON as a Format on *STREAM Output, which allows you to see the data in JSON format in a stream file. This is useful for debugging.

How SIEM Agent constructs JSON data

In JSON-formatted Event data output by SIEM Agent, each Event is represented by a single JSON object.

When SIEM Agent processes an Event, it matches the Event to a particular (1) Event Description. If the Event Source supports Subtypes, the Event is also matched to a (2) Subtype. The Event may be additionally matched to (3) Rules on the Event Description and/or Subtype. All of those items—Event Descriptions, Subtypes, and Rules—can also include Extensions. The JSON object that is output for an Event is constructed from those Extensions.

The Extensions define both what values will be included in the JSON event data, and the hierarchical structure, that is, whether any attributes are nested within other attributes.

The JSON object reflects all Extensions that are added for the Event while that Event is being processed. If ten Extensions are added, ten attributes are included in the JSON output.

The JSON object will not reflect other data, especially the Event Text defined by the Event Description or Rule. This differs from SIEM Agent's other Formats/Message Styles: For Syslog and the other Formats, Event data is constructed either from a fixed set of items, including the Event Text, or from a mixture of fixed items and Extensions. In contrast, when the JSON format is used, if no Extensions have been defined for an Event, no output is created.

This also means that if you want to include Event Text, the Event ID, or a timestamp in the JSON data, you must create Extensions for them.

How to configure extensions

Refer to the SIEM Agent User Guide for general instructions on how to define Extensions.

When no JSON structure exists, use your own best judgment to select information to go into the JSON data. Examples are:

  • Time stamp
  • Event Text
  • Event ID
  • Fully qualified Job Name
  • Current User

Assign attribute names consistently across Event Descriptions. If you call something "Current User" in one place, do not call it "CrntUsr" in another place.

When a JSON structure exists:

For the following, we will assume that a sample JSON object or schema exists.

First, create one Extension (on Event Descriptions, Subtypes, or Rules, depending on requirements) per JSON attribute as follows:

  1. As the Extension's name, use the attribute's name.

    EXAMPLE: If an attribute is named "Key source", the Extension's name should also be "Key source".
  2. Set the Extension's value to the following:

    1. If the JSON attribute contains a fixed value, specify that as the Extension's value.

      EXAMPLE: An attribute should always contain the string value "This is a Production system". Set the Extension's value to the same string.

    2. For a numerical value, the value must not include a leading zero.

      EXAMPLE: ".2345" and "-10.092" are correct. "0.2345" and "-0010.092" are incorrect.

    3. If the JSON attribute contains a variable value, try to find a corresponding value in the Fields or Common Fields available in the Extension's Value field, and insert that.

      EXAMPLE: If the attribute calls for the inclusion of a system value, you can insert the *SYSNAM (System name) field from the Common Fields.
    4. If no corresponding value exists in the Fields, then discern what value can be used instead.
    5. If the JSON attribute contains an object, at this point, insert a placeholder value like "XXX".

After this first pass, define the values for nested objects, that is, attributes whose value is made up from other attributes.

Think back to the example a few pages back:

After the first pass described above, you would have configured actual values for the attributes that themselves do not contain nested attributes, shown in green below:

After the first pass, work "up" from JobNumber/JobUser/JobName and define the actual value of FullyQualifiedJobName as follows.

Create a new Extension:

  • Name: FullyQualifiedJobName
  • Value: Insert the following Extensions in this order, pressing F9 to insert the Extensions and positioning the cursor at the end of the Value field:
    • JobNumber
    • JobUser (Note: First, position cursor after the {JobNumber})
    • JobName (Note: First, position cursor after the {JobUser})

The Extension's value should look like this:

The final list of Extensions should look like this:

At this point, you have mapped the example's JSON structure to Extensions in SIEM Agent.

Generally speaking, work your way up from the attributes without nesting to those that are nested. For example, for a four-layer hierarchical structure, start with the bottom layer, then move up to the next, until you arrive at the top. If there is no nesting, only a single pass is required. If there are two layers, do the first pass, then configure the Extensions that include other Extensions, and then again for the third layer, and so forth.

You do not need to create an Extension for the top-level JSON object, SIEM Agent generates it for you. You also do not need to include any of the markup (curly brackets, commas, quotation marks) in the Extensions. SIEM Agent inserts those for you.

Extensions are shared between Outputs

Extensions that are defined for an Event Source are applied to all Outputs that are used by that Event Source, for example, by being set up as the Default Output for the Event Source, and that support Extensions. If you are sending to more than just one Output, this may or may not be the desired configuration.

EXAMPLE:
1) Event Source AUDIT has Outputs OUTPUT_JSON and OUTPUT_MODERN configured as its Default Outputs.
2) OUTPUT_JSON uses the JSON Format, whereas OUTPUT_MODERN uses the MODERN Format. Both Formats support Extensions.
3) OUTPUT_JSON forwards Events to an Apache Kafka SIEM, while OUTPUT_MODERN forwards Events to Core Security Event Manager.
4) You have configured Extensions in Event Source AUDIT.
In this scenario, Extensions defined in AUDIT define the structure of the Event data sent to OUTPUT_JSON. However, the same Extensions are also included as part of the Event data sent to OUTPUT_MODERN.
If you have this kind of setup and want to define separate Extensions for both Outputs, use two separate Event Sources in order to maintain two separate sets of Extensions.
In the example given here, create a new Event Source of type *AUDIT, either by copying the existing built-in Event Source, or by creating it from scratch. Then assign OUTPUT_JSON as the Default Output to one of them, and OUTPUT_MODERN as the Default Output to the other. Then proceed to modify the Extensions in one of them as desired.

JSON arrays are not supported

JSON objects are enclosed in curly quotes { }. JSON also supports a structure called arrays, used to represent lists. Arrays are enclosed in square brackets [ ]. If your JSON code contains square brackets (outside of text values), it cannot be replicated in SIEM Agent 1:1. SIEM Agent supports JSON objects, with nesting, but not arrays.

NOTE: JSON schemas may contain square brackets for a different purpose, to represent a list of possible values, which SIEM Agent does support.
TIP: To check sample JSON structures, you can use tools like Notepad++ that offer syntax highlighting. (In Notepad++, to use the highlighting, set the Language to "JSON").
TIP: To include Event Text in the JSON output, manually copy the definition of the Event Text and create a new Extension for it. There are no shortcuts—you have to create the Extension manually.