DataGenerator

EventgenParser

class standard_lib.sample_generation.eventgen_parser.EventgenParser(addon_path, config_path=None)

Bases: object

This class represents the entire eventgen.conf file and handles parsing mechanism of eventgen and the rules.

Parameters

addon_path (str) – Path to the Splunk App

check_samples()

Gives a user warning when sample file is not found for the stanza peresent in the configuration file.

get_eventgen_stanzas()

Parses the eventgen.conf file and converts it into a dictionary.

Format:

{
    "sample_file_name": # Not Stanza name
    {
        "input_type": "str",
        "tokens":
        {
            1:
            {
                token: #One#
                replacementType: random
                replacement: static
            }
        }
    }
}

Returns

Dictionary representing eventgen.conf in the above format.

get_sample_stanzas()

Converts a stanza in eventgen.conf to an object of SampleStanza.

Yields

SampleStanza Object

SampleStanza

class standard_lib.sample_generation.sample_stanza.SampleStanza(sample_path, eventgen_params)

Bases: object

This class represents a stanza of the eventgen.conf. It contains all the parameters for the stanza such as:

• Sample Name

• Tokens

• Sample file’s raw data

• Tokenised events

• Sample ingestion type

Parameters

• sample_path (str) – Path to the sample file

• eventgen_params (dict) – Dictionary representing eventgen.conf

break_events(sample_raw)

Break sample file into list of raw events using breaker

Parameters

sample_raw (str) – Raw sample

Returns

List of raw events

Return type

event_list (list)

get_eventmetadata()

Return the unique host metadata for event.

get_raw_events()

Gets the raw events from the sample file.

get_tokenized_events()

Yields the tokenized events

tokenize(conf_name)

Tokenizes the raw events by replacing all the tokens in it.

Parameters

conf_name (str) – Name of the conf file, “eventgen” or “psa-data-gen”

SampleEvent

class standard_lib.sample_generation.sample_event.SampleEvent(event_string, metadata, sample_name)

Bases: object

This class represents an event which will be ingested in Splunk.

Parameters

• event_string (str) – Event content

• metadata (dict) – Contains metadata for the event

• sample_name (str) – Name of the file containing this event

classmethod copy(event)

Copies the SampleEvent object into a new one. :param event: Event object which has to be copied :type event: SampleEvent

Returns

Copy of the SampleEvent object

get_field_fqdn(rule)

Returns unique fqdn value for the key fields src, dest, host, dvc

Parameters

rule (str) – Type of rule either src, host, dest, dvc

get_field_host(rule)

Returns unique host value for the key fields src, dest, host, dvc

Parameters

rule (str) – Type of rule either src, host, dest, dvc

get_host()

Returns a unique host value

get_ipv4(rule)

Returns Ipv4 Address as per the rule.

Parameters

• rule (str) – Type of rule either src, host, dest, dvc.

• the value is not one of the key field it will return a randomly generated Ipv4 address. (If) –

get_ipv6(rule)

Returns Ipv6 Address as per the rule.

Parameters

• rule (str) – Type of rule either src, host, dest, dvc.

• the value is not one of the key field it will return a randomly generated Ipv6 address. (If) –

get_key_fields()

Returns the key field value from event

get_token_count(token)

Returns the token count in event

Parameters

token (str) – Token name

register_field_value(field, token_values)

Registers the value for the key fields in its SampleEvent object

Parameters

• field (str) – Token field name

• token_values (list/str) – Token value(s) which are replaced in the key fields

replace_token(token, token_values)

Replaces the token value in event

Parameters

• token (str) – Token name

• token_values (list/str) – Value(s) to be replaced in the token

update(new_event)

This method updates the event content

Parameters

new_event (str) – Event content

update_metadata(event, metadata, key_fields)

Processes the syslog formated samples Format:

'***SPLUNK*** source=<source> sourcetype=<sourcetype>             field_1       field2        field3             ##value1##    ##value2##   ##value3##'

Parameters

• event (str) – event string containing raw syslog data

• metadata (dict) – Contains metadata for the event

Returns

Syslog event and the updated metadata

Rule

Provides Rules for all possible replacements for tokens.

class standard_lib.sample_generation.rule.DestPortRule(token, eventgen_params=None, sample_path=None)

Bases: standard_lib.sample_generation.rule.Rule

replace(sample, token_count)

Yields a random port value from [80, 443, 25, 22, 21]

Parameters

• sample (SampleEvent) – Instance containing event info

• token_count (int) – No. of token in sample event where rule is applicable

class standard_lib.sample_generation.rule.DestRule(token, eventgen_params=None, sample_path=None)

Bases: standard_lib.sample_generation.rule.Rule

replace(sample, token_count)

Yields a random dest replacement value from the list of values mentioned in token. Possible values: [“host”, “ipv4”, “ipv6”, “fqdn”]

Parameters

• sample (SampleEvent) – Instance containing event info

• token_count (int) – No. of token in sample event where rule is applicable

class standard_lib.sample_generation.rule.DvcRule(token, eventgen_params=None, sample_path=None)

Bases: standard_lib.sample_generation.rule.Rule

replace(sample, token_count)

Yields a random dvc replacement value from the list of values mentioned in token. Possible values: [“host”, “ipv4”, “ipv6”, “fqdn”]

Parameters

• sample (SampleEvent) – Instance containing event info

• token_count (int) – No. of token in sample event where rule is applicable

class standard_lib.sample_generation.rule.EmailRule(token, eventgen_params=None, sample_path=None)

Bases: standard_lib.sample_generation.rule.Rule

replace(sample, token_count)

Yields a random email from lookupsuser_email.csv file.

Parameters

• sample (SampleEvent) – Instance containing event info

• token_count (int) – No. of token in sample event where rule is applicable

class standard_lib.sample_generation.rule.FileRule(token, eventgen_params=None, sample_path=None)

Bases: standard_lib.sample_generation.rule.Rule

get_file_path()

Returns the relative sample file path and index value

indexed_sample_file(sample, file_path, index, token_count)

Yields the column value of token by reading files.

Parameters

• sample (SampleEvent) – Instance containing event info

• file_path (str) – path of the file mentioned in token.

• index (int) – index value mentioned in file_path i.e. <file_path>:<index>

• token_count (int) – No. of token in sample event where rule is applicable

lookupfile(sample, file_path, index, token_count)

Yields the column value of token by reading files.

Parameters

• sample (SampleEvent) – Instance containing event info

• file_path (str) – path of the file mentioned in token.

• index (int) – index value mentioned in file_path i.e. <file_path>:<index>

• token_count (int) – No. of token in sample event where rule is applicable

replace(sample, token_count)

Yields the values of token by reading files.

Parameters

• sample (SampleEvent) – Instance containing event info

• token_count (int) – No. of token in sample event where rule is applicable

class standard_lib.sample_generation.rule.FloatRule(token, eventgen_params=None, sample_path=None)

Bases: standard_lib.sample_generation.rule.Rule

replace(sample, token_count)

Yields a random float no. between the range mentioned in token.

Parameters

• sample (SampleEvent) – Instance containing event info

• token_count (int) – No. of token in sample event where rule is applicable

class standard_lib.sample_generation.rule.GuidRule(token, eventgen_params=None, sample_path=None)

Bases: standard_lib.sample_generation.rule.Rule

replace(sample, token_count)

Yields a random guid.

Parameters

• sample (SampleEvent) – Instance containing event info

• token_count (int) – No. of token in sample event where rule is applicable

class standard_lib.sample_generation.rule.HexRule(token, eventgen_params=None, sample_path=None)

Bases: standard_lib.sample_generation.rule.Rule

replace(sample, token_count)

Yields a random hex value.

Parameters

• sample (SampleEvent) – Instance containing event info

• token_count (int) – No. of token in sample event where rule is applicable

class standard_lib.sample_generation.rule.HostRule(token, eventgen_params=None, sample_path=None)

Bases: standard_lib.sample_generation.rule.Rule

replace(sample, token_count)

Yields a random host replacement value from the list of values mentioned in token. Possible values: [“host”, “ipv4”, “ipv6”, “fqdn”]

Parameters

• sample (SampleEvent) – Instance containing event info

• token_count (int) – No. of token in sample event where rule is applicable

class standard_lib.sample_generation.rule.IntRule(token, eventgen_params=None, sample_path=None)

Bases: standard_lib.sample_generation.rule.Rule

replace(sample, token_count)

Yields a random int between the range mentioned in token.

Parameters

• sample (SampleEvent) – Instance containing event info

• token_count (int) – No. of token in sample event where rule is applicable

class standard_lib.sample_generation.rule.Ipv4Rule(token, eventgen_params=None, sample_path=None)

Bases: standard_lib.sample_generation.rule.Rule

replace(sample, token_count)

Yields a random ipv4 address.

Parameters

• sample (SampleEvent) – Instance containing event info

• token_count (int) – No. of token in sample event where rule is applicable

class standard_lib.sample_generation.rule.Ipv6Rule(token, eventgen_params=None, sample_path=None)

Bases: standard_lib.sample_generation.rule.Rule

replace(sample, token_count)

Yields a random ipv6 address

Parameters

• sample (SampleEvent) – Instance containing event info

• token_count (int) – No. of token in sample event where rule is applicable

class standard_lib.sample_generation.rule.ListRule(token, eventgen_params=None, sample_path=None)

Bases: standard_lib.sample_generation.rule.Rule

replace(sample, token_count)

Yields a random value from the list mentioned in token.

Parameters

• sample (SampleEvent) – Instance containing event info

• token_count (int) – No. of token in sample event where rule is applicable

class standard_lib.sample_generation.rule.MacRule(token, eventgen_params=None, sample_path=None)

Bases: standard_lib.sample_generation.rule.Rule

replace(sample, token_count)

Yields a random mac address

Parameters

• sample (SampleEvent) – Instance containing event info

• token_count (int) – No. of token in sample event where rule is applicable

class standard_lib.sample_generation.rule.Rule(token, eventgen_params=None, sample_path=None)

Bases: object

Base class for all the rules.

Parameters

• token (dict) – Dictionary containing token and its data

• eventgen_params (dict) – Eventgen stanzas dictionary

• sample_path (str) – Path to the samples directory

apply(events)

Replaces the token with appropriate values as per rules mapped with the tokens in the event. For replacement_type = all it will generate an event for each replacement value. i.e. integer[1:50] => will generate 50 events

Parameters

events (list) – List of events(SampleEvent)

get_lookup_value(sample, key, headers, value_list)

Common method to read csv and get a random row.

Parameters

• sample (SampleEvent) – Instance containing event info

• key (str) – fieldname i.e. host, src, user, dvc etc

• headers (list) – Headers of csv file in list format

• value_list (list) – list of replacement values mentioned in configuration file.

Returns

list of mapped columns(int) as per value_list csv_row (list): list of replacement values for the rule.

Return type

index_list (list)

get_rule_replacement_values(sample, value_list, rule)

Common method for replacement values of SrcRule, Destrule, DvcRule, HostRule.

Parameters

• sample (SampleEvent) – Instance containing event info

• value_list (list) – list of replacement values mentioned in configuration file.

• rule (str) – fieldname i.e. host, src, user, dvc etc

Returns

list of mapped columns(int) as per value_list csv_row (list): list of replacement values for the rule.

Return type

index_list (list)

classmethod parse_rule(token, eventgen_params, sample_path)

Returns appropriate Rule object as per replacement type of token.

Parameters

• token (dict) – Dictionary containing token and its data

• eventgen_params (dict) – Eventgen stanzas dictionary

• sample_path (str) – Path to the samples directory

class token_value(key, value)

Bases: tuple

property key

Alias for field number 0

property value

Alias for field number 1

class standard_lib.sample_generation.rule.SrcPortRule(token, eventgen_params=None, sample_path=None)

Bases: standard_lib.sample_generation.rule.Rule

replace(sample, token_count)

Yields a random port value from the range 4000-5000

Parameters

• sample (SampleEvent) – Instance containing event info

• token_count (int) – No. of token in sample event where rule is applicable

class standard_lib.sample_generation.rule.SrcRule(token, eventgen_params=None, sample_path=None)

Bases: standard_lib.sample_generation.rule.Rule

replace(sample, token_count)

Yields a random src replacement value from the list of values mentioned in token. Possible values: [“host”, “ipv4”, “ipv6”, “fqdn”]

Parameters

• sample (SampleEvent) – Instance containing event info

• token_count (int) – No. of token in sample event where rule is applicable

class standard_lib.sample_generation.rule.StaticRule(token, eventgen_params=None, sample_path=None)

Bases: standard_lib.sample_generation.rule.Rule

replace(sample, token_count)

Yields the static value mentioned in token.

Parameters

• sample (SampleEvent) – Instance containing event info

• token_count (int) – No. of token in sample event where rule is applicable

class standard_lib.sample_generation.rule.TimeRule(token, eventgen_params=None, sample_path=None)

Bases: standard_lib.sample_generation.rule.Rule

replace(sample, token_count)

Returns time according to the parameters specified in the input.

Parameters

• sample (SampleEvent) – Instance containing event info

• token_count (int) – No. of token in sample event where rule is applicable

class standard_lib.sample_generation.rule.UrlRule(token, eventgen_params=None, sample_path=None)

Bases: standard_lib.sample_generation.rule.Rule

generate_url_query_params()

Generates random query params for url

Returns

Return the query param string

replace(sample, token_count)

Yields a random url replacement value from the list of values mentioned in token.

Possible values: [“ip_host”, “fqdn_host”, “path”, “query”, “protocol”, “full”]

Parameters

• sample (SampleEvent) – Instance containing event info

• token_count (int) – No. of token in sample event where rule is applicable

class standard_lib.sample_generation.rule.UserRule(token, eventgen_params=None, sample_path=None)

Bases: standard_lib.sample_generation.rule.Rule

replace(sample, token_count)

Yields a random user replacement value from the list of values mentioned in token. Possible values: [“name”, “email”, “domain_user”, “distinquised_name”]

Parameters

• sample (SampleEvent) – Instance containing event info

• token_count (int) – No. of token in sample event where rule is applicable

standard_lib.sample_generation.rule.raise_warning(warning_string)

To raise a pytest user warning along with a log.

Parameters

warning_string (str) – warning string