DataGenerator

EventgenParser

class standard_lib.sample_generation.eventgen_parser.EventgenParser(addon_path, config_path=None)[source]

Bases: object

This class represents the entire eventgen.conf file and handles parsing mechanism of eventgen and the rules.

Parameters

addon_path (str) – Path to the Splunk App

check_samples()[source]

Gives a user warning when sample file is not found for the stanza peresent in the configuration file.

get_eventgen_stanzas()[source]

Parses the eventgen.conf file and converts it into a dictionary.

Format:

{
    "sample_file_name": # Not Stanza name
    {
        "input_type": "str",
        "tokens":
        {
            1:
            {
                token: #One#
                replacementType: random
                replacement: static
            }
        }
    }
}
Returns

Dictionary representing eventgen.conf in the above format.

get_sample_stanzas()[source]

Converts a stanza in eventgen.conf to an object of SampleStanza.

Yields

SampleStanza Object

SampleStanza

class standard_lib.sample_generation.sample_stanza.SampleStanza(sample_path, eventgen_params)[source]

Bases: object

This class represents a stanza of the eventgen.conf. It contains all the parameters for the stanza such as:

  • Sample Name

  • Tokens

  • Sample file’s raw data

  • Tokenised events

  • Sample ingestion type

Parameters
  • sample_path (str) – Path to the sample file

  • eventgen_params (dict) – Dictionary representing eventgen.conf

break_events(sample_raw)[source]

Break sample file into list of raw events using breaker

Parameters

sample_raw (str) – Raw sample

Returns

List of raw events

Return type

event_list (list)

get_eventmetadata()[source]

Return the unique host metadata for event.

get_raw_events()[source]

Gets the raw events from the sample file.

get_tokenized_events()[source]

Yields the tokenized events

tokenize(conf_name)[source]

Tokenizes the raw events by replacing all the tokens in it.

Parameters

conf_name (str) – Name of the conf file, “eventgen” or “psa-data-gen”

SampleEvent

class standard_lib.sample_generation.sample_event.SampleEvent(event_string, metadata, sample_name)[source]

Bases: object

This class represents an event which will be ingested in Splunk.

Parameters
  • event_string (str) – Event content

  • metadata (dict) – Contains metadata for the event

  • sample_name (str) – Name of the file containing this event

classmethod copy(event)[source]

Copies the SampleEvent object into a new one. :param event: Event object which has to be copied :type event: SampleEvent

Returns

Copy of the SampleEvent object

get_field_fqdn(rule)[source]

Returns unique fqdn value for the key fields src, dest, host, dvc

Parameters

rule (str) – Type of rule either src, host, dest, dvc

get_field_host(rule)[source]

Returns unique host value for the key fields src, dest, host, dvc

Parameters

rule (str) – Type of rule either src, host, dest, dvc

get_host()[source]

Returns a unique host value

get_ipv4(rule)[source]

Returns Ipv4 Address as per the rule.

Parameters
  • rule (str) – Type of rule either src, host, dest, dvc.

  • the value is not one of the key field it will return a randomly generated Ipv4 address. (If) –

get_ipv6(rule)[source]

Returns Ipv6 Address as per the rule.

Parameters
  • rule (str) – Type of rule either src, host, dest, dvc.

  • the value is not one of the key field it will return a randomly generated Ipv6 address. (If) –

get_key_fields()[source]

Returns the key field value from event

get_token_count(token)[source]

Returns the token count in event

Parameters

token (str) – Token name

register_field_value(field, token_values)[source]

Registers the value for the key fields in its SampleEvent object

Parameters
  • field (str) – Token field name

  • token_values (list/str) – Token value(s) which are replaced in the key fields

replace_token(token, token_values)[source]

Replaces the token value in event

Parameters
  • token (str) – Token name

  • token_values (list/str) – Value(s) to be replaced in the token

update(new_event)[source]

This method updates the event content

Parameters

new_event (str) – Event content

update_metadata(event, metadata, key_fields)[source]

Processes the syslog formated samples Format:

'***SPLUNK*** source=<source> sourcetype=<sourcetype>             field_1       field2        field3             ##value1##    ##value2##   ##value3##'
Parameters
  • event (str) – event string containing raw syslog data

  • metadata (dict) – Contains metadata for the event

Returns

Syslog event and the updated metadata

Rule

Provides Rules for all possible replacements for tokens.

class standard_lib.sample_generation.rule.DestPortRule(token, eventgen_params=None, sample_path=None)[source]

Bases: standard_lib.sample_generation.rule.Rule

replace(sample, token_count)[source]

Yields a random port value from [80, 443, 25, 22, 21]

Parameters
  • sample (SampleEvent) – Instance containing event info

  • token_count (int) – No. of token in sample event where rule is applicable

class standard_lib.sample_generation.rule.DestRule(token, eventgen_params=None, sample_path=None)[source]

Bases: standard_lib.sample_generation.rule.Rule

replace(sample, token_count)[source]

Yields a random dest replacement value from the list of values mentioned in token. Possible values: [“host”, “ipv4”, “ipv6”, “fqdn”]

Parameters
  • sample (SampleEvent) – Instance containing event info

  • token_count (int) – No. of token in sample event where rule is applicable

class standard_lib.sample_generation.rule.DvcRule(token, eventgen_params=None, sample_path=None)[source]

Bases: standard_lib.sample_generation.rule.Rule

replace(sample, token_count)[source]

Yields a random dvc replacement value from the list of values mentioned in token. Possible values: [“host”, “ipv4”, “ipv6”, “fqdn”]

Parameters
  • sample (SampleEvent) – Instance containing event info

  • token_count (int) – No. of token in sample event where rule is applicable

class standard_lib.sample_generation.rule.EmailRule(token, eventgen_params=None, sample_path=None)[source]

Bases: standard_lib.sample_generation.rule.Rule

replace(sample, token_count)[source]

Yields a random email from lookupsuser_email.csv file.

Parameters
  • sample (SampleEvent) – Instance containing event info

  • token_count (int) – No. of token in sample event where rule is applicable

class standard_lib.sample_generation.rule.FileRule(token, eventgen_params=None, sample_path=None)[source]

Bases: standard_lib.sample_generation.rule.Rule

get_file_path()[source]

Returns the relative sample file path and index value

indexed_sample_file(sample, file_path, index, token_count)[source]

Yields the column value of token by reading files.

Parameters
  • sample (SampleEvent) – Instance containing event info

  • file_path (str) – path of the file mentioned in token.

  • index (int) – index value mentioned in file_path i.e. <file_path>:<index>

  • token_count (int) – No. of token in sample event where rule is applicable

lookupfile(sample, file_path, index, token_count)[source]

Yields the column value of token by reading files.

Parameters
  • sample (SampleEvent) – Instance containing event info

  • file_path (str) – path of the file mentioned in token.

  • index (int) – index value mentioned in file_path i.e. <file_path>:<index>

  • token_count (int) – No. of token in sample event where rule is applicable

replace(sample, token_count)[source]

Yields the values of token by reading files.

Parameters
  • sample (SampleEvent) – Instance containing event info

  • token_count (int) – No. of token in sample event where rule is applicable

class standard_lib.sample_generation.rule.FloatRule(token, eventgen_params=None, sample_path=None)[source]

Bases: standard_lib.sample_generation.rule.Rule

replace(sample, token_count)[source]

Yields a random float no. between the range mentioned in token.

Parameters
  • sample (SampleEvent) – Instance containing event info

  • token_count (int) – No. of token in sample event where rule is applicable

class standard_lib.sample_generation.rule.GuidRule(token, eventgen_params=None, sample_path=None)[source]

Bases: standard_lib.sample_generation.rule.Rule

replace(sample, token_count)[source]

Yields a random guid.

Parameters
  • sample (SampleEvent) – Instance containing event info

  • token_count (int) – No. of token in sample event where rule is applicable

class standard_lib.sample_generation.rule.HexRule(token, eventgen_params=None, sample_path=None)[source]

Bases: standard_lib.sample_generation.rule.Rule

replace(sample, token_count)[source]

Yields a random hex value.

Parameters
  • sample (SampleEvent) – Instance containing event info

  • token_count (int) – No. of token in sample event where rule is applicable

class standard_lib.sample_generation.rule.HostRule(token, eventgen_params=None, sample_path=None)[source]

Bases: standard_lib.sample_generation.rule.Rule

replace(sample, token_count)[source]

Yields a random host replacement value from the list of values mentioned in token. Possible values: [“host”, “ipv4”, “ipv6”, “fqdn”]

Parameters
  • sample (SampleEvent) – Instance containing event info

  • token_count (int) – No. of token in sample event where rule is applicable

class standard_lib.sample_generation.rule.IntRule(token, eventgen_params=None, sample_path=None)[source]

Bases: standard_lib.sample_generation.rule.Rule

replace(sample, token_count)[source]

Yields a random int between the range mentioned in token.

Parameters
  • sample (SampleEvent) – Instance containing event info

  • token_count (int) – No. of token in sample event where rule is applicable

class standard_lib.sample_generation.rule.Ipv4Rule(token, eventgen_params=None, sample_path=None)[source]

Bases: standard_lib.sample_generation.rule.Rule

replace(sample, token_count)[source]

Yields a random ipv4 address.

Parameters
  • sample (SampleEvent) – Instance containing event info

  • token_count (int) – No. of token in sample event where rule is applicable

class standard_lib.sample_generation.rule.Ipv6Rule(token, eventgen_params=None, sample_path=None)[source]

Bases: standard_lib.sample_generation.rule.Rule

replace(sample, token_count)[source]

Yields a random ipv6 address

Parameters
  • sample (SampleEvent) – Instance containing event info

  • token_count (int) – No. of token in sample event where rule is applicable

class standard_lib.sample_generation.rule.ListRule(token, eventgen_params=None, sample_path=None)[source]

Bases: standard_lib.sample_generation.rule.Rule

replace(sample, token_count)[source]

Yields a random value from the list mentioned in token.

Parameters
  • sample (SampleEvent) – Instance containing event info

  • token_count (int) – No. of token in sample event where rule is applicable

class standard_lib.sample_generation.rule.MacRule(token, eventgen_params=None, sample_path=None)[source]

Bases: standard_lib.sample_generation.rule.Rule

replace(sample, token_count)[source]

Yields a random mac address

Parameters
  • sample (SampleEvent) – Instance containing event info

  • token_count (int) – No. of token in sample event where rule is applicable

class standard_lib.sample_generation.rule.Rule(token, eventgen_params=None, sample_path=None)[source]

Bases: object

Base class for all the rules.

Parameters
  • token (dict) – Dictionary containing token and its data

  • eventgen_params (dict) – Eventgen stanzas dictionary

  • sample_path (str) – Path to the samples directory

apply(events)[source]

Replaces the token with appropriate values as per rules mapped with the tokens in the event. For replacement_type = all it will generate an event for each replacement value. i.e. integer[1:50] => will generate 50 events

Parameters

events (list) – List of events(SampleEvent)

get_lookup_value(sample, key, headers, value_list)[source]

Common method to read csv and get a random row.

Parameters
  • sample (SampleEvent) – Instance containing event info

  • key (str) – fieldname i.e. host, src, user, dvc etc

  • headers (list) – Headers of csv file in list format

  • value_list (list) – list of replacement values mentioned in configuration file.

Returns

list of mapped columns(int) as per value_list csv_row (list): list of replacement values for the rule.

Return type

index_list (list)

get_rule_replacement_values(sample, value_list, rule)[source]

Common method for replacement values of SrcRule, Destrule, DvcRule, HostRule.

Parameters
  • sample (SampleEvent) – Instance containing event info

  • value_list (list) – list of replacement values mentioned in configuration file.

  • rule (str) – fieldname i.e. host, src, user, dvc etc

Returns

list of mapped columns(int) as per value_list csv_row (list): list of replacement values for the rule.

Return type

index_list (list)

classmethod parse_rule(token, eventgen_params, sample_path)[source]

Returns appropriate Rule object as per replacement type of token.

Parameters
  • token (dict) – Dictionary containing token and its data

  • eventgen_params (dict) – Eventgen stanzas dictionary

  • sample_path (str) – Path to the samples directory

class token_value(key, value)

Bases: tuple

property key

Alias for field number 0

property value

Alias for field number 1

class standard_lib.sample_generation.rule.SrcPortRule(token, eventgen_params=None, sample_path=None)[source]

Bases: standard_lib.sample_generation.rule.Rule

replace(sample, token_count)[source]

Yields a random port value from the range 4000-5000

Parameters
  • sample (SampleEvent) – Instance containing event info

  • token_count (int) – No. of token in sample event where rule is applicable

class standard_lib.sample_generation.rule.SrcRule(token, eventgen_params=None, sample_path=None)[source]

Bases: standard_lib.sample_generation.rule.Rule

replace(sample, token_count)[source]

Yields a random src replacement value from the list of values mentioned in token. Possible values: [“host”, “ipv4”, “ipv6”, “fqdn”]

Parameters
  • sample (SampleEvent) – Instance containing event info

  • token_count (int) – No. of token in sample event where rule is applicable

class standard_lib.sample_generation.rule.StaticRule(token, eventgen_params=None, sample_path=None)[source]

Bases: standard_lib.sample_generation.rule.Rule

replace(sample, token_count)[source]

Yields the static value mentioned in token.

Parameters
  • sample (SampleEvent) – Instance containing event info

  • token_count (int) – No. of token in sample event where rule is applicable

class standard_lib.sample_generation.rule.TimeRule(token, eventgen_params=None, sample_path=None)[source]

Bases: standard_lib.sample_generation.rule.Rule

replace(sample, token_count)[source]

Returns time according to the parameters specified in the input.

Parameters
  • sample (SampleEvent) – Instance containing event info

  • token_count (int) – No. of token in sample event where rule is applicable

class standard_lib.sample_generation.rule.UrlRule(token, eventgen_params=None, sample_path=None)[source]

Bases: standard_lib.sample_generation.rule.Rule

generate_url_query_params()[source]

Generates random query params for url

Returns

Return the query param string

replace(sample, token_count)[source]

Yields a random url replacement value from the list of values mentioned in token.

Possible values: [“ip_host”, “fqdn_host”, “path”, “query”, “protocol”, “full”]

Parameters
  • sample (SampleEvent) – Instance containing event info

  • token_count (int) – No. of token in sample event where rule is applicable

class standard_lib.sample_generation.rule.UserRule(token, eventgen_params=None, sample_path=None)[source]

Bases: standard_lib.sample_generation.rule.Rule

replace(sample, token_count)[source]

Yields a random user replacement value from the list of values mentioned in token. Possible values: [“name”, “email”, “domain_user”, “distinquised_name”]

Parameters
  • sample (SampleEvent) – Instance containing event info

  • token_count (int) – No. of token in sample event where rule is applicable

standard_lib.sample_generation.rule.raise_warning(warning_string)[source]

To raise a pytest user warning along with a log.

Parameters

warning_string (str) – warning string