CimTests

Test generation mechanism to verify the CIM compatibility of an Add-on

TestTemplates

Includes the test scenarios to check the CIM compatibility of an Add-on.

class standard_lib.cim_tests.test_templates.CIMTestTemplates

Bases: object

Test scenarios to check the CIM compatibility of an Add-on Supported Test scenarios:

• The eventtype should exctract all required fields of data model

• One eventtype should not be mapped with more than one data model

• Field Cluster should be verified (should be included with required field test)

• Verify if CIM installed or not

• Not Allowed Fields should not be extracted

test_cim_fields_not_allowed_in_props(splunk_ingest_data, splunk_setup, splunk_searchtime_cim_fields_not_allowed_in_props, record_property)

This testcase checks for cim field of type [“not_allowed_in_search_and_props”, “not_allowed_in_props”] if an extraction is defined in the configuration file.

test_cim_fields_not_allowed_in_search(splunk_ingest_data, splunk_search_util, splunk_setup, splunk_searchtime_cim_fields_not_allowed_in_search, record_property)

This test case checks the event_count for the cim fields of type [“not_allowed_in_search_and_props”, “not_allowed_in_search”]. - Expected event_count for these fields is zero.

test_cim_required_fields(splunk_search_util, splunk_ingest_data, splunk_setup, splunk_searchtime_cim_fields, record_property)

Test the the required fields in the data models are extracted with valid values. Supports 3 scenarios. The test order is maintained for better test report.

• Check that there is at least 1 event mapped with the data model

• Check that each required field is extracted in all of the events mapped with the data model.

• Check that if there are inter dependent fields, either all fields should be extracted or none of them should be extracted.

test_eventtype_mapped_multiple_cim_datamodel(splunk_search_util, splunk_ingest_data, splunk_setup, splunk_searchtime_cim_mapped_datamodel, record_property, caplog)

This test case check that event type is not be mapped with more than one data model

Parameters

• splunk_search_util (SearchUtil) – Object that helps to search on Splunk.

• splunk_searchtime_cim_mapped_datamodel – Object which contain eventtype list

• record_property (fixture) – Document facts of test cases.

• caplog (fixture) – fixture to capture logs.

TestGenerator

Generates test cases to verify the CIM compatibility .

class standard_lib.cim_tests.test_generator.CIMTestGenerator(addon_path, data_model_path, tokenized_events, test_field_type=['required', 'conditional'], common_fields_path=None)

Bases: object

Generates test cases to verify the CIM compatibility.

Parameters

• addon_path (str) – Relative or absolute path to the add-on

• data_model_path (str) – Relative or absolute path to the data model json files

• test_field_type (list) – For which types of fields, the test cases should be generated

• common_fields_path (str) – Relative or absolute path of the json file with common fields

generate_cim_fields_tests()

Generates the test cases for required/conditional/cluster fields.

1. List CIM mapped models

2. Iterate through each field in CIM data model

3. Generate & Yield pytest.param for each test case

4. Include the cluster test case as well.

generate_field_extractions_test()

Generate tests for the fields which the extractions are not allowed in props.conf

1. Get a list of fields of type in [“not_allowed_in_search_and_props”, “not_allowed_in_props”] from common fields json.

2. Get a list of fields whose extractions are defined in props.

3. Compare and get the list of fields whose extractions are not allowed but defined.

4. yield the field list

generate_fields_event_count_test()

Generates the tests which should not be extracted in an add-on

1. Get the list of type=[“not_allowed_in_search_and_props”, “not_allowed_in_search”] fields from common fields json.

2. Get the list of type=[“not_allowed_in_search_and_props”, “not_allowed_in_search”] fields from mapped datasets.

3. Combine list1 and list2

4. yield the field list

5. Expected event_count for the fields: 0

generate_mapped_datamodel_tests()

Generates the tests to check event type is not be mapped with more than one data model

1. Get a list of eventtype which defined in eventtype configuration.

2. yield the eventtype list

generate_tests(fixture)

Generate the test cases based on the fixture provided supported fixtures:

• splunk_searchtime_cim_fields

• splunk_searchtime_cim_fields_not_allowed

• splunk_searchtime_cim_fields_not_extracted

Parameters

fixture (str) – fixture name

get_common_fields(test_type=[])

To obtain list object of common fields mentioned in COMMON_FIELDS_PATH

get_mapped_datasets()

Get all mapped data_sets for each tags stanza from an add-on

Yields

tuple – Tag Stanza, mapped DataSet

DataModelHandler

Provides Data Model handling functionalities. Such as

• Parse all the data model JSON files

• Get Mapped data model for an eventtype

class standard_lib.cim_tests.data_model_handler.DataModelHandler(data_model_path)

Bases: object

Provides Data Model handling functionalities. Such as

• Parse all the data model JSON files

• Get Mapped data model for an eventtype

Parameters

data_model_path (str) – path to the data model JSON files

get_mapped_data_models(addon_parser)

Get list of eventtypes mapped with Data-Sets. The reason addon_parser is an argument & not attribute of the class is that, the loaded handler should be used with multiple addons.

Parameters

addon_parser (addon_parser.AddonParser) – Object of Addon_parser

Yields

tag stanza mapped with list of data sets

“eventtype=sample”, DataSet(performance)

load_data_models(data_model_path)

Parse all the data model JSON files one by one

Yields

(cim_tests.data_model.DataModel) – parsed data model object

DataModel

Includes DataModel class which handles the DataSets within a data model.

class standard_lib.cim_tests.data_model.DataModel(data_model_json)

Bases: object

Handles the DataSets within a data model.

Parameters

data_model_json (dict) – Dictionary of the data model Json file

get_mapped_datasets(addon_tags)

Get all mapped dataSets for an Add-on’s tags stanza

Parameters

addon_tags (list) – Contains tags mapped to a stanza

Yields

data_set.DataSet – data set object mapped with the tags

DataSet

Includes DataSet class which handles a single data set

class standard_lib.cim_tests.data_set.DataSet(data_set_json, data_model)

Bases: object

Handles a single data set

Parameters

data_set_json (dict) – Json of a single DataSet

classmethod load_dataset(dataset_list, data_model)

Parse all the fields from the data_model_json

Parameters

• dataset_list (list) – Contains list of datasets

• data_model – Name of the data model

Yields

data_set.DataSet – Dataset object for the given list

match_tags(addon_tag_list)

Check if the tags are mapped with this data set

FieldTestAdapter

class standard_lib.cim_tests.field_test_adapter.FieldTestAdapter(field)

Bases: standard_lib.addon_parser.fields.Field

Field adapter to include the testing related properties on top of Field

Properties:

• valid_field (str): New field generated which can only have the valid values

• invalid_field (str): New field generated which can only have the invalid values

• validity_query (str): The query which extracts the valid_field out of the field

gen_validity_query()

Generate validation search query:

| eval valid_field = <validity>
| eval valid_field = if(searchmatch(valid_field in <expected_values>), valid_field, null())
| eval valid_field = if(searchmatch(valid_field in <negative_values>), null(), valid_field)
| eval invalid_field=if(isnull(valid_field),field, null())

static get_query_from_values(values)

List of values into SPL list

Example::

[“a”, “b”] to ‘“a”, “b”’

Parameters

values (list) – List of str values

Returns

SPL query list

Return type

str

get_stats_query()

Generate stats search query:

count(field) as field_count, count(valid_field) as valid_field_count,
    values(invalid_field) as invalid_values

FieldTestHelper

Provides the helper methods to test addon_parser.Field object

class standard_lib.cim_tests.field_test_helper.FieldTestHelper(search_util, fields, interval=10, retries=4)

Bases: object

Provides the helper methods to test addon_parser.Field object

Parameters

• search_util (SearchUtil) – the util class to search on the Splunk instance

• fields (list addon_parser.Field) – The field to be tested

• interval (int) – at what interval each retry should be made

• retries (int) – number of retries to make if no results found

format_exc_message()

Format the exception message to display

1. There’s no field in the result:

   Source          Sourcetype      Event Count
   -------------------------------------------
   splunkd.log     splunkd         10
   scheduler.log   scheduler       0
   -------------------------------------------
   Search = <search query>
   

2. There are multiple fields in the result:

   Source          Sourcetype  Field  Event Count  Field Count  Invalid Field Count  Invalid Values
   ------------------------------------------------------------------------------------------------
   splunkd.log     splunkd     One    10           10           5                   'unknown'
   scheduler.log   scheduler   Two    20           20           7                   '-', 'invalid'
   ------------------------------------------------------------------------------------------------
   Event count = 20
   Search = <search_query>
   
   Properties for the field :: One
   . . .
   

test_field(base_search, record_property=None)

Generate a query for the list of fields and return the result

Format of the query is:

<condition>
| eval <validity>
| eval <expected_values>
| eval <not negative_values>
| eval <invalid_fields>
| stats count as event_count, count(field) as field_count,
    count(valid_field) as valid_field_count,
    values(invalid_field) by sourcetype, source

Parameters

• base_search (str) – Base search. Must be a search command.

• record_property (fixture) – Document facts of test cases.

Yields

dict –

with source, sourcetype, field, event_count, field_count,

valid_field_count, invalid_values keys

JsonSchema

Includes JSON schema for data models

class standard_lib.cim_tests.json_schema.JSONSchema(schema_path=None)

Bases: standard_lib.cim_tests.base_schema.BaseSchema

JsonSchema + Parser of the Data model json files

Parameters

schema_path (str) – Relative or absolute path of the schema file

classmethod parse_data_model(file_path)

Parse and validate the Json file

Parameters

schema_path (str) – Relative or absolute path of the data model json file

BaseSchema

Includes base class for data model schema.

class standard_lib.cim_tests.base_schema.BaseSchema

Bases: abc.ABC

Abstract class to parse the Data model files. The possible format can be JSON, YML, CSV, Cim_json

abstract classmethod parse_data_model(file_path)

Parse the DataModel file Convert it to JSON

Expected Output:

{
    "name":"Default_Authentication",
    "tags": ["authentication","default"],
    "fields_cluster":[],
    "fields":[
        {
            "fieldname": "action",
            "field_type": "required",
            "condition": "action IN ('success','failure','error')",
            "comment":"The action performed on the resource."
        },
        ],
    "child_dataset": [
        {
            "name":"SuccessFul_Default_Authentication",
            "tags": ["authentication","default"],
            "fields_cluster":[],
            "fields":[]
            "child_dataset":[],
            "search_constraints": "action='success'"
        }
    ],
    "search_constraints":"action='failure'"
}