CimTests

Test generation mechanism to verify the CIM compatibility of an Add-on

TestTemplates

Includes the test scenarios to check the CIM compatibility of an Add-on.

class standard_lib.cim_tests.test_templates.CIMTestTemplates[source]

Bases: object

Test scenarios to check the CIM compatibility of an Add-on Supported Test scenarios:

  • The eventtype should exctract all required fields of data model

  • One eventtype should not be mapped with more than one data model

  • Field Cluster should be verified (should be included with required field test)

  • Verify if CIM installed or not

  • Not Allowed Fields should not be extracted

test_cim_fields_not_allowed_in_props(splunk_ingest_data, splunk_setup, splunk_searchtime_cim_fields_not_allowed_in_props, record_property)[source]

This testcase checks for cim field of type [“not_allowed_in_search_and_props”, “not_allowed_in_props”] if an extraction is defined in the configuration file.

This test case checks the event_count for the cim fields of type [“not_allowed_in_search_and_props”, “not_allowed_in_search”]. - Expected event_count for these fields is zero.

test_cim_required_fields(splunk_search_util, splunk_ingest_data, splunk_setup, splunk_searchtime_cim_fields, record_property)[source]

Test the the required fields in the data models are extracted with valid values. Supports 3 scenarios. The test order is maintained for better test report.

  • Check that there is at least 1 event mapped with the data model

  • Check that each required field is extracted in all of the events mapped with the data model.

  • Check that if there are inter dependent fields, either all fields should be extracted or none of them should be extracted.

test_eventtype_mapped_multiple_cim_datamodel(splunk_search_util, splunk_ingest_data, splunk_setup, splunk_searchtime_cim_mapped_datamodel, record_property, caplog)[source]

This test case check that event type is not be mapped with more than one data model

Parameters
  • splunk_search_util (SearchUtil) – Object that helps to search on Splunk.

  • splunk_searchtime_cim_mapped_datamodel – Object which contain eventtype list

  • record_property (fixture) – Document facts of test cases.

  • caplog (fixture) – fixture to capture logs.

TestGenerator

Generates test cases to verify the CIM compatibility .

class standard_lib.cim_tests.test_generator.CIMTestGenerator(addon_path, data_model_path, test_field_type=['required', 'conditional'], common_fields_path=None)[source]

Bases: object

Generates test cases to verify the CIM compatibility.

Parameters
  • addon_path (str) – Relative or absolute path to the add-on

  • data_model_path (str) – Relative or absolute path to the data model json files

  • test_field_type (list) – For which types of fields, the test cases should be generated

  • common_fields_path (str) – Relative or absolute path of the json file with common fields

generate_cim_fields_tests()[source]

Generates the test cases for required/conditional/cluster fields.

  1. List CIM mapped models

  2. Iterate through each field in CIM data model

  3. Generate & Yield pytest.param for each test case

  4. Include the cluster test case as well.

generate_field_extractions_test()[source]

Generate tests for the fields which the extractions are not allowed in props.conf

  1. Get a list of fields of type in [“not_allowed_in_search_and_props”, “not_allowed_in_props”] from common fields json.

  2. Get a list of fields whose extractions are defined in props.

  3. Compare and get the list of fields whose extractions are not allowed but defined.

  4. yield the field list

generate_fields_event_count_test()[source]

Generates the tests which should not be extracted in an add-on

  1. Get the list of type=[“not_allowed_in_search_and_props”, “not_allowed_in_search”] fields from common fields json.

  2. Get the list of type=[“not_allowed_in_search_and_props”, “not_allowed_in_search”] fields from mapped datasets.

  3. Combine list1 and list2

  4. yield the field list

  5. Expected event_count for the fields: 0

generate_mapped_datamodel_tests()[source]

Generates the tests to check event type is not be mapped with more than one data model

  1. Get a list of eventtype which defined in eventtype configuration.

  2. yield the eventtype list

generate_tests(fixture)[source]

Generate the test cases based on the fixture provided supported fixtures:

  • splunk_searchtime_cim_fields

  • splunk_searchtime_cim_fields_not_allowed

  • splunk_searchtime_cim_fields_not_extracted

Parameters

fixture (str) – fixture name

get_common_fields(test_type=[])[source]

To obtain list object of common fields mentioned in COMMON_FIELDS_PATH

get_mapped_datasets()[source]

Get all mapped data_sets for each tags stanza from an add-on

Yields

tuple – Tag Stanza, mapped DataSet

DataModelHandler

Provides Data Model handling functionalities. Such as

  • Parse all the data model JSON files

  • Get Mapped data model for an eventtype

class standard_lib.cim_tests.data_model_handler.DataModelHandler(data_model_path)[source]

Bases: object

Provides Data Model handling functionalities. Such as

  • Parse all the data model JSON files

  • Get Mapped data model for an eventtype

Parameters

data_model_path (str) – path to the data model JSON files

get_mapped_data_models(addon_parser)[source]

Get list of eventtypes mapped with Data-Sets. The reason addon_parser is an argument & not attribute of the class is that, the loaded handler should be used with multiple addons.

Parameters

addon_parser (addon_parser.AddonParser) – Object of Addon_parser

Yields

tag stanza mapped with list of data sets

“eventtype=sample”, DataSet(performance)

load_data_models(data_model_path)[source]

Parse all the data model JSON files one by one

Yields

(cim_tests.data_model.DataModel) – parsed data model object

DataModel

Includes DataModel class which handles the DataSets within a data model.

class standard_lib.cim_tests.data_model.DataModel(data_model_json)[source]

Bases: object

Handles the DataSets within a data model.

Parameters

data_model_json (dict) – Dictionary of the data model Json file

get_mapped_datasets(addon_tags)[source]

Get all mapped dataSets for an Add-on’s tags stanza

Parameters

addon_tags (list) – Contains tags mapped to a stanza

Yields

data_set.DataSet – data set object mapped with the tags

DataSet

Includes DataSet class which handles a single data set

class standard_lib.cim_tests.data_set.DataSet(data_set_json, data_model)[source]

Bases: object

Handles a single data set

Parameters

data_set_json (dict) – Json of a single DataSet

classmethod load_dataset(dataset_list, data_model)[source]

Parse all the fields from the data_model_json

Parameters
  • dataset_list (list) – Contains list of datasets

  • data_model – Name of the data model

Yields

data_set.DataSet – Dataset object for the given list

match_tags(addon_tag_list)[source]

Check if the tags are mapped with this data set

FieldTestAdapter

class standard_lib.cim_tests.field_test_adapter.FieldTestAdapater(field)[source]

Bases: standard_lib.addon_parser.fields.Field

Field adapter to include the testing related properties on top of Field

Properties:

  • valid_field (str): New field generated which can only have the valid values

  • invalid_field (str): New field generated which can only have the invalid values

  • validity_query (str): The query which extracts the valid_field out of the field

gen_validity_query()[source]

Generate validation search query:

| eval valid_field = <validity>
| eval valid_field = if(searchmatch(valid_field in <expected_values>), valid_field, null())
| eval valid_field = if(searchmatch(valid_field in <negative_values>), null(), valid_field)
| eval invalid_field=if(isnull(valid_field),field, null())
static get_query_from_values(values)[source]

List of values into SPL list

Example::

[“a”, “b”] to ‘“a”, “b”’

Parameters

values (list) – List of str values

Returns

SPL query list

Return type

str

get_stats_query()[source]

Generate stats search query:

count(field) as field_count, count(valid_field) as valid_field_count,
    values(invalid_field) as invalid_values

FieldTestHelper

Provides the helper methods to test addon_parser.Field object

class standard_lib.cim_tests.field_test_helper.FieldTestHelper(search_util, fields, interval=10, retries=4)[source]

Bases: object

Provides the helper methods to test addon_parser.Field object

Parameters
  • search_util (SearchUtil) – the util class to search on the Splunk instance

  • fields (list addon_parser.Field) – The field to be tested

  • interval (int) – at what interval each retry should be made

  • retries (int) – number of retries to make if no results found

format_exc_message()[source]

Format the exception message to display

  1. There’s no field in the result:

    Source          Sourcetype      Event Count
    -------------------------------------------
    splunkd.log     splunkd         10
    scheduler.log   scheduler       0
    -------------------------------------------
    Search = <search query>
    
  2. There are multiple fields in the result:

    Source          Sourcetype  Field  Event Count  Field Count  Invalid Field Count  Invalid Values
    ------------------------------------------------------------------------------------------------
    splunkd.log     splunkd     One    10           10           5                   'unknown'
    scheduler.log   scheduler   Two    20           20           7                   '-', 'invalid'
    ------------------------------------------------------------------------------------------------
    Event count = 20
    Search = <search_query>
    
    Properties for the field :: One
    . . .
    
static get_table_output(headers, value_list)[source]

Generate a table output of the following format:

Header1 | Header2
---------------
One     | Value1
Two     | Value2
--------------
Parameters
  • headers (list) – list of headers

  • value_list (list of list) – list of rows for the table

test_field(base_search, record_property=None)[source]

Generate a query for the list of fields and return the result

Format of the query is:

<condition>
| eval <validity>
| eval <expected_values>
| eval <not negative_values>
| eval <invalid_fields>
| stats count as event_count, count(field) as field_count,
    count(valid_field) as valid_field_count,
    values(invalid_field) by sourcetype, source
Parameters
  • base_search (str) – Base search. Must be a search command.

  • record_property (fixture) – Document facts of test cases.

Yields

dict

with source, sourcetype, field, event_count, field_count,

valid_field_count, invalid_values keys

JsonSchema

Includes JSON schema for data models

class standard_lib.cim_tests.json_schema.JSONSchema(schema_path=None)[source]

Bases: standard_lib.cim_tests.base_schema.BaseSchema

JsonSchema + Parser of the Data model json files

Parameters

schema_path (str) – Relative or absolute path of the schema file

classmethod parse_data_model(file_path)[source]

Parse and validate the Json file

Parameters

schema_path (str) – Relative or absolute path of the data model json file

BaseSchema

Includes base class for data model schema.

class standard_lib.cim_tests.base_schema.BaseSchema[source]

Bases: abc.ABC

Abstract class to parse the Data model files. The possible format can be JSON, YML, CSV, Cim_json

abstract classmethod parse_data_model(file_path)[source]

Parse the DataModel file Convert it to JSON

Expected Output:

{
    "name":"Default_Authentication",
    "tags": ["authentication","default"],
    "fields_cluster":[],
    "fields":[
        {
            "fieldname": "action",
            "field_type": "required",
            "condition": "action IN ('success','failure','error')",
            "comment":"The action performed on the resource."
        },
        ],
    "child_dataset": [
        {
            "name":"SuccessFul_Default_Authentication",
            "tags": ["authentication","default"],
            "fields_cluster":[],
            "fields":[]
            "child_dataset":[],
            "search_constraints": "action='success'"
        }
    ],
    "search_constraints":"action='failure'"
}