fix: increase rabbitmq timeout, fix trap mib loading, add mongo indexers (#398)

omrozowicz-splunk · semantic-release-bot · Ryan Faircloth · web-flow · commit 274062b17bfc · 2022-02-14T09:30:23.000+01:00
* fix: increase rabbitmq timeout to 40 minutes (#390) * chore(release): 1.5.7-beta.1 ## [1.5.7-beta.1](v1.5.6...v1.5.7-beta.1) (2022-02-09) ### Bug Fixes * increase rabbitmq timeout to 40 minutes ([#390](#390)) ([4ed1ba0](4ed1ba0)) * fix: traps values translation (#392) * fix: add loading mib for trap values * fix: apply pre-commit * chore(release): 1.5.7-beta.2 ## [1.5.7-beta.2](v1.5.7-beta.1...v1.5.7-beta.2) (2022-02-09) ### Bug Fixes * traps values translation ([#392](#392)) ([a459d1d](a459d1d)) * fix: Support sharing IP for traps (#393) * fix: Support sharing IP for traps * fix: This fix allows the service to share its IP with other services such as sc4s (syslog) * triggering int tests [run-int-tests] Co-authored-by: omrozowicz-splunk <omrozowicz@splunk.com> Co-authored-by: Lukasz Loboda <76950960+uoboda-splunk@users.noreply.github.com> * chore(release): 1.5.7-beta.3 ## [1.5.7-beta.3](v1.5.7-beta.2...v1.5.7-beta.3) (2022-02-10) ### Bug Fixes * Support sharing IP for traps ([#393](#393)) ([33cbb87](33cbb87)) * fix: handle error during mib loading (#391) * fix: handle error during mib loading * fix: handle error during mib loading * fix: handle error during mib loading * chore(release): 1.5.7-beta.4 ## [1.5.7-beta.4](v1.5.7-beta.3...v1.5.7-beta.4) (2022-02-10) ### Bug Fixes * handle error during mib loading ([#391](#391)) ([4a3aaea](4a3aaea)) * ci: trigger int tests fix (#394) * test: [run-int-tests] * triggering int tests [run-int-tests] * Update ci-main.yaml * test: [run-int-tests] * doc: pdate README.md * ci: pdate ci-main.yaml Co-authored-by: omrozowicz-splunk <omrozowicz@splunk.com> * doc: document way of requesting MIBs (#396) * doc: document way of requesting MIBs * doc: document way of MIBs submission and add mkdocs.yml * fix: remove unused clusterName (#395) * fix: remove unused clusterName * fix: remove unused clusterName * chore(release): 1.5.7-beta.5 ## [1.5.7-beta.5](v1.5.7-beta.4...v1.5.7-beta.5) (2022-02-11) ### Bug Fixes * remove unused clusterName ([#395](#395)) ([9db4f82](9db4f82)) * fix: add index creation (#397) * fix: add index creation * chore(release): 1.5.7-beta.6 ## [1.5.7-beta.6](v1.5.7-beta.5...v1.5.7-beta.6) (2022-02-11) ### Bug Fixes * add index creation ([#397](#397)) ([490f54d](490f54d)) Co-authored-by: semantic-release-bot <semantic-release-bot@martynus.net> Co-authored-by: Ryan Faircloth <rfaircloth@splunk.com> Co-authored-by: Lukasz Loboda <76950960+uoboda-splunk@users.noreply.github.com> Co-authored-by: weliasz <77732905+weliasz@users.noreply.github.com>
diff --git a/.github/workflows/ci-main.yaml b/.github/workflows/ci-main.yaml
@@ -84,11 +84,33 @@ jobs:
         with:
           name: test-results-unit-python_${{ matrix.python-version }}
           path: test-results/*
+  
+  integration-tests-check:
+    name: Check if run integration tests
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout Project
+        uses: actions/checkout@v2
+        with:
+          fetch-depth: 2
+      - name: Get commit message
+        id: get_commit_message
+        run: |
+          if [[ '${{ github.event_name }}' == 'push' ]]; then
+            echo ::set-output name=commit_message::$(git log --format=%B -n 1 HEAD)
+          elif [[ '${{ github.event_name }}' == 'pull_request' ]]; then
+            echo ::set-output name=commit_message::$(git log --format=%B -n 1 HEAD^2)
+          fi
+    outputs:
+      commit_message:
+        echo "${{ steps.get_commit_message.outputs.commit_message }}"
 
   test-integration:
     name: Run integration tests
+    needs:
+      - integration-tests-check
     runs-on: ubuntu-latest
-    if: "contains(github.event.head_commit.message, '[run-int-tests]')"
+    if: "contains(needs.integration-tests-check.outputs.commit_message, '[run-int-tests]')"
     steps:
       - uses: actions/checkout@v2
       - name: Archieve repository
diff --git a/README.md b/README.md
@@ -12,4 +12,4 @@ of components.
 For deployment and user documentation [see](https://splunk-connect-for-snmp.readthedocs.io/)
 
 #
-#
+#
diff --git a/charts/splunk-connect-for-snmp/Chart.yaml b/charts/splunk-connect-for-snmp/Chart.yaml
@@ -14,12 +14,12 @@ type: application
 # to the chart and its templates, including the app version.
 # Versions are expected to follow Semantic Versioning (https://semver.org/)
 
-version: 1.5.6
+version: 1.5.7-beta.6
 # This is the version number of the application being deployed. This version number should be
 # incremented each time you make changes to the application. Versions are not expected to
 # follow Semantic Versioning. They should reflect the version the application is using.
 # It is recommended to use it with quotes.
-appVersion: "1.5.6"
+appVersion: "1.5.7-beta.6"
 #
 dependencies:
   - name: mongodb
diff --git a/charts/splunk-connect-for-snmp/templates/traps/service.yaml b/charts/splunk-connect-for-snmp/templates/traps/service.yaml
@@ -5,7 +5,10 @@ metadata:
   labels:
     {{- include "splunk-connect-for-snmp.traps.labels" . | nindent 4 }}
   annotations:
-    metallb.universe.tf/allow-shared-ip: "true"    
+  {{- if .Values.traps.service.usemetallb }}
+    metallb.universe.tf/allow-shared-ip: {{ .Values.traps.service.metallbsharingkey | default "splunk-connect" | quote }}
+  {{- end }}
+    
 spec:
   type: {{ .Values.traps.service.type }}
   externalTrafficPolicy: Local
diff --git a/charts/splunk-connect-for-snmp/values.yaml b/charts/splunk-connect-for-snmp/values.yaml
@@ -224,6 +224,8 @@ traps:
     runAsGroup: 10001
 
   service:
+    usemetallb: true
+    metallbsharingkey: "splunk-connect"
     type: LoadBalancer
     port: 162
 
@@ -327,6 +329,8 @@ mongodb:
         prometheus.io/scrape: "true"
         prometheus.io/port: "9216"
 rabbitmq:
+  extraConfiguration: |
+    consumer_timeout =  2400000
   replicaCount: 1
   pdb:
     create: true
diff --git a/docs/gettingstarted/sc4snmp-installation.md b/docs/gettingstarted/sc4snmp-installation.md
@@ -29,7 +29,6 @@ splunk:
   token: ###SPLUNK_TOKEN###
   insecureSSL: "false"
   port: "###SPLUNK_PORT###"
-  clusterName: my-cluster
 image:
   pullPolicy: "Always"
 traps:
@@ -111,7 +110,6 @@ Other optional variables can be configured:
 | --- | --- | --- |
 | splunk.protocol | port of splunk instance| https |
 | splunk.insecure_ssl| is insecure ssl allowed | "true" |
-| splunk.cluster_name | name of the cluster | "foo" |
 | splunk.eventIndex | name of the events index | "netops" |
 | splunk.metricsIndex | name of the metrics index | "netmetrics" |
 
diff --git a/docs/mib-request.md b/docs/mib-request.md
@@ -0,0 +1,29 @@
+#MIB submission process
+
+In order to achieve human-readable OIDs,the corresponding MIB files are necessary.
+They are being stored in one of the components of SC4SNMP - the MIB server. 
+
+The lis of currently available MIBs is here:
+[https://pysnmp.github.io/mibs/index.csv](https://pysnmp.github.io/mibs/index.csv)
+
+An alternative way to check if the MIB you're interested in is being served is to check the link:
+`https://pysnmp.github.io/mibs/asn1/@mib@` where `@mib@` is the name of MIB (for example `IF-MIB`). If the file 
+is downloading, that means the MIB file exists in the mib server.
+
+## Submit a new MIB file
+
+In case you want to add a new MIB file to the MIB server, follow the steps:
+
+1. Create a fork of the [https://github.com/pysnmp/mibs](https://github.com/pysnmp/mibs) repository 
+   
+2. Put MIB file/s under `src/vendor/@vendor_name@` where `@vendor_name@` is the name of the MIB file's vendor (in case
+there is no directory of vendor you need, create it by yourself)
+   
+3. Create a pull request to a `main` branch
+   
+4. Name the pull request the following way: `feat: add @vendor_name@ MIB files`
+
+
+An alternative way of adding MIBs to MIB server is to create an issue on 
+[https://github.com/pysnmp/mibs](https://github.com/pysnmp/mibs) repository, attaching the files and information about 
+the vendor.
diff --git a/integration_tests/values.yaml b/integration_tests/values.yaml
@@ -5,7 +5,6 @@ splunk:
   token: ###SPLUNK_TOKEN###
   insecureSSL: "true"
   port: "8088"
-  clusterName: my-cluster
 image:
   repository: "snmp-local"
   tag: "latest"
diff --git a/mkdocs.yml b/mkdocs.yml
@@ -40,6 +40,7 @@ nav:
       - Splunk Infrastructure Monitoring: "configuration/sim-configuration.md"
   - Planning: "planning.md"
   - Security: "security.md"
+  - Request MIB: "mib-request.md"
   - Troubleshooting : "bestpractices.md"
   - Releases: "releases.md"
 
diff --git a/pyproject.toml b/pyproject.toml
@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "splunk-connect-for-snmp"
-version = "1.5.6"
+version = "1.5.7-beta.6"
 description = ""
 authors = ["rfaircloth-splunk <rfaircloth@splunk.com>"]
 license = "Apache-2.0"
diff --git a/splunk_connect_for_snmp/__init__.py b/splunk_connect_for_snmp/__init__.py
@@ -15,4 +15,4 @@
 #
 
 
-__version__ = "1.5.6"
+__version__ = "1.5.7-beta.6"
diff --git a/splunk_connect_for_snmp/celery_config.py b/splunk_connect_for_snmp/celery_config.py
@@ -48,5 +48,5 @@
 task_acks_on_failure_or_timeout = True
 task_reject_on_worker_lost = True
 task_track_started = True
-task_time_limit = 1800
+task_time_limit = 2400
 task_ignore_result = True
diff --git a/splunk_connect_for_snmp/common/schema_migration.py b/splunk_connect_for_snmp/common/schema_migration.py
@@ -17,6 +17,8 @@
 import os
 import sys
 
+from pymongo import ASCENDING
+
 from splunk_connect_for_snmp.common.customised_json_formatter import (
     CustomisedJSONFormatter,
 )
@@ -33,7 +35,7 @@
 logger.addHandler(handler)
 
 
-CURRENT_SCHEMA_VERSION = 2
+CURRENT_SCHEMA_VERSION = 3
 MONGO_URI = os.getenv("MONGO_URI")
 
 
@@ -83,3 +85,12 @@ def migrate_to_version_2(mongo_client, task_manager):
     task_manager.delete_all_poll_tasks()
     attributes_collection.drop()
     task_manager.rerun_all_walks()
+
+
+def migrate_to_version_3(mongo_client, task_manager):
+    logger.info("Migrating database schema to version 3")
+    attributes_collection = mongo_client.sc4snmp.attributes
+
+    attributes_collection.create_index(
+        [("address", ASCENDING), ("group_key_hash", ASCENDING)]
+    )
diff --git a/splunk_connect_for_snmp/snmp/manager.py b/splunk_connect_for_snmp/snmp/manager.py
@@ -16,6 +16,7 @@
 import typing
 
 from pysnmp.proto.errind import EmptyResponse
+from pysnmp.smi import error
 from requests import Session
 
 from splunk_connect_for_snmp.inventory.loader import transform_address_to_key
@@ -353,7 +354,10 @@ def load_mibs(self, mibs: List[str]) -> None:
         logger.info(f"loading mib modules {mibs}")
         for mib in mibs:
             if mib:
-                self.builder.loadModules(mib)
+                try:
+                    self.builder.loadModules(mib)
+                except error.MibLoadError as e:
+                    logger.exception(f"Error loading mib for {mib}, e")
 
     def is_mib_known(self, id: str, oid: str, target: str) -> tuple[bool, str]:
 
diff --git a/splunk_connect_for_snmp/snmp/tasks.py b/splunk_connect_for_snmp/snmp/tasks.py
@@ -13,6 +13,8 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 #
+import re
+from contextlib import suppress
 
 from pysnmp.smi.error import SmiError
 
@@ -41,6 +43,7 @@
 MONGO_URI = os.getenv("MONGO_URI")
 MONGO_DB = os.getenv("MONGO_DB", "sc4snmp")
 CONFIG_PATH = os.getenv("CONFIG_PATH", "/app/config/config.yaml")
+OID_VALIDATOR = re.compile(r"^([0-2])((\.0)|(\.[1-9][0-9]*))*$")
 
 
 @shared_task(
@@ -123,9 +126,18 @@ def trap(self, work):
     var_bind_table = []
     not_translated_oids = []
     remaining_oids = []
+    oid_values = set()
     remotemibs = set()
     metrics = {}
     for w in work["data"]:
+
+        if OID_VALIDATOR.match(w[1]):
+            with suppress(Exception):
+                found, mib = self.is_mib_known(w[1], w[1], work["host"])
+                if found and mib not in oid_values:
+                    self.load_mibs(self.builder, [mib])
+                    oid_values.add(mib)
+
         try:
             var_bind_table.append(
                 ObjectType(ObjectIdentity(w[0]), w[1]).resolveWithMib(
diff --git a/test/common/test_schema_migration.py b/test/common/test_schema_migration.py
@@ -42,7 +42,7 @@ def test_migrate_database(self, m_version_1, m_version_2, m_save, m_fetch):
 
         calls = m_save.call_args_list
         self.assertEqual(1, len(calls))
-        self.assertEqual(2, calls[0].args[1])
+        self.assertEqual(3, calls[0].args[1])
 
         m_version_1.assert_called()
         m_version_2.assert_called()
diff --git a/test/snmp/test_mibs.py b/test/snmp/test_mibs.py
@@ -1,6 +1,8 @@
 from unittest import TestCase
 from unittest.mock import Mock
 
+from pysnmp.smi import error
+
 from splunk_connect_for_snmp.snmp.manager import Poller, isMIBResolved
 
 
@@ -44,3 +46,9 @@ def test_is_mib_resolved(self):
         self.assertFalse(isMIBResolved("SNMPv2-SMI::enterprises."))
         self.assertFalse(isMIBResolved("SNMPv2-SMI::mib-2"))
         self.assertTrue(isMIBResolved("OTHER"))
+
+    def test_exception_during_loading(self):
+        poller = Poller.__new__(Poller)
+        poller.builder = Mock()
+        poller.builder.loadModules.side_effect = error.MibLoadError()
+        poller.load_mibs(["a"])

Original file line number	Diff line number	Diff line change
`@@ -12,4 +12,4 @@ of components.`
`12`	`12`	`For deployment and user documentation [see](https://splunk-connect-for-snmp.readthedocs.io/)`
`13`	`13`
`14`	`14`	`#`
`15`		`-#`
	`15`	`+#`