[Atlassian JIRA and Atlassian Conflunce] Update Cursor Logic to Remove Duplicate Events (#13665)

After reviewing the Atlassian Jira[1] and Atlassian Confluence[2] API
documentation, It has been noticed that the APIs return data on or
after the specified start date. Currently, the date from the response
body is getting saved into the cursor. Because of this, when the
request is being made in the next interval, it includes data we've
already fetched—leading to duplicate events being published to
Elasticsearch. Hence, updated the cursor logic, added 1ms to it would
fetch the data afterwards.

This change has been tested on the data available in the test folder as
well as using the mock server.

[1]https://developer.atlassian.com/cloud/jira/platform/rest/v3/api-group-audit-records/#api-rest-api-3-auditing-record-get
[2]https://developer.atlassian.com/cloud/confluence/rest/v1/api-group-audit/#api-wiki-rest-api-audit-get

Author: mohitjha-elastic

packages/atlassian_confluence/changelog.yml

@@ -1,4 +1,9 @@
 # newer versions go on top
+- version: "1.29.1"
+  changes:
+    - description: Update cursor logic to remove duplicate events.
+      type: bugfix
+      link: https://github.com/elastic/integrations/pull/13665
 - version: "1.29.0"
   changes:
     - description: Improve error reporting.

packages/atlassian_confluence/data_stream/audit/agent/stream/httpjson.yml.hbs

@@ -65,7 +65,7 @@ response.pagination:
       value: '{{limit}}'
 cursor:
   last_timestamp:
-    value: '[[(toInt .first_event.creationDate)]]'
+    value: '[[add (toInt .first_event.creationDate) 1]]'
 
 fields_under_root: true
 fields:
@@ -110,7 +110,7 @@ response.pagination:
 
 cursor:
   last_timestamp:
-    value: '[[.first_event.timestamp]]'
+    value: '[[formatDate (parseTimestampMilli (add (parseDate .first_event.timestamp).UnixMilli 1))]]'
 {{/if}}
 
 tags:
@@ -128,4 +128,4 @@ publisher_pipeline.disable_host: true
 {{#if processors}}
 processors:
 {{processors}}
-{{/if}} 
+{{/if}}

packages/atlassian_confluence/manifest.yml

@@ -1,7 +1,7 @@
 format_version: "3.0.2"
 name: atlassian_confluence
 title: Atlassian Confluence
-version: "1.29.0"
+version: "1.29.1"
 description: Collect logs from Atlassian Confluence with Elastic Agent.
 type: integration
 categories:

packages/atlassian_jira/changelog.yml

@@ -1,4 +1,9 @@
 # newer versions go on top
+- version: "1.30.2"
+  changes:
+    - description: Update cursor logic to remove duplicate events.
+      type: bugfix
+      link: https://github.com/elastic/integrations/pull/13665
 - version: "1.30.1"
   changes:
     - description: Handle multiple IP address.

packages/atlassian_jira/data_stream/audit/agent/stream/httpjson.yml.hbs

@@ -62,7 +62,7 @@ response.pagination:
       value: '{{limit}}'
 cursor:
   last_timestamp:
-    value: "[[.first_event.created]]"
+    value: '[[formatDate (parseTimestampMilli (add (parseDate .first_event.created).UnixMilli 1))]]'
 {{else}}
 {{#unless username}}
 {{#unless password}}
@@ -91,7 +91,7 @@ response.pagination:
 
 cursor:
   last_timestamp:
-    value: "[[.first_event.timestamp]]"
+    value: '[[formatDate (parseTimestampMilli (add (parseDate .first_event.timestamp).UnixMilli 1))]]'
 {{/if}}
 
 tags:
@@ -116,4 +116,4 @@ fields:
 {{#if processors}}
 processors:
 {{processors}}
-{{/if}} 
+{{/if}}

packages/atlassian_jira/manifest.yml

@@ -1,7 +1,7 @@
 format_version: "3.0.2"
 name: atlassian_jira
 title: Atlassian Jira
-version: "1.30.1"
+version: "1.30.2"
 description: Collect logs from Atlassian Jira with Elastic Agent.
 type: integration
 categories: