Skip to content

[TI_MISP] - Updated transform to sort by '@timestamp' to ensure the latest documents are selected for each unique key #15816

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
Oct 30, 2025
Merged

[TI_MISP] - Updated transform to sort by '@timestamp' to ensure the latest documents are selected for each unique key #15816

merged 3 commits into from
Oct 30, 2025

Conversation

@ShourieG
Copy link
Contributor

@ShourieG ShourieG commented Oct 30, 2025

Type of change

  • Bug

Proposed commit message

Existing sorting logic in the transform was by event.ingested field.
This however created issues with bulk ingestion where the order of ingestion 
is not gauaranteed by elasticsearch, which can sometimes lead to recent events 
being not updated in the destination index of the transform. Sorting by '@timestamp'
helps mitigate this issue.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

  • [ ]

How to test this PR locally

Related issues

Screenshots

@ShourieG ShourieG self-assigned this Oct 30, 2025
@ShourieG ShourieG requested a review from a team as a code owner October 30, 2025 10:07
@ShourieG ShourieG added Integration:ti_misp MISP bugfix Pull request that fixes a bug issue Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] labels Oct 30, 2025
@elasticmachine
Copy link

elasticmachine commented Oct 30, 2025

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

@ShourieG ShourieG requested a review from kcreddy October 30, 2025 10:09
kcreddy
kcreddy reviewed Oct 30, 2025
View reviewed changes
packages/ti_misp/elasticsearch/transform/latest_ioc/transform.yml
- misp.attribute.uuid
- misp.attribute.Event.uuid
sort: "event.ingested"
sort: "@timestamp"
Copy link
Contributor

@kcreddy kcreddy Oct 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also needs to update fleet_transform_version.

packages/ti_misp/changelog.yml Outdated
# newer versions go on top
- version: "1.38.1"
changes:
- description: Updated transform to sort by '@timestamp' to ensure the latest documents are selected for each unique key.
Copy link
Contributor

@kcreddy kcreddy Oct 30, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- description: Updated transform to sort by '@timestamp' to ensure the latest documents are selected for each unique key.
- description: Updated transform to sort by `@timestamp` to ensure the latest documents are selected for each unique key.

@ShourieG
Copy link
Contributor Author

ShourieG commented Oct 30, 2025

@kcreddy, addressed all suggestions

@elastic-vault-github-plugin-prod
Copy link

elastic-vault-github-plugin-prod bot commented Oct 30, 2025

🚀 Benchmarks report

Package ti_misp 👍(0) 💚(0) 💔(2)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
threat 1964.64 1633.99 -330.65 (-16.83%) 💔
threat_attributes 3311.26 1818.18 -1493.08 (-45.09%) 💔

To see the full report comment with /test benchmark fullreport

@elasticmachine
Copy link

elasticmachine commented Oct 30, 2025

💚 Build Succeeded

History

cc @ShourieG

@ShourieG ShourieG merged commit 601fa61 into elastic:main Oct 30, 2025
7 checks passed
@ShourieG ShourieG deleted the bugfix/ti_misp_6571 branch October 30, 2025 13:30
@elastic-vault-github-plugin-prod
Copy link

elastic-vault-github-plugin-prod bot commented Oct 30, 2025

Package ti_misp - 1.38.1 containing this change is available at https://epr.elastic.co/package/ti_misp/1.38.1/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kcreddy kcreddy kcreddy approved these changes

Assignees

@ShourieG ShourieG

Labels

bugfix Pull request that fixes a bug issue Integration:ti_misp MISP Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations]

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@ShourieG @elasticmachine @kcreddy