Remove hard-coded dataset value

[rdner]

Otherwise, it overrides the configured value.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
  - [ ] I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

Build the package and run the stack using elastic-package:

cd ./packages/filestream
elastic-package build 
elastic-package stack up -v --version 9.2.0-SNAPSHOT

Create a policy with the filestream integration setting a custom dataset value. I've set elastic-agent.test.

sudo ./elastic-agent install --develop
sudo elastic-development-agent enroll --url=https://fleet-server:8220 --enrollment-token=<string> --insecure

Then check the ingested documents in Kibana Discover:

verify that dataset fields and the _index are correct and match the configuration.

Before this change the events didn't show up at all due to mismatch in routing.

I also tested with the default dataset value:

Related issues

• Relates Migrate filestream package to input type #12878

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[rdner]

@jsoriano I think we should remove the [email protected] package from the registry to avoid data loss on the user's end.

In theory, [email protected] should never be installed anyway because there is a newer available version. It's better to be safe though.

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 760697a

History

• 💚 Build #32755 succeeded c535cda

cc @rdner

[belimawr]

LGTM

One thing, that's not necessarily related to this PR: I tested installing the version 2.0.0, changing the default dataset and then upgrading the integration to 2.0.1 (this PR's version). When this is done data is not ingested after the upgrade, the problem persists.

I didn't investigate much, but it seem that the problem is not with the integration/its upgrade, the problem is using the same dataset as with the buggy version. I even tried just installing this version (2.0.1) and using the same dataset name as with a 2.0.0, even with this fresh installation install data was not ingested.

We likely need to add some extra documentation/warning and investigate if there is a way to "clean up" ES so the same dataset name can be used.

[belimawr]

@jsoriano I think we should remove the [email protected] package from the registry to avoid data loss on the user's end.

In theory, [email protected] should never be installed anyway because there is a newer available version. It's better to be safe though.

+1 to removing [email protected] from the package registry because users can still install it if they look for the old package (e.g: by editing the URL to get the 'old' version).

[belimawr]

LGTM

One thing, that's not necessarily related to this PR: I tested installing the version 2.0.0, changing the default dataset and then upgrading the integration to 2.0.1 (this PR's version). When this is done data is not ingested after the upgrade, the problem persists.

I didn't investigate much, but it seem that the problem is not with the integration/its upgrade, the problem is using the same dataset as with the buggy version. I even tried just installing this version (2.0.1) and using the same dataset name as with a 2.0.0, even with this fresh installation install data was not ingested.

We likely need to add some extra documentation/warning and investigate if there is a way to "clean up" ES so the same dataset name can be used.

After writing this I realised that [email protected] requires Kibanan 9.2.0, which has not been released yet. So unless users are running a snapshot version, there is no way they installed the buggy integration version.

[elastic-vault-github-plugin-prod]

Package filestream - 2.0.1 containing this change is available at https://epr.elastic.co/package/filestream/2.0.1/

[belimawr]

@rdner, [email protected] is available in Serverless. So we'll have to deal with users upgrading from it to 2.0.1.

[philippkahr]

We would need to make sure we force a rollover :D then the wrong setting in the index mapping from the 2.0.0 shouldn't matter anymore.

[jsoriano]

@jsoriano I think we should remove the [email protected] package from the registry to avoid data loss on the user's end.

Package removed from the registry. It can be still available during some time in some locations depending on CDN caches.