Repository clean up proposals

[jzonthemtn]

This repository was under a lot of back and forth development during the design and creation of UBI. Because of that, there are some things in the repository that may not be needed anymore. This issue is to identify what needs to stay and what can be removed (or moved to another repository or somewhere else).

The goal is to streamline the repository and make it easier for a new developer to understand the what's important in the repository.

Data Prepper

• The dataprepper directory contains an example blueprint for ingesting UBI events into OpenSearch. I think this directory can be removed.
• I would like to propose to remove the Data Prepper functionality from the UBI plugin. The plugin currently supports sending OpenSearch queries to Data Prepper instead of indexing the events in the local index. This functionality was largely a proof of concept and I think can be removed from the code. I am now aware of any users of this functionality, and it is missing some configuration values that would be needed for a real-world deployment.

Dashboard files

• The search-quality-evaluation-dashboards directory contains example OpenSearch dashboards for viewing the UBI data. I think this code (if still needed) best belongs in the search relevance dashbords plugin project.
• The ubi-dashboards directory seems to contain the same files, or if not the same files, at least files for a similar purpose.

Data files

• The ubi-data-generator scripts are very useful to creating data for testing but I am not sure this is the appropriate place for them.

Other

• The ubi-javascript-collector is not needed by the plugin. For clarity, I propose moving it to a more closely related repository, such as https://www.github.com/o19s/ubi.
• The scripts directory contains scripts that are useful for developing, testing, and learning about the UBI plugin. I propose moving them to a non-top-level directory, perhaps under src/test since that best describes their intended purpose.

[jzonthemtn]

@epugh Would like your thoughts on some of these.

[epugh]

@jzonthemtn can we do them stepwise (as some other things get fixed/released?)...

I think we could totally start with the dataprepper example now. I also like your suggestion about the scripts directory as that does clarify things.

And then, as we get some other things released in 3.1 and 3.2, keep winnowing at the list? Maybe update this ticket to have a list of tasks, one for each of your suggestions, so we can keep this open for a bit, but see the progress?

[jzonthemtn]

@epugh Yes, I think it would be a good idea to make individual tickets for each thing we want to do.

I wrote #89 for relocating the scripts directory.

[epugh]

i notice github now has create subissue option... JIRA here we come!