Quality of life improvement for documentation (#604)

* Quality of life improvement for documentation (#602)

* Readme qualify of life improvements

* Update README.md

* Add upgrade and move ansible install location

* add note to note change passwords manually

* Update README.md

* Update README.md

* Update elastic-agent-mangement.md

* Update elastic-agent-mangement.md

* change hardcoded url to relative path

* formatting improvements for Kibana URL

* formatting improvements for Kibana URL

* Update duplicate title in 'Retrieving Passwords'

* update relative path anchor reference

* Adding in Kendall's updates to community engagement + some small fixes

* Adding note on scaling to these docs

* Add small updates:

1. add a discussion around LME users and passwords to refer to elastic's
   docs
2. add changes to add in 24.04 notes around initial testing.
3. small fixes

* Adding link and removing link

* Update and rename elastic-agent-mangement.md to elastic-agent-management.md

update url for passwords on readme, update url to remove 5601 port, and rename file

* PUshing updates to Readme

---------

Co-authored-by: Andrew Arz <[email protected]>
Co-authored-by: mitchelbaker-cisa <[email protected]>
Co-authored-by: Michael Reeves <[email protected]>

* revise changes to match develop

* add missing docs

* Added video links and made small grammar corrections

* Updated video link

* grammar changes

* fixing clunky wording

---------

Co-authored-by: Tyler Morris <[email protected]>
Co-authored-by: Andrew Arz <[email protected]>
Co-authored-by: Michael Reeves <[email protected]>
Co-authored-by: Connor Aubry <[email protected]>

Author: 5 people

README.md

@@ -5,7 +5,9 @@ This guide will walk you through the process of enrolling an Elastic agent.
 ## Steps to Enroll an Agent
 
 1. **Access the Fleet Menu**
-   - Open the LME dashboard
+   - Open the LME dashboard: `https://{SERVER_IP}`
+      - Password information can be found in the [README](/README.md#retrieving-passwords).
+   - Open the "hamburger" menu icon in the top left (three horizontal lines)
    - Scroll down and select "Fleet" from the menu
 
 2. **Add a New Agent**
@@ -40,7 +42,8 @@ This guide will walk you through the process of enrolling an Elastic agent.
 ![example-screenshot](/docs/imgs/insecure-powershell.png)
 
 8. **Execute the Command**
-   - Recommend running each line individually so you can see a clear picture of the status of each command ran. The entire process will download an agent, unzip it, and install it.
+   - Run the command on the desired host.
+      - It is recommended to run each line individually so you can see the status of each command. The entire process will download an agent, unzip it, and install it.
 
 From Fleet you should see the agent enrolled now.
 
@@ -51,7 +54,9 @@ This guide will walk you through the process of adding a Windows integration to
 ## Steps to Add Windows Integration
 
 1. **Access Fleet and Agent Policies**
-   - Open the LME dashboard
+   - Open the LME dashboard: `https://{SERVER_IP}`
+      - Password information can be found in the [Readme](/README.md#retrieving-passwords).
+   - Open the "hamburger" menu icon in the top left (three horizontal lines)
    - Select "Fleet" from the menu
    - Click on "Agent policies"
 
@@ -64,6 +69,7 @@ This guide will walk you through the process of adding a Windows integration to
 
 4. **Choose Windows Integration**
    - From the list of available integrations, select "Windows"
+   - Select "add Windows"
 
 5. **Configure Windows Integration**
    - Scroll down to review the options available
@@ -100,5 +106,3 @@ The Elastic agent has multiple debugging commands that can be run to troubleshoo
 In addition, you can use this [link](https://www.elastic.co/guide/en/fleet/current/installation-layout.html) to navigate/find the directories for where Elastic agent is installed on the operating system.
 
 If there are issues with running the command involving a pipe file, the elastic endpoint service (a windows service started by the agent) is in a failed state, and retarting the machine will most likely fix it, check out this [link](https://discuss.elastic.co/t/windows-pipe-elastic-agent-system-access-is-denied/316344) However, this isn't required if the agent is showing as healthy, only if you want to run other cli agent debugging commands.
-
-

docs/markdown/agents/elastic-agent-mangement.md renamed to docs/markdown/agents/elastic-agent-management.md

@@ -45,7 +45,7 @@ You can confirm the version is accurate with a list from wazuh's versions [HERE]
 1. **Download the Wazuh Agent**
    - Download the Wazuh agent MSI installer from the following URL:
      ```
-     https://packages.wazuh.com/4.x/windows/wazuh-agent-{WAZUH_AGENT_VERSION}.msi
+     https://packages.wazuh.com/4.x/windows/wazuh-agent-{WAZUH_AGENT_VERSION}-1.msi
      ```
    - Replace `{WAZUH_AGENT_VERSION}` with the appropriate version number.
    - You can also use the below powershell command: 
@@ -153,4 +153,4 @@ Replace `[agent_id]` with the ID of the agent you want to check. This will provi
 
 This command gives you a quick overview of how many agents are active, disconnected, or never connected.
 
-See official Wazuh documentation for more steps on [agent_control](https://documentation.wazuh.com/current/user-manual/reference/tools/agent-control.html)
+See official Wazuh documentation for more steps on [agent_control](https://documentation.wazuh.com/current/user-manual/reference/tools/agent-control.html)

docs/markdown/agents/wazuh-agent-mangement.md renamed to docs/markdown/agents/wazuh-agent-management.md

@@ -1,20 +1,21 @@
 # Certificates
-# 
+ 
 The LME installation makes use of a number of TLS certificates to protect communications between the server components and agents, and also secures the connections between Elasticsearch and Kibana. 
 By default the installation will create certificates and this documentation describes how to modify and update the cert store.
 
 ## Regenerating Self-Signed Certificates
 The easiest way to do this is to delete the `lme_certs` volume, and restart lme.service:
 
-This is destructive and not recommended,but there could be cases.
+This is destructive and not recommended, but there could be cases.
 ```bash
 sudo -i podman volume rm lme_certs
 sudo systemctl restart lme.service
 ```
 
 ## Using Your Own Certificates
-You can certificates signed by an existing root CA as part of the LME install by generating certificates manually with the correct settings and placing these within the required directory inside the LME folder. **NOTE: The default supported method of LME installation is to use the automatically created self-signed certificates, and we will be unable to support any problems that arise from generating the certificates manually incorrectly.**
+You can use certificates signed by an existing root CA as part of the LME install by generating certificates manually with the correct settings and placing these within the required directory inside the LME folder.
 
+**NOTE: The default supported method of LME installation is to use the automatically created self-signed certificates, and we will be unable to support any problems that arise from generating the certificates manually incorrectly.**
 
 ### Certificate Creation
 If you create certificates ensure their subject alt names allow for the ips/dns entries listed below, as well as the ips/domains you'll be connecting to the service as: 
@@ -52,7 +53,7 @@ instances:
       - "127.0.0.1"
 ```
 
-For example, the new kibana cert would need to support the above alternative names... you can also ensure its setup properly by viewing the current cert (assuming you've already mounted the `lme_certs` podman volume.
+For example, the new kibana cert would need to support the above alternative names. You can also ensure its setup properly by viewing the current cert (assuming you've already mounted the `lme_certs` podman volume).
 ```bash
 root@ubuntu:~$ cat /var/lib/containers/storage/volumes/lme_certs/_data/kibana/kibana.crt  | openssl x509 -text | grep -i Alternative -A 1
             X509v3 Subject Alternative Name:

docs/markdown/maintenance/certificates.md

@@ -54,6 +54,8 @@ Users must pay for hosting, bandwidth and time; for an estimate of server specs
 To keep LME simple, our guide only covers single server setups.  Considering the differences across environments and scaling needs, we cannot provide an estimate of server resources beyond single server setups.
 It’s possible to scale the solution to multiple event collectors and ELK nodes, but that will require more experience with the technologies involved. However, we plan to publish documentation for scaling LME in the future.
 
+Please see the above blogpost from elastic for discussion on how to scale an elastic stack cluster. 
+
 ## Required infrastructure
 
 To begin installing LME, you will need access to the following servers or you will need to create them:
@@ -98,6 +100,4 @@ Servers can be either on premise, in a public cloud, or in a private cloud. It i
 ## What firewall rules are needed?
 Please see our cloud documentation for a discussion on firewalls [LME in the Cloud](/docs/markdown/loggging-guidance/cloud.md). 
 
-You must ensure that the client machine you want to monitor can reach the main LME ports as described in the ReadMe [Required Ports section](/README.md#required-ports).
-
-
+You must ensure that the client machine you want to monitor can reach the main LME ports as described in the ReadMe [Required Ports section](/README.md#required-ports).

docs/markdown/prerequisites.md

@@ -0,0 +1,49 @@
+# Architecture:
+LME runs on Ubuntu 22.04 and leverages Podman containers for security, performance, and scalability. We’ve integrated Wazuh,  Elastic, and ElastAlert open source tools to provide log management, endpoint security monitoring, alerting, and data visualization capabilities. This modular, flexible architecture supports efficient log storage, search, and threat detection, and enables you to scale as your logging needs evolve.
+
+### Diagram: 
+
+![diagram](/docs/imgs/lme-architecture-v2.png) 
+
+### Containers:
+Containerization allows each component of LME to run independently, increasing system security, improving performance, and making troubleshooting easier. 
+
+LME uses Podman as its container engine because it is more secure (by default) against container escape attacks than other engines like Docker. It's far more debug and programmer friendly. We’re making use of Podman’s unique features, such as Quadlets (Podman's systemd integration) and User Namespacing,  to increase system security and operational efficiency.
+
+LME uses these containers:
+
+  - **Setup**: Runs `/config/setup/init-setup.sh` based on the configuration of DNS defined in `/config/setup/instances.yml`. The script will create a certificate authority (CA), underlying certificates for each service, and initialize the admin accounts for Elasticsearch(user:`elastic`) and Kibana(user:`kibana_system`). 
+  - **Elasticsearch**: Runs LME's database and indexes all logs.
+  - **Kibana**: The front end for querying logs, visualizing data, and managing fleet agents.
+  - **Elastic Fleet-Server**: Executes an [elastic agent ](https://github.com/elastic/elastic-agent) in fleet-server mode. Coordinates elastic agents to  gather client logs and status. Configuration is inspired by the [elastic-container](https://github.com/peasead/elastic-container) project.
+  - **Wazuh-Manager**: Allows LME to deploy and manage Wazuh agents.
+    -  Wazuh (open source) gives EDR (Endpoint Detection Response) with security dashboards to cover the security of all of the machines.
+  - **LME-Frontend** (*coming in a future release*): Will host an API and GUI that unifies the architecture behind one interface.
+   
+### Required Ports:
+Ports required are as follows:
+ - Elasticsearch: *9200*
+ - Kibana: *443,5601*
+ - Wazuh: *1514,1515,1516,55000,514*
+ - Agent: *8220*
+
+**Note**: For Kibana, 5601 is the default port. We've also set kibana to listen on 443 as well.
+
+### Agents and Agent Management: 
+LME leverages both Wazuh and Elastic agents providing more comprehensive logging and security monitoring across various log sources. The agents gather critical data from endpoints and send it back to the LME server for analysis, offering organizations deeper visibility into their security posture. We also make use of the Wazuh Manager and Elastic Fleet for agent orchestration and management.
+
+- **Wazuh Agents**: Enables Endpoint Detection and Response (EDR) on client systems, providing advanced security features like intrusion detection and anomaly detection. For more information, see [Wazuh's agent documentation](https://github.com/wazuh/wazuh-agent). 
+- **Wazuh Manager**: Responsible for managing Wazuh Agents across endpoints, and overseeing agent registration, configuration, and data collection, providing centralized control for monitoring security events and analyzing data. 
+- **Elastic Agents**: Enhance log collection and management, allowing for greater control and customization in how data is collected and analyzed. Agents also feature a vast collection of integrations for many log types/applications. For more information, see [Elastic's agent documentation](https://github.com/elastic/elastic-agent).
+- **Elastic Fleet**: Manages Elastic Agents across your infrastructure, providing centralized control over agent deployment, configuration, and monitoring. It simplifies the process of adding and managing agents on various endpoints. ElasticFleet also supports centralized updates and policy management.
+
+
+### Alerting:
+LME has setup [ElastAlert](https://elastalert2.readthedocs.io/en/latest/index.html), an open-source alerting framework, to automate alerting based on data stored in Elasticsearch. It monitors Elasticsearch for specific patterns, thresholds, or anomalies, and generates alerts when predefined conditions are met. This provides proactive detection of potential security incidents, enabling faster response and investigation. ElastAlert’s flexible rule system allows for custom alerts tailored to your organization’s security monitoring needs, making it a critical component of the LME alerting framework. 
+
+### Log Storage and Search:
+
+As the core component for log search and storage, [Elasticsearch](https://www.elastic.co/elasticsearch) indexes and stores logs and detections collected from Elastic and Wazuh Agents, allowing for fast, real-time querying of security events. Elasticsearch enables users to search and filter large datasets efficiently, providing a powerful backend for data analysis and visualization in Kibana. Its scalability and flexibility make it essential for handling the high-volume log data generated across different endpoints within LME's architecture.
+
+### Data Visualization and Querying:
+[Kibana](https://www.elastic.co/kibana) is the visualization and analytics interface in LME, providing users with tools to visualize and monitor log data stored in Elasticsearch. It enables the creation of custom dashboards and visualizations, allowing users to easily track security events, detect anomalies, and analyze trends. Kibana's intuitive interface supports real-time insights into the security posture of an organization, making it an essential tool for data-driven decision-making in LME’s centralized logging and security monitoring framework.

docs/markdown/reference/architecture.md

@@ -0,0 +1,9 @@
+# General notes on custom configuration for LME
+The configuration files are located in /config/. These steps will guide you through setting up LME.
+
+## certificates and user passwords:
+  - instances.yml defines the certificates to be created.
+  - Shell scripts will initialize accounts and generate certificates. They run from the quadlet definitions lme-setup-accts and lme-setup-certs.
+   
+## Podman Quadlet Configuration
+- Quadlet configuration for containers is located in /quadlet/. These map to the root systemd unit files but execute as non-privileged users.

docs/markdown/reference/configuration.md

@@ -0,0 +1,41 @@
+# Password Encryption:
+Ansible-vault is used to enable password encryption, securely storing all LME user and service user passwords at rest
+We do submit a hash of the password to Have I Been Pwned to check to see if it is compromised: [READ MORE HERE](https://haveibeenpwned.com/FAQs), but since they're all randomly generated this should be rare.
+
+### Where Are Passwords Stored?:
+```bash
+# Define user-specific paths
+USER_VAULT_DIR="/etc/lme/vault"
+PASSWORD_FILE="/etc/lme/pass.sh"
+```
+
+### Grabbing Passwords: 
+To view the appropriate service user password run the following commands:
+```bash
+#script:
+$CLONE_DIRECTORY/scripts/extract_secrets.sh -p #to print
+
+#add them as variables to your current shell
+source $CLONE_DIRECTORY/scripts/extract_secrets.sh #without printing values
+source $CLONE_DIRECTORY/scripts/extract_secrets.sh -q #with no output
+```
+
+### Manually Setting Up Passwords and Accessing Passwords **Unsupported**:
+**These steps are not fully supported by CISA and are left if others would like to support this in their environment**
+
+Run the password_management.sh script:
+```bash
+lme-user@ubuntu:~/LME-TEST$ sudo -i ${PWD}/scripts/password_management.sh -h
+-i: Initialize all password environment variables and settings
+-s: set_user: Set user password
+-p: Manage Podman secret
+-l: List Podman secrets
+-h: print this list
+```
+
+A cli one liner to grab passwords (this also demonstrates how we're using Ansible-vault in extract_secrets.sh):
+```bash
+#where wazuh_api is the service user whose password you want:
+USER_NAME=wazuh_api
+sudo -i ansible-vault view /etc/lme/vault/$(sudo -i podman secret ls | grep $USER_NAME | awk '{print $1}')
+```