|
| 1 | +# Guidance for boosting database performance reads with AWS ElastiCache |
| 2 | + |
| 3 | +## Table of Content |
| 4 | + |
| 5 | +1. [Overview](#overview) |
| 6 | + - [Cost](#cost) |
| 7 | +2. [Prerequisites](#prerequisites) |
| 8 | + - [Operating System](#operating-system) |
| 9 | +3. [Deployment Steps](#deployment-steps) |
| 10 | +4. [Deployment Validation](#deployment-validation) |
| 11 | +5. [Running the Guidance](#running-the-guidance) |
| 12 | +6. [Next Steps](#next-steps) |
| 13 | +7. [Cleanup](#cleanup) |
| 14 | +8. [FAQ, known issues, additional considerations, and limitations](#faq-known-issues-additional-considerations-and-limitations) |
| 15 | +9. [Notices](#notices) |
| 16 | + |
| 17 | +## Overview |
| 18 | + |
| 19 | +1. This guidance was created to help customers with database workloads that have high read:write (70:30) ratios and are looking to boost application performance, and at the same time reduce overall cost. Qualifying database workloads will see an increase in the number of transactions, a reduction in response time, and an overall reduction in cost. It is expected that two services together can perform a task faster. However, when AWS ElastiCache is paired with qualifying database workloads not only the performance increases but the total cost of the two services is lower than the cost of scaling the database alone to deliver a similar performance. |
| 20 | + |
| 21 | +#### Architecture overview #### |
| 22 | + |
| 23 | +<img width="676" alt="architecture-for-adding-caching-to-a-database" src="assets/images/architecture-diagram.png"> |
| 24 | + |
| 25 | +### Cost |
| 26 | + |
| 27 | +You are responsible for the cost of the AWS services used while running this Guidance. |
| 28 | + |
| 29 | +As of 03/28/2024, the cost for running this guidance with the default settings using AWS ElastiCache instance type cache.t2.x.small utilizing 1 primary, in the US East (N. Virginia) for on-demand pricing is approximately $24.82 USD total per month. AWS The RDS database cost is estimated at $14.71. However, cost will greatly depend on the, instance size, and RDS licensing model selected. Reserved instance pricing will reduce cost for both RDS and ElastiCache. ElastiCache is also available in a serverless offering where pay-per-consumption cost model is applicable. |
| 30 | + |
| 31 | +| Service | Assumptions | Estimated Cost Per Month | |
| 32 | +| --------------------- | ------------------------------------------------- | ------------- | |
| 33 | +| Amazon ElastiCache | 2 Instance (cache.t2.small) used for 730 hours | $24.82 | |
| 34 | +| Amazon RDS MySQL | 1 Instance (db.t3.micro) used for 730 hours | $14.71 | |
| 35 | + |
| 36 | + |
| 37 | +## Prerequisites |
| 38 | + |
| 39 | +This guidance is targeted towards those familiar with the AWS RDS Service. The users are expected to have a basic understanding of AWS RDS service and database access patterns. It guides users how to utilize AWS ElastiCache in addition to their existing relational database. Effectively paring your database with a caching service. It should be run in US East N.Virginia region. |
| 40 | + |
| 41 | + |
| 42 | +### Operating System |
| 43 | + |
| 44 | +Tis guidance runs in the AWS cloud utilizing an AWS EC2 compute instance based on the Amazon Linux 2023 AMI. With network access to both AWS RDS database service and AWS ElastiCache (both are required). In addition the EC2 compute instance will require public access on port 8888. The included Cloud Formation template can be used to create such an EC2 instance. The sample code also includes two Jupyter notebooks to analyze and visually plot the performance results. Note that public access to the EC2 host on port 8888 should be enabled from your computer only not all end user computers. |
| 45 | + |
| 46 | +### Software dependencies |
| 47 | + |
| 48 | +Install dependencies by executing the setup_host.sh script. This script will install gcc python3-devel at the host level. In addition to the two packages installed a python virtual environment is created with dependent modules installed from the requirements.txt file. The python modules are only committed to the virtual environment not the host. The includes commands are optimized to work on the EC2 instance created by the included CloudFormation template and are specific for the Amazon Linux 2023 AMI al2023-ami-2023.4.20240319.1-kernel-6.1-arm64. This image is specific to the us-east-1 region. Other OS or AMI configuration may require additional steps. |
| 49 | + |
| 50 | +- At the OS prompt execute the setup_host.sh script to install and configure all necessary software and to create the Python virtual environment. |
| 51 | + |
| 52 | +### Third-party tools |
| 53 | + |
| 54 | +This guidance uses Jupyter lab and the included notebooks to visualize/plot the performance data captured in json logs. |
| 55 | +For convenience reasons a Jupyter password has been set to 'test123'. |
| 56 | +Seed data is from the Airportdb sample database located here: https://dev.mysql.com/doc/airportdb/en/. However, by modifying the read and write queries, any seed data can be used. |
| 57 | + |
| 58 | +### AWS account requirements |
| 59 | + |
| 60 | +Ability to create an EC2 instance and networking configuration to permit access to both the RDS server and the ElastiCache service and public access to the EC2 on port 8888 from the customer end computer only. (CIDR/32) |
| 61 | + |
| 62 | +**Example resources:** |
| 63 | +- RDS MySQL Database with the airportdb data loaded |
| 64 | +- AWS ElastiCache |
| 65 | +- VPC |
| 66 | +- SSH key in your region of choice |
| 67 | + |
| 68 | +### Supported Regions |
| 69 | + |
| 70 | +All regions where AWS RDS MySQL and AWS ElastiCache are offered. |
| 71 | + |
| 72 | +## Deployment Steps |
| 73 | + |
| 74 | +1. Create an EC2 instance with at least 1GB of memory using the Amazon Linux 2023 image. For convenience reason the repository includes the cloud formation template called guidance-ec2.yaml. Use AWS CloudFormation and with this template to create and EC2 instance. |
| 75 | +2. Log in to your instance from the AWS console via Session Manager or via SSH. |
| 76 | +3. Clone the repository by executing ```git clone <repo name> ``` |
| 77 | +4. Change directory to the guidance directory ```cd guidance``` |
| 78 | +5. Execute the setup_host script ```./setup_host.sh``` |
| 79 | +6. Log in to the same instance from a separate session and navigate to the same directory and execute ```./setup_jupyter.sh``` script. |
| 80 | +7. In your computer browser enter the EC2's public IP address and port for example ```http://1.2.3.4:8888`` Note that this is not a secured service. |
| 81 | +8. Enter the preset password for your Jupyter notebook "test123" |
| 82 | +9. In your first session edit the .env file and update it with your database and ElastiCache related information. |
| 83 | +10. Source the .env file ```source .env``` to export the parameters. |
| 84 | + |
| 85 | +## Deployment Validation |
| 86 | + |
| 87 | +It is not part of this guidance to install and configure client applications for database and ElastiCache connectivity. However, at this point you can install client applications to validate connectivity to both the database and ElastiCache. |
| 88 | + |
| 89 | +## Running the Guidance |
| 90 | + |
| 91 | +* Execute the `scenario01.py` script. This workload accesses the database only and captures command level performance data in a logfile. In the directory where you executed the `setup_host.sh` and the Python virtual environment is activated, the first connection, execute: |
| 92 | +```bash |
| 93 | +(.venv) [ec2-user]$ python scenario01.py --users 10 --queries 1000 --read_rate 80 |
| 94 | +``` |
| 95 | + |
| 96 | +* If deployment was correct you should see a response similar to this. (small sample execution) |
| 97 | + |
| 98 | +```bash |
| 99 | +(.venv) [ec2-user]$ python scenario01.py --users 1 --queries 10 --read_rate 80 |
| 100 | +Reads: 8 |
| 101 | +Writes: 2 |
| 102 | +Logfile located here: logs/scenario01_139007_mwae8c4k.json |
| 103 | +``` |
| 104 | +* Open the Jupyter notebook `plot_results_db_only.ipynb` file and update the logfile name in the second cell. For example ```log_pattern = 'scenario01_139007_mwae8c4k.json``` |
| 105 | + |
| 106 | +* From the run option select run all cells. The output of the last cell will show both the number of executions per second and the average response time. |
| 107 | + |
| 108 | +* To compare the performance boost provided by ElastiCache repeat the above steps but use the scenario02.py script. For example execute ```python scenario02.py --users 1 --queries 10 --read_rate 80``` The output should be similar. |
| 109 | + |
| 110 | +```bash |
| 111 | +(.venv) [ec2-user]$ python scenario02.py --users 1 --queries 10 --read_rate 80 |
| 112 | +Connected to Database |
| 113 | +Connected to ElastiCache |
| 114 | +Reads: 10 |
| 115 | +Writes: 0 |
| 116 | +Cache hits: 10 |
| 117 | +Cache misses: 0 |
| 118 | +Logfile located here: logs/scenario02_176908_0y2qr55f.json |
| 119 | +``` |
| 120 | + |
| 121 | +* Open the Jupyter notebook `plot_results_db_and_cache.ipynb` file and update the logfile name in the second cell. For example ```log_pattern = 'scenario02_176908_0y2qr55f.json``` |
| 122 | + |
| 123 | +Then select run all cells to plot the performance of the second scenario. Note that a small execution may not be sufficient to demonstrate the performance advantage of adding a cache. |
| 124 | + |
| 125 | +## Next Steps |
| 126 | + |
| 127 | +To see the potential improvements ElastiCache can provide to your application. Replace the database connection parameters with your test database values and modify the READ_QUERY and WRITE_QUERY text parameters to fit your schema. |
| 128 | + |
| 129 | +## Cleanup |
| 130 | + |
| 131 | +To clean up your environment stop all services and delete the MySQL database and ElastiCache cluster. Finally delete the EC2 instance from where you executed the commands. |
| 132 | + |
| 133 | + |
| 134 | +## FAQ, known issues, additional considerations, and limitations |
| 135 | + |
| 136 | +Caching is beneficial for databases that execute a high read to write ratio workloads. For example 80:20 read to write workloads. The sample data provided in the airportdb is subject to change and accepting end user licensing agreement. |
| 137 | + |
| 138 | + |
| 139 | +**Additional considerations ** |
| 140 | + |
| 141 | +- This Guidance creates insecure Jupyter Notebook |
| 142 | +- Cashing is not applicable for workloads that are write intensive meaning that the majority of transactions are not repetitive read transactions. |
| 143 | + |
| 144 | +For feedback please access the github page <here> |
| 145 | + |
| 146 | +## Notices |
| 147 | + |
| 148 | +Include a legal disclaimer |
| 149 | + |
| 150 | +*Customers are responsible for making their own independent assessment of the information in this Guidance. This Guidance: (a) is for informational purposes only, (b) represents AWS current product offerings and practices, which are subject to change without notice, and (c) does not create any commitments or assurances from AWS and its affiliates, suppliers or licensors. AWS products or services are provided “as is” without warranties, representations, or conditions of any kind, whether express or implied. AWS responsibilities and liabilities to its customers are controlled by AWS agreements, and this Guidance is not part of, nor does it modify, any agreement between AWS and its customers.* |
0 commit comments