[1/5] Set up basic Azure sandbox environment, providers, etc. (#136)

Author: kmcclenn

.gitignore

@@ -58,3 +58,45 @@ scripts/python/env
 # Ignore autogenerated dummy tokens for local docker-compose cluster
 infra/recipes/docker-compose/oh-hadoop-spark/openhouse.token
 infra/recipes/docker-compose/oh-hadoop-spark/u_tableowner.token
+
+# For Terraform:
+
+# Local .terraform directories
+**/.terraform/*
+
+# .tfstate files
+*.tfstate
+*.tfstate.*
+
+# Crash log files
+crash.log
+crash.*.log
+
+# Exclude all .tfvars files, which are likely to contain sensitive data, such as
+# password, private keys, and other secrets. These should not be part of version 
+# control as they are data points which are potentially sensitive and subject 
+# to change depending on the environment.
+*.tfvars
+*.tfvars.json
+
+# Ignore override files as they are usually used to override resources locally and so
+# are not checked in
+override.tf
+override.tf.json
+*_override.tf
+*_override.tf.json
+
+# Ignore transient lock info files created by terraform apply
+.terraform.tfstate.lock.info
+
+# Include override files you do wish to add to version control using negated pattern
+# !example_override.tf
+
+# Include tfplan files to ignore the plan output of command: terraform plan -out=tfplan
+# example: *tfplan*
+
+# Ignore CLI configuration files
+.terraformrc
+terraform.rc
+
+.terraform.lock.hcl

infra/recipes/terraform/azure/README.md

@@ -0,0 +1,36 @@
+# Project structure and description
+
+This directory is intended to provide a customizable configuration for Microsoft Azure using Terraform.
+
+## Directory structure
+
+### Overall structure
+    .
+    ├── environments            # Various specific environment configurations for Azure
+        ├── sandbox             # The sandbox environment
+    ├── modules                 # Reusable Azure components that are pre-defined
+        ├── k8s                 # The Azure kubernetes configurations and services
+        ├── mysql               # The Azure MySQL configurations and services
+        ├── storage             # The Azure storage configurations and services
+        ├── vm                  # The Azure virtual machine and network configurations and services
+    ├── k8s_variables.tf             # Specific variables used for the kubernetes provider configuration
+    ├── provider.tf             # The global azure and other providers needed for the configurations
+    ├── variables.tf            # Global variables that can be reused across modules/environments
+    ├── versions.tf             # The Azure Resource Manager version
+    └── README.md
+
+### Module/environment structure
+
+Each module and environment contains a structure that is similar to the following (with occasional other specific files):
+
+    .
+    ├── main.tf                 # The main terraform file that defines all the necessary services
+    ├── variables.tf            # The module/environment specific variables
+    ├── outputs.tf              # The output variables from the module/environment
+    └── common_variables.tf     # A symlink from the global variables file
+
+## Adding/editing modules/environments
+
+To add a new module, create a new folder in the `/modules` directory with the same format as above. Define the services in `main.tf`, the local variables in `variables.tf`, and the outputs in `outputs.tf`. `main.tf` will be the primary place to edit/update modules. 
+
+To add a new environment, create a new folder in the `/environments` directory with the same format as above. For each environment add a `README` file as well as optional other files such as a `backend.tf` file. See the `environments/sandbox/` environment for examples. Instead of adding numerous new resources to the `main.tf` file in an environment, add modules that have been previously defined to maximize code reusability.

infra/recipes/terraform/azure/environments/sandbox/README.md

@@ -0,0 +1,58 @@
+# Initialize and Run Azure Sandbox
+Ensure you have a Azure account. As part of sandbox we use Terraform to provision a resource group, storage account, and blob container within that storage account. 
+
+## Creating an Azure Account
+
+Create an Azure account [here](https://azure.microsoft.com/en-us/pricing/purchase-options/azure-account). If you haven't created one already, you can use the free trial to get free credits. Create a subscription using this free trial, or with another option if the free trial is over.
+
+### Creating a Storage Account
+
+A storage account must be created to hold the backend terraform files. Create a new storage account using an arbitrary resource group. It is typical to name these resources starting with `tfbackend`. Then, within this storage account, create a new blob container. Note the resource group name, storage account name, and blob container name. Also note the storage account key, which can be found in the "Access keys" sidebar option in the storage account. These variables will be used when configuring the backend.
+
+## Azure Login
+
+Install the [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/), and ensure that you are logged in with your Azure account. You can login in multiple ways.
+
+### Using username/password
+
+This is the simplest version. Run `az login` and login to your account on a web browser.
+
+### Using Service Principal
+
+Following the steps [here](https://learn.microsoft.com/en-us/cli/azure/authenticate-azure-cli-service-principal), you can run `az login --service-principal -u <app-id> -p <password-or-cert> --tenant <tenant>` to gain more control over your subscription and account.
+
+## Terraform Backend
+ 
+Terraform backend state can be stored locally or in Azure blob store. To configure this you will be setting the variables like those in `infra/recipes/terraform/azure/backend.tfvars.template`. These variables are the ones found when setting up the storage account.
+
+### Using Environment Variables
+You could set the environment variables for variables to stored terraform backend state. For example, to initialize `storage_account_name` you can set the following environment variable.
+`export TF_VAR_storage_account_name="foobar"`
+
+### Using tfvars
+1. Create a copy of the file backend.tfvars.template by running
+`cp backend.tfvars.template backend.tfvars`.
+2. Edit the backend.tfvars to configure the necessary TF backend state.
+
+## Deployment
+
+### Installation
+
+Install the `terraform` CLI by following [this](https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli) link.
+
+### Initialization
+
+Run `terraform init` when setting up a new configuration, changing backend settings, or modifying provider versions. You don't need to run it for regular operations like `terraform plan` and `terraform apply` unless the configuration changes.
+
+_If using tfvars_
+`terraform init -backend-config="./backend.tfvars"`
+_If using environment variables_
+`terraform init`
+
+### Planning & Applying
+
+`terraform plan` to see the plan generated by terraform for the specified deployment configuration. If you are satisfied with the plan, run `terraform apply` to apply the configuration.
+
+## Deconfiguration
+
+Run `terraform destroy` once you are done to tear down the services you have created.

infra/recipes/terraform/azure/environments/sandbox/backend.tf

@@ -0,0 +1,3 @@
+terraform {
+    backend "azurerm" {}
+}

infra/recipes/terraform/azure/environments/sandbox/backend.tfvars.template

@@ -0,0 +1,4 @@
+resource_group_name = "<RESOURCE GROUP NAME HERE>"
+storage_account_name = "<ACCOUNT NAME HERE>"
+container_name = "<BLOB CONTAINER NAME HERE>"
+key = "<BLOB CONTAINER KEY HERE>"

infra/recipes/terraform/azure/environments/sandbox/common_provider.tf

@@ -0,0 +1 @@
+../../provider.tf

infra/recipes/terraform/azure/environments/sandbox/common_variables.tf

@@ -0,0 +1 @@
+../../variables.tf

infra/recipes/terraform/azure/environments/sandbox/main.tf

@@ -0,0 +1,35 @@
+resource "azurerm_resource_group" "openhouse_sandbox" {
+    name = var.resource_group_name
+    location = var.resource_group_location
+}
+
+module "vm" {
+    source = "../../modules/vm"
+    virtual_network_name = "openhouse-sandbox-network"
+    resource_group_name = var.resource_group_name
+    subnet_name = "openhouse-sandbox-subnet"
+}
+
+module "mysql" {
+    source = "../../modules/mysql"
+    subnet_id = module.vm.subnet_id
+    resource_group_name = azurerm_resource_group.openhouse_sandbox.name
+    server_name = "openhouse-sandbox-mysql-server"
+    db_admin_login = "azureadmin"
+    db_admin_password = "Pa33word"
+    db_name = "openhouse-sandbox-db"
+}
+
+module "k8s" {
+  source = "../../modules/k8s"
+  k8s_cluster_name = "openhouse-sandbox-k8s"
+  resource_group_name = azurerm_resource_group.openhouse_sandbox.name
+  node_count = 1
+  vm_size = "Standard_D2s_v3"
+}
+
+module "storage" {
+    source = "../../modules/storage"
+    storage_account_name = "openhousestorage729387" // added random string of numbers to make it unique
+    resource_group_name = azurerm_resource_group.openhouse_sandbox.name
+}

infra/recipes/terraform/azure/environments/sandbox/outputs.tf

@@ -0,0 +1,3 @@
+output "resource_group_name" {
+  value = azurerm_resource_group.openhouse_sandbox.name
+}