Merge branch 'main' of https://github.com/data-engineer-hub/spark-on-aks

Author: sharad-hub

readme.md

@@ -0,0 +1,209 @@
+### **Deploying Apache Spark on Azure Kubernetes Service (AKS)**
+
+---
+
+#### **1. Introduction**
+
+Apache Spark is a powerful, open-source engine for big data processing and analytics. Known for its speed and ease of use, Spark has become the backbone of many data-driven organizations. While it traditionally ran on Hadoop, deploying Spark on Kubernetes has gained traction due to Kubernetes' scalability and flexibility.
+
+Azure Kubernetes Service (AKS) further simplifies this process by providing a managed Kubernetes service integrated with Azure's ecosystem. By deploying Spark on AKS, you can unlock powerful data processing capabilities while leveraging Azure’s scalability and monitoring tools.
+
+In this article, we’ll guide you through deploying Apache Spark on AKS, covering prerequisites, setup, deployment, and best practices.
+
+---
+
+#### **2. Prerequisites**
+
+Before we dive into deployment, ensure the following are in place:
+
+- **Knowledge Prerequisites**:  
+  Familiarity with Kubernetes basics, Spark’s architecture, and Azure services.
+
+- **Tools Required**:
+  - An active Azure subscription.
+  - Azure CLI and kubectl CLI installed.
+  - A working AKS cluster.
+  - Docker installed for creating custom images (optional).
+
+Install Azure CLI and kubectl if you haven’t already:
+
+```bash
+# Install Azure CLI
+curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash
+
+# Install kubectl
+az aks install-cli
+```
+
+---
+
+#### **3. Setting up AKS**
+
+Creating an AKS cluster is the first step. You can do this via the Azure Portal or the CLI. Here’s how to use the CLI:
+
+1. **Login to Azure**:
+   ```bash
+   az login
+   ```
+
+2. **Create a Resource Group**:
+   ```bash
+   az group create --name MyResourceGroup --location eastus
+   ```
+
+3. **Create an AKS Cluster**:
+   ```bash
+   az aks create \
+     --resource-group MyResourceGroup \
+     --name MyAKSCluster \
+     --node-count 3 \
+     --enable-addons monitoring \
+     --generate-ssh-keys
+   ```
+
+4. **Connect to the Cluster**:
+   ```bash
+   az aks get-credentials --resource-group MyResourceGroup --name MyAKSCluster
+   kubectl get nodes
+   ```
+
+You should see a list of nodes, confirming your cluster is ready.
+
+---
+
+#### **4. Preparing Apache Spark**
+
+Apache Spark requires Docker images for deployment on Kubernetes. You can use prebuilt images from Docker Hub or build your own.
+
+1. **Using Prebuilt Images**:
+   Pull a prebuilt Spark image:
+   ```bash
+   docker pull bitnami/spark
+   ```
+
+2. **Building a Custom Image**:
+   If your application requires additional dependencies, create a Dockerfile:
+   ```Dockerfile
+   FROM bitnami/spark:latest
+   ADD your-app.jar /opt/spark/jars/
+   CMD ["spark-submit", "--class", "MainClass", "your-app.jar"]
+   ```
+
+   Build and push the image:
+   ```bash
+   docker build -t yourregistry.azurecr.io/spark-custom .
+   docker push yourregistry.azurecr.io/spark-custom
+   ```
+
+3. **Configuration**:
+   Spark uses environment variables like `SPARK_MASTER` to set up the master node. Define these in Kubernetes ConfigMaps.
+
+---
+
+#### **5. Deploying Spark on AKS**
+
+##### **Step 1: Create Kubernetes Manifests**
+Define deployment YAML files for Spark Master and Worker pods.
+
+**spark-master.yaml**:
+```yaml
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: spark-master
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app: spark
+      role: master
+  template:
+    metadata:
+      labels:
+        app: spark
+        role: master
+    spec:
+      containers:
+      - name: spark-master
+        image: yourregistry.azurecr.io/spark-custom
+        ports:
+        - containerPort: 7077
+```
+
+**spark-worker.yaml**:
+```yaml
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: spark-worker
+spec:
+  replicas: 2
+  selector:
+    matchLabels:
+      app: spark
+      role: worker
+  template:
+    metadata:
+      labels:
+        app: spark
+        role: worker
+    spec:
+      containers:
+      - name: spark-worker
+        image: yourregistry.azurecr.io/spark-custom
+        ports:
+        - containerPort: 8081
+```
+
+Apply the manifests:
+```bash
+kubectl apply -f spark-master.yaml
+kubectl apply -f spark-worker.yaml
+```
+
+##### **Step 2: Running a Sample Job**
+Submit a job to your Spark cluster:
+```bash
+kubectl exec -it <master-pod-name> -- spark-submit \
+  --class org.apache.spark.examples.SparkPi \
+  --master spark://<master-service>:7077 \
+  local:/opt/spark/examples/jars/spark-examples.jar 100
+```
+
+---
+
+#### **6. Monitoring and Scaling**
+
+##### **Monitoring**:
+- Use Azure Monitor for node-level insights.
+- Integrate Prometheus and Grafana for detailed metrics on Spark jobs.
+
+##### **Scaling**:
+- Enable **horizontal pod autoscaling** to dynamically adjust worker pods based on workload:
+  ```bash
+  kubectl autoscale deployment spark-worker --cpu-percent=70 --min=2 --max=10
+  ```
+
+---
+
+#### **7. Best Practices**
+
+- **Resource Optimization**: Allocate appropriate CPU and memory limits in your Kubernetes manifests.
+- **Storage Management**: Use Azure Files or Azure Blob Storage for persistent data storage.
+- **Security**: Use RBAC for access control and Secrets to manage sensitive data like credentials.
+
+---
+
+#### **8. Conclusion**
+
+Deploying Apache Spark on AKS offers a robust, scalable solution for big data processing. The combination of Spark’s analytical capabilities and Kubernetes' orchestration ensures your applications run efficiently. With Azure's rich ecosystem, you can integrate Spark with other Azure services for end-to-end data processing pipelines. Start experimenting today and unlock new possibilities in big data analytics!
+
+---
+
+#### **9. Additional Resources**
+- [Apache Spark Official Documentation](https://spark.apache.org/docs/latest/)
+- [Azure Kubernetes Service (AKS) Documentation](https://learn.microsoft.com/en-us/azure/aks/)
+- [Kubernetes Documentation](https://kubernetes.io/docs/home/)
+
+---
+