[Initiative]: Cloud Native AI Sustainability - Challenges & Opportunities

[payamohajeri]

Name

Practical Challenges & Opportunities for Implementing Sustainable AI in Cloud-Native Environments

Short description

To discuss practical challenges and opportunities for implementing sustainable AI in cloud-native environments, focusing on data, models, systems, and operations.

Responsible group

TOC

Does the initiative belong to a subproject?

Yes

Subproject name

Cloud Native AI Working Group

Primary contact

Payam Mohajeri ( @payamohajeri )

Additional contacts

Marisol Palmero ( @marisolpalmero )

Initiative description

This is the second whitepaper orginially developed as part of: #1675

The paper "Practical Challenges & Opportunities for Implementing Sustainable AI in Cloud-Native Environments," explores the difficulties and chances to make artificial intelligence more environmentally friendly when it's used with cloud computing. It looks at various problems, such as the large amount of data AI uses, making AI models more efficient, and managing the computer systems that run AI. At the same time, it highlights opportunities to improve these areas through better data handling, smarter AI models, and more efficient system operations, all with the goal of reducing AI's impact on the environment.

Source: https://docs.google.com/document/d/1J1xxNiyFybO6LS5YHh5cTNGuTXsZ3ke0hlGGFBMoj_A/edit?usp=sharing

Deliverable(s) or exit criteria

Final draft versions of Cloud Native AI Sustainability - Challenges & Opportunities paper to be handed off to the CNCF publishing staff.

Tracking document for meeting and progress

https://docs.google.com/document/d/1DVkhCRQ9r0D1VOJpVgalR5GwCrQDhCLsffiEKwUN46U/edit?usp=sharing

[marisolpalmero]

This is the ToC that I would suggest to follow, considering the content already created for this document:

Cloud Native AI Sustainability - Challenges & Opportunities

Authors: xxxx

1. Introduction: Sustainable AI in the Cloud

The Urgency of Sustainable AI

[Explain the growing energy consumption of AI, the environmental impact, and why it's crucial to adopt sustainable practices now.]

Scope: Generative AI and Cloud-Native Environments

[Clearly define the scope: Focusing on Generative AI models within cloud-native infrastructures like Kubernetes. Exclude areas not directly relevant.]

Document Goals

[State the purpose: to identify key challenges and actionable solutions for building more sustainable AI systems within cloud-native environments.]

2. Key Challenges in Cloud-Native Generative AI

Data Inefficiency: Data Footprint, Lifecycle, and Management

[Describe the problem of large data volumes, energy costs of storage, and the need for efficient data lifecycle practices. Identify stakeholders: Data Scientists, Infrastructure Operators.]

Model Complexity: Size, Efficiency, and Power Consumption

[Explain the challenge of large model sizes, high computational requirements, and energy consumption during training and inference. Identify stakeholders: AI/ML Engineers, SREs.]

Inference Bottlenecks: KV Cache, Scaling, and Infrastructure

[Detail the challenges of scaling inference while managing memory pressure, latency, and resource utilization. Identify stakeholders: Infrastructure Operators, Al Application Developers.]

Training Overhead: Scheduling, Tuning, and Resource Utilization

[Describe inefficiencies in training job scheduling, hyperparameter tuning, and overall resource usage. Identify stakeholders: AI/ML Engineers, Data Scientists.]

Systemic Inefficiencies: Resource Management and Monitoring

[Address the lack of real-time energy optimization and sustainability metrics in existing AI infrastructures. Identify stakeholders: Operations Engineers, SREs.]

3. Technical Solutions and Green Practices

Data Efficiency

Synthetic Data Generation

[Explain synthetic data, its benefits (reducing reliance on energy-intensive data collection), and practical implementation tips.]

Data Compression and Optimization

[Describe efficient compression techniques, optimized file formats, and deduplication methods.]

Data Lifecycle Management

[Discuss data versioning, access control, and automated archiving strategies.]

Model Optimization

Pruning

[Explain the concept of pruning (removing unimportant connections), its benefits, and tools.]

Quantization

[Describe quantization (reducing numerical precision), its benefits, and hardware support.]

Knowledge Distillation

[Explain knowledge distillation (transferring knowledge from larger to smaller models), its benefits, and use cases.]

Infrastructure & Scheduling

GPU Partitioning & Slicing

[Describe how partitioning and slicing GPUs can improve resource utilization.]

Dynamic Resource Allocation (Kueue, Volcano)

[Explain how dynamic scheduling in Kubernetes can optimize resource allocation.]

Multi-Tenancy Optimization

[Discuss strategies for efficiently sharing resources across multiple tenants.]

Inference Optimization

KV Cache Reuse

[Describe techniques for reusing cached keys and values to reduce recomputation.]

In-Flight Batching

[Explain how in-flight batching can improve throughput.]

Speculative Inference

[Describe speculative inference and its benefits for latency and efficiency.]

4. Monitoring and Observability

Importance of Energy Tracking and Sustainability Metrics

[Explain why it's crucial to track energy consumption and sustainability metrics in AI systems.]

Key Tools and Standards

Kepler

[Describe Kepler and its ability to measure power consumption.]

Prometheus & Grafana

[Explain how to use these tools to monitor and visualize sustainability metrics.]

Telemetry Standards

[Highlight the need for standardized telemetry for interoperability.]

5. Call to Action and Conclusion

CNCF's Role

[Outline specific actions the CNCF can take to promote sustainable AI practices (e.g., standards, integrations, certifications).]

Key Recommendations

[Summarize the most important recommendations from the document.]

Authors/Contributors

[List the authors.]

[marisolpalmero]

By the way, not sure if we have a glossary somewhere. I will avoid terms like Power consumption vs Energy consumption. Power efficiency vs Energy Efficiency. And "green" related terms. When we talk about Sustainable(more linked to the lifecycle assesment practices) let's talk about it, or Energy Optimization practices, we can also talk about it.