feat!: Upgrade AWS provider and min required Terraform version to 6.19 and 1.5.7 respectively

[bryantbiggs]

Description

List of backwards incompatible changes

• Terraform v1.5.7 is now minimum supported version
• AWS provider v6.19 is now minimum supported version
• Kubernetes provider v2.38 is now minimum supported version (EMR on EKS virtual cluster sub-module)
• aws_security_group_rule resources have been split into aws_vpc_security_group_ingress_rule and aws_vpc_security_group_egress_rule resources to better match the AWS API and allow for more flexibility in defining security group rules. Prior variable names of *_security_group_rules have been split into *_security_group_ingress_rules and *_security_group_egress_rules to match.

Additional changes

Added

• Support for region parameter to specify the AWS region for the resources created if different from the provider region.

Modified

• Variable definitions now contain detailed object types in place of the previously used any type.
• Ensure data sources are gated behind create flags to prevent unnecessary API calls.
• release_label_filters.prefix now defaults to emr-7, was previously emr-6.
• unhealthy_node_replacement now defaults to true
• aws_service_principal data source is now used to fetch the correct service principals (instead of trying to construct them psuedo-manually with the DNS suffix).

Variable and output changes

1. Removed variables:

   • serverless sub-module

     • None

   • studio sub-module

     • None

   • virtual_cluster sub-module

2. Renamed variables:

   • master_security_group_rules -> master_security_group_ingress_rules and master_security_group_egress_rules

   • slave_security_group_rules -> slave_security_group_ingress_rules and slave_security_group_egress_rules

   • service_security_group_rules -> service_security_group_ingress_rules and service_security_group_egress_rules

   • serverless sub-module

     • security_group_rules -> security_group_ingress_rules and security_group_egress_rules
     • release_label_prefix -> release_label_filters

   • studio sub-module

     • engine_security_group_rules -> engine_security_group_ingress_rules and engine_security_group_egress_rules
     • workspace_security_group_rules -> workspace_security_group_ingress_rules and workspace_security_group_egress_rules

   • virtual_cluster sub-module

     • eks_cluster_id -> eks_cluster_name to better match API of EKS module/resources
     • oidc_provider_arn -> eks_oidc_provider_arn for clarity to show its related to EKS authentication

3. Added variables:

   • os_release_label

   • serverless sub-module

     • monitoring_configuration
     • runtime_configuration
     • scheduler_configuration

   • studio sub-module

     • None

   • virtual_cluster sub-module

     • cloudwatch_log_group_class

Motivation and Context

• Resolves feat: Support new EMR label in 6.2.0 of aws provider #40
• Resolves Add SchedulerConfiguration to EMR serverless module #42
• Resolves Add support for Runtime and Monitoring configuring in aws_emrserverless_application #44

Breaking Changes

• Yes

How Has This Been Tested?

• I have updated at least one of the examples/* to demonstrate and validate my change(s)
• I have tested and validated these changes using one or more of the provided examples/* projects
• I have executed pre-commit run -a on my pull request

[antonbabenko]

LGTM, minor things

[antonbabenko]

Redshift?

[bryantbiggs]

too many modules at once 😅 - corrected in e6b4943

[antonbabenko]

This PR is included in version 3.0.0 🎉