Skip to content

Conversation

@GarrettBeatty
Copy link
Contributor

@GarrettBeatty GarrettBeatty commented Nov 11, 2025

This change adds config options for multi part download.

Description

The three config options we are adding are

  1. MultiPartDownloadType - this is self explanatory, it determines which strategy we will use to call s3 to get each file part.
  2. PartSize - This is the part size to use when MultipartDownloadType is set to RANGE. The default value will eventually be set to 8MB.
  3. MaxInMemoryParts - this is the number of parts that will be buffered in memory when using stream based downloading. The reason that this value is in the TransferUtilityConfig instead of the individual request is because it will follow the same pattern of MultiPartUpload and ConcurrentServiceRequests, where MaxInMemoryParts will be shared across all files in a directory download.

Motivation and Context

  1. This is the first PR in implementing multi part download.
  2. S3 SDK is far slower than S3 CLI #3806

Testing

  1. Dry Run f4f3a190-0165-4432-ab54-11ec09e43a90 pass

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist

  • My code follows the code style of this project
  • My change requires a change to the documentation
  • I have updated the documentation accordingly
  • I have read the README document
  • I have added tests to cover my changes
  • All new and existing tests passed

License

  • I confirm that this pull request can be released under the Apache 2 license

@GarrettBeatty GarrettBeatty changed the title Update TransferUtilityConfig and BaseDownloadRequest to add multi par… Update TransferUtilityConfig and BaseDownloadRequest to add multipartdownload config options Nov 11, 2025
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds configuration options for multipart download functionality to the AWS S3 Transfer Utility. It introduces memory management controls and download strategy options as the first step toward implementing multipart download support (issue #3806).

Key changes:

  • Added MaxInMemoryParts property to control memory buffering during downloads
  • Added PartSize property to configure download chunk sizes
  • Added MultipartDownloadType enum to select between part-based and range-based download strategies

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File Description
TransferUtilityConfig.cs Adds MaxInMemoryParts property with default value of 1024 parts to control memory usage during multipart downloads
BaseDownloadRequest.cs Adds MultipartDownloadType enum and properties (PartSize, MultipartDownloadType) to configure multipart download behavior
19ed68ce-9f46-4e1e-a0ff-45a2b3641946.json DevConfig file specifying patch version bump with changelog entries for the new properties

Comment on lines +90 to +96
public int MaxInMemoryParts
{
get { return this._maxInMemoryParts; }
set
{
if (value < 1)
value = 1;
Copy link

Copilot AI Nov 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The setter silently modifies invalid input values instead of throwing an exception. This differs from standard .NET property validation patterns and can mask configuration errors. Consider throwing an ArgumentOutOfRangeException for values less than 1, similar to how the ConcurrentServiceRequests property's remarks indicate that invalid values are 'silently ignored' but the actual behavior is to clamp to a minimum. For consistency and clarity, either document this clamping behavior in the XML documentation or throw an exception for invalid values.

Suggested change
public int MaxInMemoryParts
{
get { return this._maxInMemoryParts; }
set
{
if (value < 1)
value = 1;
/// <remarks>
/// Throws <see cref="ArgumentOutOfRangeException"/> if the value is less than 1.
/// </remarks>
public int MaxInMemoryParts
{
get { return this._maxInMemoryParts; }
set
{
if (value < 1)
throw new ArgumentOutOfRangeException(nameof(value), "MaxInMemoryParts must be greater than 0.");

Copilot uses AI. Check for mistakes.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wondering what the correct procedure is. i think argumentoutorrange makes sense but im not sure why we silently updated the value for concurrentservicerequests?

public long PartSize
{
get { return this.partSize.GetValueOrDefault(); }
set { this.partSize = value; }
Copy link

Copilot AI Nov 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PartSize property lacks input validation. Negative or zero values could lead to unexpected behavior when used for multipart downloads. Consider adding validation in the setter to ensure the value is positive, similar to the validation pattern used in TransferUtilityConfig.MaxInMemoryParts.

Suggested change
set { this.partSize = value; }
set
{
if (value <= 0)
{
throw new ArgumentOutOfRangeException(nameof(PartSize), "PartSize must be a positive value.");
}
this.partSize = value;
}

Copilot uses AI. Check for mistakes.
Copy link
Contributor Author

@GarrettBeatty GarrettBeatty Nov 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we dont do this for multi part upload in this area

not sure if this makes sense to do here, or have the validation later on

@GarrettBeatty GarrettBeatty marked this pull request as ready for review November 11, 2025 20:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant