Skip to content

Feature request: conditions API #129

@alsenz

Description

@alsenz

Firstly thank you for this library I've just started using it and the experience is really good.

Many object stores now support conditional gets and writes. The former allows better caching, and the latter allows developers to write data to an object if and only if the object hasn't been changed since a previous point in time, which forms the backbone for 'atomic' updates to datasets on object storage (a reference use case would be databricks delta format).

Generally object store providers either provide this functionality using a conditional header or payload containing an etag, a version or generation or even a last modified.

Examples:
GCS: https://pkg.go.dev/cloud.google.com/go/storage#hdr-Conditions
Azure: https://learn.microsoft.com/en-us/rest/api/storageservices/specifying-conditional-headers-for-blob-service-operations

Note - (infamously?) AWS S3 doesn't support this. The standard response to this seems to be to ask library users to implement their own locking provider, and not address this in the library.

Here is an example of another library that has nice API for this: https://docs.rs/object_store/latest/object_store/struct.UpdateVersion.html

My thought would be that the ObjectAttributes API could include this metadata, and then I'm not sure how the API would be be changed conservatively to support this for Get or Upload; I quite like the minimal addition of an If(condition) receiver function that returns a wrapped object (similar to the cloud-storage library).

If this feature gets the maintainers' blessing I am happy to contribute pull requests for GCS and a suitable-for-testing purposes version for LocalFilesystem.

Thanks again.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions