Performance: `getImage()` or `parseFileDirectoryAt()` fetches huge TileOffsets and TileByteCounts arrays, causing massive downloads for highly-tiled COGs

[MariDani]

Problem

When reading a very large, highly-tiled Cloud-Optimized GeoTIFF (e.g., 7 GB), the initial call to getImage() or parseFileDirectoryAt() triggers an unexpectedly large data download. For one of our test files, this initial metadata request downloads ~500 MB of data.

We have observed that this behavior is caused by the library fetching the entire TileOffsets and TileByteCounts arrays from the IFDs. For an image with millions of tiles, these arrays can be hundreds of megabytes in size, which defeats the purpose of the COG format's selective data access.

This makes the library difficult to use for efficiently visualizing very large-scale raster data on the web.

Behavior Comparison

The library @cogeotiff/core handles this scenario differently. Due to its lazy-loading approach, it appears to parse the IFD without immediately fetching the tile index arrays. For the same 7 GB COG file, it only requires ~7 MB of initial data transfer to begin visualization, which is the expected behavior.

Steps to Reproduce

The issue can be easily verified on the official geotiff.js demo applications.

A. Reproducing on Official Demo Sites (No Code Required)

Navigate to geotiff.io or the COG Explorer.
Open the browser's developer tools and select the Network tab.
In the application, open the test file via URL: https://gisat-data.eu-central-1.linodeobjects.com/WorldCereal_GST-10/project/demo/merged_cog.tif
Observe the network requests. A large download of ~276 MB will be initiated to fetch the IFD before any part of the map is rendered.

B. Reproducing with Code

Use geotiff.js to open a large COG that contains a high number of tiles.
Call the getImage() method on the highest-resolution image.
Observe the network tab in the browser's developer tools and note the large download size.

import { fromUrl } from 'geotiff';

const cogUrl = 'https://gisat-data.eu-central-1.linodeobjects.com/WorldCereal_GST-10/project/demo/merged_cog.tif';

const tiff = await fromUrl(cogUrl);
const image = await tiff.getImage();

Technical Details of Example Data

The problem can be reproduced with the following public file:
https://gisat-data.eu-central-1.linodeobjects.com/WorldCereal_GST-10/project/demo/merged_cog.tif

File Analysis: The file is a valid COG created with the rio-cogeo tool (files created with the gdal_translate tool behaved the same way). The first image (Image 0) contains over 17 million tiles (1083648x1044736 pixels at 256x256 tile size).
IFD Size: The total size of the IFD headers, primarily TileOffsets and TileByteCounts, is ~276 MB. This corresponds directly to the data being downloaded on initialization.
Below is a print screen of the console log of the first IFD with visible large arrays for TileOffsets and TileByteCounts.

Proposed Solution / Feature Request

We are curious if we are missing some important piece of the logic within the geotiff.js library functions in solving our problem. Is there a way to use the geotiff.js library without fetching the full TileOffsets and TileByteCounts arrays? For example, an optional lazy loading? This would keep the initial metadata download minimal and align with the performance expectations of the COG format.

[constantinius]

@MariDani

Thanks for reporting this issue. geotiff.js takes a simplified approach when parsing an IFD, as it simply aligns values that are directly stored as tags and values that are referenced from the tags but are stored outside the IFD as they don't fit the very narrow size limitations. Whenever we access an IFDs field values, we just treat them as "already retrieved and parsed", as this is way simpler and faster than doing this via lazy loading (which also might be async).

When implementing a lazy-loading approach, it may introduce performance issues for other use cases, as it may introduce additional round-trip requests. So we would need to add an option to opt-in to the new functionality.

In my opinion, this is certainly doable, but would require a considerable re-write of the library.

[MariDani]

Hi @constantinius,

Thank you for your answer and the detailed explanation of the geotiff.js approach to parsing the IFD. I was hoping there was an existing feature I was missing that would solve our issue.

I completely understand that implementing lazy loading would be a significant undertaking and not a simple patch.

Is this a direction you might consider for the library's future roadmap? We are very interested in this functionality and want to see if it aligns with your vision for the project.

Many thanks.