UnicodeDecodeError when trying to load netcdf file via dap4 (from THREDDS)

[FObersteiner]

What happened?

Loading a netcdf4 file from a THREDDS server via dap4 failed with UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 6202: ordinal not in range(128)

What did you expect to happen?

Dataset loaded.

Minimal Complete Verifiable Example

# /// script
# requires-python = ">=3.12"
# dependencies = [
#   "xarray[complete]@git+https://github.com/pydata/xarray.git@main",
# ]
# ///

import xarray as xr

url = "dap4://thredds.atmohub.kit.edu/thredds/dap4/iagos-caribic/IAGOS-CARIBIC_MS_files_collection_20250711/CARIBIC-2/MS_20200304_591_CPT_MUC_10s_V16.nc"

ds = xr.load_dataset(url, engine="pydap", decode_cf=False, decode_times=False, decode_timedelta=False, decode_coords=False)
print(ds)

Relevant log output

Traceback (most recent call last):
  File "/home/user/Code/Python/pyTesting/netcdf/./dap4_xarray.py", line 18, in <module>
    ds = xr.load_dataset(url, engine="pydap", decode_cf=False, decode_times=False, decode_timedelta=False, decode_coords=False)
  File "/home/user/.cache/uv/environments-v2/dap4-xarray-92e2e08d4d47f0ca/lib/python3.13/site-packages/xarray/backends/api.py", line 165, in load_dataset
    with open_dataset(filename_or_obj, **kwargs) as ds:
         ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.cache/uv/environments-v2/dap4-xarray-92e2e08d4d47f0ca/lib/python3.13/site-packages/xarray/backends/api.py", line 612, in open_dataset
    ds = _dataset_from_backend_dataset(
        backend_ds,
    ...<11 lines>...
        **kwargs,
    )
  File "/home/user/.cache/uv/environments-v2/dap4-xarray-92e2e08d4d47f0ca/lib/python3.13/site-packages/xarray/backends/api.py", line 302, in _dataset_from_backend_dataset
    ds = _maybe_create_default_indexes(backend_ds)
  File "/home/user/.cache/uv/environments-v2/dap4-xarray-92e2e08d4d47f0ca/lib/python3.13/site-packages/xarray/backends/api.py", line 278, in _maybe_create_default_indexes
    return ds.assign_coords(Coordinates(to_index))
                            ~~~~~~~~~~~^^^^^^^^^^
  File "/home/user/.cache/uv/environments-v2/dap4-xarray-92e2e08d4d47f0ca/lib/python3.13/site-packages/xarray/core/coordinates.py", line 315, in __init__
    index, index_vars = create_default_index_implicit(var, list(coords))
                        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
  File "/home/user/.cache/uv/environments-v2/dap4-xarray-92e2e08d4d47f0ca/lib/python3.13/site-packages/xarray/core/indexes.py", line 1638, in create_default_index_implicit
    index = PandasIndex.from_variables(dim_var, options={})
  File "/home/user/.cache/uv/environments-v2/dap4-xarray-92e2e08d4d47f0ca/lib/python3.13/site-packages/xarray/core/indexes.py", line 720, in from_variables
    data = var._data if isinstance(var._data, PandasIndexingAdapter) else var.data  # type: ignore[redundant-expr]
                                                                          ^^^^^^^^
  File "/home/user/.cache/uv/environments-v2/dap4-xarray-92e2e08d4d47f0ca/lib/python3.13/site-packages/xarray/core/variable.py", line 456, in data
    duck_array = self._data.get_duck_array()
  File "/home/user/.cache/uv/environments-v2/dap4-xarray-92e2e08d4d47f0ca/lib/python3.13/site-packages/xarray/core/indexing.py", line 943, in get_duck_array
    duck_array = self.array.get_duck_array()
  File "/home/user/.cache/uv/environments-v2/dap4-xarray-92e2e08d4d47f0ca/lib/python3.13/site-packages/xarray/core/indexing.py", line 897, in get_duck_array
    return self.array.get_duck_array()
           ~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/home/user/.cache/uv/environments-v2/dap4-xarray-92e2e08d4d47f0ca/lib/python3.13/site-packages/xarray/coding/variables.py", line 71, in get_duck_array
    return duck_array_ops.astype(self.array.get_duck_array(), dtype=self.dtype)
                                 ~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/home/user/.cache/uv/environments-v2/dap4-xarray-92e2e08d4d47f0ca/lib/python3.13/site-packages/xarray/core/indexing.py", line 737, in get_duck_array
    array = self.array[self.key]
            ~~~~~~~~~~^^^^^^^^^^
  File "/home/user/.cache/uv/environments-v2/dap4-xarray-92e2e08d4d47f0ca/lib/python3.13/site-packages/xarray/backends/pydap_.py", line 51, in __getitem__
    return indexing.explicit_indexing_adapter(
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        key, self.shape, indexing.IndexingSupport.BASIC, self._getitem
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/user/.cache/uv/environments-v2/dap4-xarray-92e2e08d4d47f0ca/lib/python3.13/site-packages/xarray/core/indexing.py", line 1129, in explicit_indexing_adapter
    result = raw_indexing_method(raw_key.tuple)
  File "/home/user/.cache/uv/environments-v2/dap4-xarray-92e2e08d4d47f0ca/lib/python3.13/site-packages/xarray/backends/pydap_.py", line 56, in _getitem
    result = robust_getitem(self.array, key, catch=ValueError)
  File "/home/user/.cache/uv/environments-v2/dap4-xarray-92e2e08d4d47f0ca/lib/python3.13/site-packages/xarray/backends/common.py", line 296, in robust_getitem
    return array[key]
           ~~~~~^^^^^
  File "/home/user/.cache/uv/environments-v2/dap4-xarray-92e2e08d4d47f0ca/lib/python3.13/site-packages/pydap/model.py", line 526, in __getitem__
    data = self._get_data_index(index)
  File "/home/user/.cache/uv/environments-v2/dap4-xarray-92e2e08d4d47f0ca/lib/python3.13/site-packages/pydap/model.py", line 575, in _get_data_index
    return self._get_data()[index]
           ~~~~~~~~~~~~~~~~^^^^^^^
  File "/home/user/.cache/uv/environments-v2/dap4-xarray-92e2e08d4d47f0ca/lib/python3.13/site-packages/pydap/handlers/dap.py", line 548, in __getitem__
    dataset = UNPACKDAP4DATA(r, self.checksums, self.user_charset).dataset
              ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.cache/uv/environments-v2/dap4-xarray-92e2e08d4d47f0ca/lib/python3.13/site-packages/pydap/handlers/dap.py", line 1002, in __init__
    self.dmr, self.endianness = self.safe_dmr_and_data()
                                ~~~~~~~~~~~~~~~~~~~~~~^^
  File "/home/user/.cache/uv/environments-v2/dap4-xarray-92e2e08d4d47f0ca/lib/python3.13/site-packages/pydap/handlers/dap.py", line 1062, in safe_dmr_and_data
    dmr = self.raw.read(dmr_length).decode(self.user_charset)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 5394: ordinal not in range(128)

Anything else we need to know?

Related?

#4052

My guess

Looking at the traceback, the issue seems to be that pydap expects ASCII encoding while the file might not satisfy this expectation. As a test for this hypothesis, I replaced

    dataset = UNPACKDAP4DATA(r, self.checksums, self.user_charset).dataset

with

    dataset = UNPACKDAP4DATA(r, self.checksums, "UTF-8").dataset

in the pydap code and everything worked just fine!

So I attempted to set the encoding as a keyword arg to load_dataset;

ds = xr.load_dataset(url, engine="pydap", decode_cf=False, decode_times=False, decode_timedelta=False, decode_coords=False, user_charset="UTF-8")

but unfortunately, this gets lost somewhere - might be the pydap code, so not necessarily xarray's fault.

Environment

>>> xr.show_versions()

INSTALLED VERSIONS
------------------
commit: None
python: 3.13.7 (main, Aug 18 2025, 19:20:03) [Clang 20.1.4 ]
python-bits: 64
OS: Linux
OS-release: 6.12.48+deb13-amd64
machine: x86_64
processor: 
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: ('en_US', 'UTF-8')
libhdf5: 1.14.6
libnetcdf: 4.9.3

xarray: 2025.10.2.dev18+ge49cfc4f2
pandas: 2.3.3
numpy: 2.3.4
scipy: 1.16.2
netCDF4: 1.7.3
pydap: 3.5.8
h5netcdf: 1.7.3
h5py: 3.15.1
zarr: 3.1.3
cftime: 1.6.5
nc_time_axis: 1.4.1
iris: None
bottleneck: 1.6.0
dask: 2025.10.0
distributed: 2025.10.0
matplotlib: 3.10.7
cartopy: 0.25.0
seaborn: 0.13.2
numbagg: 0.9.3
fsspec: 2025.9.0
cupy: None
pint: None
sparse: 0.17.0
flox: 0.10.7
numpy_groupies: 0.11.3
setuptools: None
pip: None
conda: None
pytest: None
mypy: None
IPython: None
sphinx: None

[dcherian]

cc @Mikejmnez

[Mikejmnez]

@FObersteiner I took a quick look and this is indeed a bug on the pydap backend. In dap4 the encoding is part of the downloaded response, and it should always be automatically and correctly discovered without input from the user. But older versions (possibly dap2 related) enabled user-defined decoding. This is a quick fix (I got one already as draft PR that works with this example provided), but need to spend a little longer to make it foolproof. Thanks for creating this issue!

[FObersteiner]

@Mikejmnez thanks for jumping on to this so quickly :) I did a few checks and at first I thought your update to pydap does the trick; with

# /// script
# requires-python = ">=3.12"
# dependencies = [
#     "pydap",
#     "xarray",
# ]
#
# [tool.uv.sources]
# pydap = { git = "https://github.com/pydap/pydap.git", rev="d30383a5c9123601722061903d2b6003b4a281e7" }
# xarray = { git = "https://github.com/pydata/xarray.git", branch="main" }
#
# ///

I could load the data from the URL in the OP. However when trying other files from our THREDDS, for instance some satellite data from

url = "dap4://thredds.atmohub.kit.edu/thredds/dap4/musica-iasi/Full_v030301/2021/IASIA_MUSICA_030301_L2_AllTargetProducts_20210101054454_73699.nc"

the error changes to

[...]
  File "/home/user/.cache/uv/environments-v2/dap4-xarray-92e2e08d4d47f0ca/lib/python3.13/site-packages/pydap/handlers/dap.py", line 548, in __getitem__
    dataset = UNPACKDAP4DATA(r, self.checksums, self.user_charset).dataset
              ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.cache/uv/environments-v2/dap4-xarray-92e2e08d4d47f0ca/lib/python3.13/site-packages/pydap/handlers/dap.py", line 1003, in __init__
    self.dataset = self.unpack_dap4_data(dataset)
                   ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/home/user/.cache/uv/environments-v2/dap4-xarray-92e2e08d4d47f0ca/lib/python3.13/site-packages/pydap/handlers/dap.py", line 1078, in unpack_dap4_data
    data = decode_variable(
        buffer,
    ...<3 lines>...
        endian=self.endianness,
    )
  File "/home/user/.cache/uv/environments-v2/dap4-xarray-92e2e08d4d47f0ca/lib/python3.13/site-packages/pydap/handlers/dap.py", line 894, in decode_variable
    data = numpy.array(decode_utf8_string_array(buffer)).astype(dtype.kind)
                       ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
  File "/home/user/.cache/uv/environments-v2/dap4-xarray-92e2e08d4d47f0ca/lib/python3.13/site-packages/pydap/handlers/dap.py", line 918, in decode_utf8_string_array
    decoded_str = str_bytes.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x9f in position 64048: invalid start byte

...while some model data e.g. from

url = "dap4://thredds.atmohub.kit.edu/thredds/dap4/icon/CH3CN/BBE/MACCity/R2B07_0018/ART_BBE_iconR2B07-grid-1960-01-01-00_0018.nc"

loads just fine with the standard versions of xarray and pydap.

So my first impression is that there is a more general issue. It seems we try to decode bytes to characters that aren't characters at all. Is that possible? I'm not that deep into pydap or DAP in general, so can't judge how realistic that is...

[Mikejmnez]

That is interesting. The original error was related to failure to properly decode metadata. This error takes place while decoding the serialized binary data itself, based on where it is taking place. Thanks for this example! I will take a look at this.

[Mikejmnez]

@FObersteiner The more I look at this, I am starting to think that the problem is in the way the TDS dap4 service serializes the time_string variable. I looked at the serialized binary data you get from the Thredds dap4 server, and even though the variable is all contained within a single chunk, the TDS is serializing it as two chunks resulting in string data looking kind of funny at the edge of one of these chunks:

array([b'20210101T055450Z', b'20210101T055450Z', b'20210101T055450Z',
       b'20210101T055450Z', b'20210101T055450Z', b'20210101T055451Z',
       b'20210101T055451Z', b'20210101T055451Z', b'20210101T055451Z',
       b'20210101T055451Z', b'2021010\x10\x00\x00\x00\x00\x00\x00\x002'],
      dtype='|S16')

Could you share the original file? I know the Thredds person, but before I can bring this to his attention, I want to make sure it is a Thredds issue. I'll like to test this with another (interoperable) DAP4 opendap server. Thanks again!

[FObersteiner]

Hi @Mikejmnez that indeed looks weird, like some random bytes from uninitialized/unused memory, which could explain why decoding sometimes works. But even if decoding doesn't fail, the result would likely still be incorrect.

The files are all public on our THREDDS, I have enabled http download there as well; I suppose this will leave the file structure in tact when you download it.

Example from the OP: https://thredds.atmohub.kit.edu/thredds/catalog/iagos-caribic/IAGOS-CARIBIC_MS_files_collection_20250711/CARIBIC-2/catalog.html?dataset=iagos-caribic/IAGOS-CARIBIC_MS_files_collection_20250711/CARIBIC-2/MS_20200304_591_CPT_MUC_10s_V16.nc

The IASI satellite data file: https://thredds.atmohub.kit.edu/thredds/catalog/musica-iasi/Full_v030301/2021/catalog.html?dataset=musica-iasi/Full_v030301/2021/IASIA_MUSICA_030301_L2_AllTargetProducts_20210101054454_73699.nc

And one of the smallest ICON model data files I could find ;-) https://thredds.atmohub.kit.edu/thredds/catalog/icon/CH3CN/BBE/MACCity/R2B07_0018/catalog.html?dataset=icon/CH3CN/BBE/MACCity/R2B07_0018/ART_BBE_iconR2B07-grid-1960-01-01-00_0018.nc

Hope this works for you!