Data value return inconsistent

[martinseul]

when requesting datavalues form HIS nd dendra wof the resulting dataset is not identical. HIS-WOF and dendra-wof interpret the begin and enddate differently. For HIS if a day is specified the return start with datavalues beginning at 00:00 while dendra wof the return start 00:00 + timezone? so in this case +7h so the resulting set is shifted. this needs to be aligned. I tested it for czo_ariz:Tower_Vcp,122 and czo_ariz:Barometricpressure10 for the dates 2008/01/01 to 2008/01/07(left HIS right Dendra)

[jscottsf]

Think I found the issue. The station “Valles Caldera Ponderosa Pine Flux Tower” has its time zone set to MST (UTC offset -25200).

When the input parameters are passed in as local time (WOF -> API) the time is shifted to UTC to perform the query. It uses the offset on the station.

The data loaded has the wrong time offset. It should not be 0; however, that offset is being used to compute the local time in the WOF service.

In short, we have different sources of truth for the offset, and offset in the data is wrong.

---

Per usual, each system is respectively doing everything correct in its own domain.

Evidence of this discrepancy can be seen on a datapoint.

Below you see the local time lt as computed by Denda based on the configured offset o -25200. We loaded and carried through the original datavalues offset d.UTCOffset. The configured offset in the metadata does not match what was loaded.

    {
      "lt": "2008-01-01T00:00:00.000",
      "o": -25200,
      "v": 76.596,
      "d": {
        "CensorCode": "nc",
        "UTCOffset": 0
      },
      "q": {
        "annotation_ids": [
          "642b77ab34190283e0718167",
          "642b77ac04100f3f025d2c39",
          "642b77ad57e9ce64a370da25",
          "6436f3e18c9fe1b07ca787cc"
        ],
        "attrib": {
          "height": {
            "value": 2,
            "unit_tag": "dt_Unit_Meter"
          }
        },
        "flag": [
          "his.odm.offsettypes.OffsetTypeID:1",
          "his.odm.offsettypes.OffsetDescription:Height of sensor above ground",
          "his.odm.offsettypes.OffsetUnitsID:52",
          "his.odm.offsettypes.OffsetTypeID:1",
          "his.odm.offsettypes.OffsetDescription:Height of sensor above ground",
          "his.odm.offsettypes.OffsetUnitsID:52",
          "his.odm.offsettypes.OffsetTypeID:1",
          "his.odm.offsettypes.OffsetDescription:Height of sensor above ground",
          "his.odm.offsettypes.OffsetUnitsID:52"
        ]
      }
    }, 

This mismatch will occur naturally for daylight savings time shifts since our o is a standard time offset.

For this case, the offset was loaded incorrectly and it’s causing a mismatch in selection and what’s returned since WOF uses the loaded offset UTCOffset to compute the local time.

One solution is to just have a single source of truth, the standard time as we configure it on Dendra. It means there will be no duplicate local times in the data since DST will not be supported.
Another solution is to fix the data.

This is challenging since the data is loaded as UTC and yet we’re trying to query using local time. The offset on the station, which is used to write the query, does not agree with the offset on the row.

In Dendra we decided not to use an offset on the row since it can cause these issues. We let the metadata do the work.

Here is the station. https://ui.dendradev-gke1.dendra.cuahsi.org/orgs/czo-arizona/stations/63e6a1b63cdc613d4c5e135d

[cbode]

I am adding to datastreams an external_reference, his.wof.query.UTCOffset which will be set to the DataValues UTCOffset if it does not match the stations's offset. Note: offset will be in seconds not hours.

{
            "identifier": "-25200",
            "type": "his.wof.query.UTCOffset"
}

[ergjustin]

As per the above issue
location: Tower_Vcp
variable: Barometricpressure10
startDate- 2008-01-01T00:00:00
endDate- 2008-01-07T00:00:00

external_reference, his.wof.query.UTCOffset that should be view in https://ui.dendradev-gke1.dendra.cuahsi.org/orgs/czo-arizona/datastreams/63e6a1c9e92751757317b58d , but I could not able to find it in external_reference

[cbode]

I am delayed in deploying the above fix due to another issue with annotations.