bug: path-gateway requiring ipld data model conversions (PB/JSON/CBOR)

[SgtPooki]

cc @lidel

This CID references some text: I am a txt file on path with utf8

Why are we trying to confirm that the content-type is dag-json? shouldn't this be failing...

more details in ipfs/helia-verified-fetch#35

Gateway conformance output:

[gateway-conformance] �=== RUN   TestDagPbConversion/GET_UnixFS_file_as_DAG-JSON_with_format=dag-json_converts_to_the_expected_Content-Type/Status_code
[gateway-conformance] �--- PASS: TestDagPbConversion/GET_UnixFS_file_as_DAG-JSON_with_format=dag-json_converts_to_the_expected_Content-Type/Status_code (0.00s)
[gateway-conformance] �=== NAME  TestDagPbConversion/GET_UnixFS_file_as_DAG-JSON_with_format=dag-json_converts_to_the_expected_Content-Type
[gateway-conformance] �=== RUN   TestDagPbConversion/GET_UnixFS_file_as_DAG-JSON_with_format=dag-json_converts_to_the_expected_Content-Type/Header_Content-Type
[gateway-conformance] 
[gateway-conformance]     run.go:53: 
[gateway-conformance]         Name: GET UnixFS file as DAG-JSON with format=dag-json converts to the expected Content-Type
[gateway-conformance]         Hint: 
[gateway-conformance]         
[gateway-conformance]         Error: Header 'Content-Type' expected 'application/vnd.ipld.dag-json', got 'text/plain; charset=utf-8'
[gateway-conformance]         
[gateway-conformance]         Expected Request:
[gateway-conformance]         {
[gateway-conformance]           "method": "GET",
[gateway-conformance]           "path": "/ipfs/bafkreialihlqnf5uwo4byh4n3cmwlntwqzxxs2fg5vanqdi3d7tb2l5xkm",
[gateway-conformance]           "query": {
[gateway-conformance]             "format": [
[gateway-conformance]               "dag-json"
[gateway-conformance]             ]
[gateway-conformance]           }
[gateway-conformance]         }
[gateway-conformance]         
[gateway-conformance]         Actual Request:
[gateway-conformance]         GET /ipfs/bafkreialihlqnf5uwo4byh4n3cmwlntwqzxxs2fg5vanqdi3d7tb2l5xkm?format=dag-json HTTP/1.1
[gateway-conformance]         Host: host.docker.internal:8080
[gateway-conformance]         User-Agent: Go-http-client/1.1
[gateway-conformance]         Accept-Encoding: gzip
[gateway-conformance]         
[gateway-conformance]         
[gateway-conformance]         
[gateway-conformance]         Expected Response:
[gateway-conformance]         {
[gateway-conformance]           "statusCode": 200,
[gateway-conformance]           "headers": [
[gateway-conformance]             {
[gateway-conformance]               "key": "Content-Type",
[gateway-conformance]               "check": {}
[gateway-conformance]             },
[gateway-conformance]             {
[gateway-conformance]               "key": "Content-Disposition",
[gateway-conformance]               "check": {}
[gateway-conformance]             },
[gateway-conformance]             {
[gateway-conformance]               "key": "Content-Type",
[gateway-conformance]               "check": {},
[gateway-conformance]               "not": true
[gateway-conformance]             }
[gateway-conformance]           ],
[gateway-conformance]           "body": "eyIvIjp7ImJ5dGVzIjoiU1NCaGJTQmhJSFI0ZENCbWFXeGxJRzl1SUhCaGRHZ2dkMmwwYUNCMWRHWTRDZyJ9fQ=="
[gateway-conformance]         }
[gateway-conformance]         
[gateway-conformance]         Actual Response:
[gateway-conformance]         HTTP/1.1 200 OK
[gateway-conformance]         Content-Length: 31
[gateway-conformance]         Access-Control-Allow-Headers: Content-Type, Range, User-Agent, X-Requested-With
[gateway-conformance]         Access-Control-Allow-Methods: GET, HEAD, OPTIONS
[gateway-conformance]         Access-Control-Allow-Origin: *
[gateway-conformance]         Access-Control-Expose-Headers: Content-Range, Content-Length, X-Ipfs-Path, X-Ipfs-Roots, X-Chunked-Output, X-Stream-Output
[gateway-conformance]         Connection: keep-alive
[gateway-conformance]         Content-Type: text/plain; charset=utf-8
[gateway-conformance]         Date: Fri, 29 Mar 2024 23:56:49 GMT
[gateway-conformance]         Keep-Alive: timeout=72

[lidel]

Apologies for missing notification on this one.

This looks like a bug, but is in fact an old "feature" of IPLD :)
We added this test to ensure this IPLD "magic" does not get silently broken.

In good old PL times, IPLD Data Model (DM) was supposed to cover everything, and everything that follow Data Model was going to be mappable to meaningful DAG-JSON/CBOR. Because UnixFS and dag-pb were created before IPLD DM, a "Logical Format" was invented to allow representing UnixFS in IPLD DM terms.

The intention was, to be able to read files and directory listing UnixFS details as "useful JSON/CBOR".

You can test on CLI:

ipfs dag get --output-codec dag-json $UNIXFS_CID > logical_format.json

Or dweb.link gateway: https://cid-ipfs-tech.ipns.dweb.link/?format=dag-json

The test you mention is effectively the same as the above command: represent an UnixFS directory or file (chunked into dag-pb blocks) as "Logical Format" map and then turn that Data Model map into DAG-JSON.

Note: PB<>JSON, unlike CBOR<>JSON, is a lossy mapping (it does not preserve any extra PB fields like UnixFS 1.5's mtime/mode etc)

Technically, there is no bug, IPLD Data Model is mentioned in https://specs.ipfs.tech/http-gateways/path-gateway/#accept-request-header:

[...] requests IPLD Data Model representation serialized into DAG-JSON format.

In reality, TestDagPbConversion (PB<>JSON) on HTTP gateway is a niche feature, probably the bottom of priority queue.
The entire PathGatewayDAG is a set of JSON<>CBOR conversion tests, an echo of the IPLD that never came to be.
Unsure if anyone would miss it.

At this point in time, I would be fine with IPIP that removes IPLD Data Model mentions, and the implied "IPLD Data Model Conversions" PB<>JSON<>CBOR from https://specs.ipfs.tech/http-gateways/path-gateway/ and same with tests, and only support reading original format (PB,CBOR,JSON). Would simplify new implementations.

@SgtPooki are we supporting IPLD conversions on inbrowser.link, skip, or wip?
https://cid-ipfs-tech.ipns.inbrowser.link/?format=dag-json seems to return some different payload than https://cid-ipfs-tech.ipns.dweb.link/?format=dag-json

Is it closer to being done (easy fix), or should we look into removing conversions from specs/tests?

[SgtPooki]

@lidel I looked into this a little bit, and it looks like @helia/dag-json and @ipld/dag-json don't support converting this content, we could make it happen inside verified-fetch, or push it down further into @helia/dag-json, but it would require a little bit of extra work.

if it's not something that would be missed we can delay on supporting it, and I can add TestDagPbConversion to skipped tests in verified-fetch

[lidel]

+1, skipping for now sgtm. Let's see if anyone requests it, and discuss then (like we did with ?format=car).
If we don't see any signal from users in next 6-12 months, after inbrowser.link migration, we could consider removal from spec.

[achingbrain]

it looks like @helia/dag-json and @ipld/dag-json don't support converting this content

I just want to clarify this comment. What they don't support is deserializing arbitrary Uint8Arrays.

That is, if you start with Uint8Array(5) [104, 101, 108, 108, 111], the decode method of @ipld/dag-json will throw, because the bytes interpreted as a string is hello, which is not valid JSON.

Uint8Array(7) [34, 104, 101, 108, 108, 111, 34] on the other hand, is "hello" which is valid JSON and so decodes without throwing.

The expectation here appears to be to start with a serialized protobuf message or some bytes with no framing (e.g. raw) and for @ipld/dag-json to be able to do something with that data, which it (probably) cannot, since it only operates on valid JSON.

Instead you need to deserialize the original buffer to convert it to an object, then @ipld/dag-json can serialize it as valid JSON and it can be sent in the HTTP response to the user.

E.g.

import * as dagCbor from '@ipld/dag-cbor'
import * as dagJson from '@ipld/dag-json'
import * as pb from '@ipld/dag-pb'
import * as json from '@ipld/json'
import { createHelia } from 'helia'
import { CID } from 'multiformats/cid'

const CODEC_DAG_PB = 0x70
const CODEC_DAG_CBOR = 0x71
const CODEC_DAG_JSON = 0x0129
const CODEC_JSON = 0x0200
const CODEC_RAW = 0x55

const cid = CID.parse('ba...')

const helia = await createHelia()

const block = await helia.blockstore.get(cid, {
  signal: AbortSignal.timeout(5_000)
})

let obj

if (cid.codec === CODEC_DAG_PB) {
  obj = pb.decode(block)
} else if (cid.codec === CODEC_DAG_CBOR) {
  obj = dagCbor.decode(block)
} else if (cid.codec === CODEC_DAG_JSON) {
  obj = dagJson.decode(block)
} else if (cid.codec === CODEC_JSON) {
  obj = json.decode(block)
} else if (cid.codec === CODEC_RAW) {
  obj = block
} else {
  throw new Error('Unsupported codec')
}

const out = dagJson.encode(obj)

console.info(new TextDecoder().decode(out))
// ...prints some JSON

Note: PB<>JSON, unlike CBOR<>JSON, is a lossy mapping (it does not preserve any extra PB fields like UnixFS 1.5's mtime/mode etc)

I may be missing something here but mtime/mode is encoded as part of the UnixfsV1 Data message (.proto) which is stored in the similarly named Data field of the root PBNode struct.

When converting an object deserialized from a DAG-PB protobuf message to DAG-JSON any Uint8Array fields are serialized as { "/": { "bytes": ... } } so mtime/mode would be preserved:

{
  "Data": { "/": { "bytes": "...mtime.mode.in.here.somewhere...==" } },
  "Links": []
}

The conversion would only be lossy if fields were added to the DAG-PB .proto definition and they were not added to the deserialized output. This has yet to happen though.

If the user or the user's agent downloads the DAG-JSON representation of the UnixFS DAG-PB message, and doesn't apply the mtime/mode, then 🤷‍♂