Network data collector is asynchronous?

[nvborisenko]

I am experimenting with network data collector.

Simple code (C#/Selenium):

await using var collector = await bidi.Network.AddDataCollectorAsync([DataType.Response], 10000000);

await using var subscription = await bidi.Network.OnResponseCompletedAsync(async res =>
{
    //await Task.Delay(300); // give a chance for remote-end to materialize data

    Console.WriteLine((await bidi.Network.GetDataAsync(DataType.Response, res.Request.Request)).GetType());
});

await context.NavigateAsync("https://selenium.dev", new() { Wait = ReadinessState.Complete });

Here I do:

1. Enable data collector for responses
2. Subscribe on responseCompleted event, and get data for the response as soon as event is raised

Sometimes, I get weird error from remote-end:

no such network data: No collected data for request ED804ADE49D4B69A470EDB66E5C0BE37

But, as soon as I add small delay (300ms) before getting response data, it works smoothly.

PS: this is for Chrome. Firefox works worse: unavailable network data: Network data content for request id 2 and DataType response is unavailable - raising more often.

[whimboo]

@juliandescottes could you please take a look?

[OrKoN]

@sadym-chromium i think it is related to the issue of responseCompleted emitted too early. We should fix it in the way we discussed.

[sadym-chromium]

Related issue: GoogleChromeLabs/chromium-bidi#3574

[juliandescottes]

@nvborisenko can you attach trace logs for Firefox as well as details about the version of Firefox you are using?
It would be great to file bugs against the implementations rather than spec issues though.

[juliandescottes]

In theory you should be able to call getData as early as after network.responseStarted.
A message such as unavailable network data: Network data content for request id 2 and DataType response is unavailable is emitted from https://searchfox.org/firefox-main/rev/6f3037f71baa242161ed343cdccb12c1b8e50227/remote/webdriver-bidi/modules/root/network.sys.mjs#1242-1246 so it could mean that the response was evicted because the buffer is full?

(in case this is a buffer issue, you could bump the int preference remote.network.maxTotalDataSize to a difference value - it's 200MB by default).

[nvborisenko]

Here is trace logs, FF 143.0.1:

geckodriver.log

It would be great to file bugs against the implementations rather than spec issues though.

I posted the issue (question?) here because I observed the common behavior for different browsers.

[juliandescottes]

Here is trace logs, FF 143.0.1:

geckodriver.log

Thanks for the logs.

I can see the errors coming from 2 reasons, actually similar to the issues you had with network interception.

• resources in memory cache (eg css or images). In general when using data collection or network interception, disabling the cache via network.setCacheBehavior leads to more consistent results. Not sure if Chrome can handle those, but at least at the moment it's technically not feasible on Firefox
• data URLs. We don't support getting response data for them at the moment, but I should file a bug for that, it shouldn't be that difficult to handle.

I will file bugs to: support support data url, support cached responses (but that one will be blocked for now) and to improve logging (so that we get a clearer reason in nosuchdata or unavailabledata errors)

It would be great to file bugs against the implementations rather than spec issues though.

I posted the issue (question?) here because I observed the common behavior for different browsers.

Right, and thanks for testing and reporting, it's really useful. The issues on Chrome and Firefox are most likely separate and unrelated to the spec, but I definitely appreciate the feedback.

[juliandescottes]

I filed the following bugs to improve the situation on Firefox:

• improve logs and error messages https://bugzilla.mozilla.org/show_bug.cgi?id=1992216
• always throw correct error when data missing for technical reasons https://bugzilla.mozilla.org/show_bug.cgi?id=1992214
• support cached css/images/etc. https://bugzilla.mozilla.org/show_bug.cgi?id=1992213
• support data scheme https://bugzilla.mozilla.org/show_bug.cgi?id=1992210

I don't think we can completely solve your scenario in the near future, I expect cached css/images to be a problem for now - I still suggest to disable caching for more consistent results. But at least it should make the output clearer, so that you can understand why a getData call failed.

[nvborisenko]

Honestly I am expecting to see DevTools data both for Chromium/FF, as I see it in Browser UI.

Setting network.setCacheBehavior is not a solution for all us - it changes browser behavior, what we want to avoid.

In Network module if we can implement HAR listener on client side, it would be good enough.

[juliandescottes]

Honestly I am expecting to see DevTools data both for Chromium/FF, as I see it in Browser UI.

Setting network.setCacheBehavior is not a solution for all us - it changes browser behavior, what we want to avoid.

In Network module if we can implement HAR listener on client side, it would be good enough.

Understood. but DevTools is not necessarily very accurate / correct. For cached stylesheets, they are not visible in Firefox DevTools (because of the exact same issue). We do show an image preview for cached images, but it has nothing to do with the data which actually came from the network, it uses the rendered bitmap on the page. So it could be extremely large depending on how the image is used.

[juliandescottes]

FYI: we already have issues filed here to discuss about cached responses and data scheme.