[Feature]: Add a way to block all cookies and site data in headless mode puppeteer/chromium

[PapiJalopy]

Proposed solution

I am consistently able to avoid getting 403ed on newegg as long as I block all cookies and delete site data on chromium when i first start running it.

This is a theory so take it with a grain of salt, but I've noticed that newegg allows you to visit their website one time with any IP, and as long as you don't store and block all cookies it seem to think it is always your first visit and wont give you a captcha or 403 you on your next scrape/refresh.

The issue is that it requires headless=false and settings are not persistent. If you can add or inform me of a way to modify chromium user settings to make them persistent and run in headless mode that would be perfect.

Objective

N/A

Goals

N/A

Non-goals

N/A

Anti-goals

N/A

[andrewmackrodt]

You can try setting INCOGNITO=true so that a new browser is used for each store lookup, it shouldn't persist any cookies this way.

[PapiJalopy]

You can try setting INCOGNITO=true so that a new browser is used for each store lookup, it shouldn't persist any cookies this way.

INCOGNITO=true does not work unfortunately because it only blocks third party cookies. If all cookies are not blocked the page will load but streetmerchant gives a timeout error for every listing.

When enabling "Block all cookies" the first listing might time out but from there its smooth sailing.

I've able to run it on already "banned/403'ed" IP's for days now.

If possible, I would actually suggest modifying the INCOGNITO=* option so that chromium defaults to "Block all cookies" when enabled.

[PapiJalopy]

Is there an argument or a function we can add to puppeteer to do this? There has to be some kind of call to set UI settings via API.

[EbonyEquivocal]

I get a ERR_HTTP2_PROTOCOL_ERROR when checking Nvidia store stock. I think that this would be resolved if I could clear all cookies/cache. I have tried incognito mode, and no luck.

[PapiJalopy]

I've been working on crudely adding persistence to settings for chromium puppeteer to allow for example, cookies and site data to stay selected to your preferred preference. i've found some puppeteer arguments that allow creating a profile location and disabling dev profile so that setting are not wiped when script is terminated or restarted.

In src/index.ts - add the following arg to disable dev profile -
--disable-dev-profile
and to select a user profile location -
--user-data-dir=

(Example)

export async function launchBrowser(): Promise<Browser> {
  const args: string[] = [];

  // Skip Chromium Linux Sandbox
  // https://github.com/puppeteer/puppeteer/blob/main/docs/troubleshooting.md#setting-up-chrome-linux-sandbox
  if (config.browser.isTrusted) {
    args.push('--no-sandbox');
    args.push('--disable-setuid-sandbox');
    args.push('--user-data-dir=/home/user/streetmerchant/profilename');
    args.push('--disable-dev-profile');
  }

This disables the dev profile and sets a custom user profile directory to save user data to allowing for further customization of the browser user environment. For example, logging into websites and staying logged in by storing cookies, or disabling all cookies outright and keeping it disabled allowing to then run in headless with cookies disabled.

This has not fixed all my issues but its a start.

It would be nice to make this function selectable using the dotenv file. But im not a 1337 haxor coder and only know my way around a little bit so idk how to do that unless I do more research which takes a bit of time to learn. @jef

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 10 days

[github-actions]

This issue has been closed because it is stale. Reopen if necessary.