dist.ipfs.io unreliable or blocking automated requests

[aphelionz]

npm install is failing fairly regularly now, hanging at:

> go-ipfs-dep@0.4.19 install /home/mark/Projects/orbitdb/orbit-db/node_modules/go-ipfs-dep
> node src/bin.js

I believe three's some automated protection on dist.ipfs.io now that's blocking these requests and causing 504 timeouts. You can emulate it by simply calling wget over and over again until it freezes.

Reproduced on multiple devices and internet connection situations.

[Stebalien]

This is a reliability issue with our gateway that we're currently debugging. (not the normal issues, this appears to be a regression)

[aphelionz]

Thank you!

[gnidan]

Just adding a +1... this has been causing huge problems for github.com/trufflesuite/truffle's development process.

[0xGabi]

+1 Same on our side with aragon/aragon-cli

[ottodevs]

this is unreliable also for low speed connections:

(After 1 hour waiting for the download to complete)

This is one of many random errors when trying to download the module from dist.ipfs.io

[aphelionz]

For those who want a (hopefully) temporary workaround, you can try my fork here: https://github.com/aphelionz/npm-go-ipfs-dep/ which downloads from GitHub releases.

I know it's not what we want to achieve in the end, but at least this can unblock some people like it unblocked me. Maybe we can use the centralized solution in GH as a fallback or something, I dunno.

[aphelionz]

Another option I thought of, but didn't attempt, was to package the CIDs of the releases inside the repo itself and then use ipfs.get to grab them. I tried it with one of the .cid files from GitHub releases and it seemed to work since it was downloading from the swarm instead of from the HTTP gateway.

[lidel]

Proposal to support error recovery via a customizable list of gateways: #31