@@ -43,34 +43,44 @@ Crawlera middleware won't be able to handle them.
4343
4444### Settings
4545
46- * ` CRAWLERA_FETCH_ENABLED ` (type ` bool ` , default ` False ` ). Whether or not the middleware will be enabled,
47- i.e. requests should be downloaded using the Crawlera Fetch API. The ` crawlera_fetch_enabled ` spider
48- attribute takes precedence over this setting.
46+ * ` CRAWLERA_FETCH_ENABLED ` (type ` bool ` , default ` False ` )
4947
50- * ` CRAWLERA_FETCH_APIKEY ` (type ` str ` ). API key to be used to authenticate against the Crawlera endpoint
51- (mandatory if enabled)
48+ Whether or not the middleware will be enabled, i.e. requests should be downloaded using
49+ the Crawlera Fetch API. The ` crawlera_fetch_enabled ` spider attribute takes precedence
50+ over this setting.
51+
52+ * ` CRAWLERA_FETCH_APIKEY ` (type ` str ` )
53+
54+ API key to be used to authenticate against the Crawlera endpoint (mandatory if enabled)
55+
56+ * ` CRAWLERA_FETCH_URL ` (Type ` str ` , default ` "http://fetch.crawlera.com:8010/fetch/v2/" ` )
5257
53- * ` CRAWLERA_FETCH_URL ` (Type ` str ` , default ` "http://fetch.crawlera.com:8010/fetch/v2/" ` ).
5458 The endpoint of a specific Crawlera instance
5559
56- * ` CRAWLERA_FETCH_RAISE_ON_ERROR ` (type ` bool ` , default ` True ` ). Whether or not the middleware will
57- raise an exception if an error occurs while downloading or decoding a request. If ` False ` , a
58- warning will be logged and the raw upstream response will be returned upon encountering an error.
60+ * ` CRAWLERA_FETCH_RAISE_ON_ERROR ` (type ` bool ` , default ` True ` )
61+
62+ Whether or not the middleware will raise an exception if an error occurs while downloading
63+ or decoding a response. If ` False ` , a warning will be logged and the raw upstream response
64+ will be returned upon encountering an error.
5965
6066* ` CRAWLERA_FETCH_DOWNLOAD_SLOT_POLICY ` (type ` enum.Enum ` - ` crawlera_fetch.DownloadSlotPolicy ` ,
61- default ` DownloadSlotPolicy.Domain ` ).
67+ default ` DownloadSlotPolicy.Domain ` )
68+
6269 Possible values are ` DownloadSlotPolicy.Domain ` , ` DownloadSlotPolicy.Single ` ,
6370 ` DownloadSlotPolicydefault ` (Scrapy default). If set to ` DownloadSlotPolicy.Domain ` , please
6471 consider setting ` SCHEDULER_PRIORITY_QUEUE="scrapy.pqueues.DownloaderAwarePriorityQueue" ` to
65- make better usage of concurrency options and avoid delays.
72+ make better usage of concurrency options and avoiding delays.
6673
6774* ` CRAWLERA_FETCH_DEFAULT_ARGS ` (type ` dict ` , default ` {} ` )
75+
6876 Default values to be sent to the Crawlera Fetch API. For instance, set to ` {"device": "mobile"} `
6977 to render all requests with a mobile profile.
7078
7179### Spider attributes
7280
73- * ` crawlera_fetch_enabled ` (type ` bool ` , default ` False ` ). Whether or not the middleware will be enabled.
81+ * ` crawlera_fetch_enabled ` (type ` bool ` , default ` False ` )
82+
83+ Whether or not the middleware will be enabled.
7484 Takes precedence over the ` CRAWLERA_FETCH_ENABLED ` setting.
7585
7686### Log formatter
0 commit comments