Skip to content
This repository was archived by the owner on Dec 28, 2023. It is now read-only.

Commit 1492753

Browse files
committed
Readme format changes
1 parent ee88226 commit 1492753

File tree

1 file changed

+22
-12
lines changed

1 file changed

+22
-12
lines changed

README.md

Lines changed: 22 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -43,34 +43,44 @@ Crawlera middleware won't be able to handle them.
4343

4444
### Settings
4545

46-
* `CRAWLERA_FETCH_ENABLED` (type `bool`, default `False`). Whether or not the middleware will be enabled,
47-
i.e. requests should be downloaded using the Crawlera Fetch API. The `crawlera_fetch_enabled` spider
48-
attribute takes precedence over this setting.
46+
* `CRAWLERA_FETCH_ENABLED` (type `bool`, default `False`)
4947

50-
* `CRAWLERA_FETCH_APIKEY` (type `str`). API key to be used to authenticate against the Crawlera endpoint
51-
(mandatory if enabled)
48+
Whether or not the middleware will be enabled, i.e. requests should be downloaded using
49+
the Crawlera Fetch API. The `crawlera_fetch_enabled` spider attribute takes precedence
50+
over this setting.
51+
52+
* `CRAWLERA_FETCH_APIKEY` (type `str`)
53+
54+
API key to be used to authenticate against the Crawlera endpoint (mandatory if enabled)
55+
56+
* `CRAWLERA_FETCH_URL` (Type `str`, default `"http://fetch.crawlera.com:8010/fetch/v2/"`)
5257

53-
* `CRAWLERA_FETCH_URL` (Type `str`, default `"http://fetch.crawlera.com:8010/fetch/v2/"`).
5458
The endpoint of a specific Crawlera instance
5559

56-
* `CRAWLERA_FETCH_RAISE_ON_ERROR` (type `bool`, default `True`). Whether or not the middleware will
57-
raise an exception if an error occurs while downloading or decoding a request. If `False`, a
58-
warning will be logged and the raw upstream response will be returned upon encountering an error.
60+
* `CRAWLERA_FETCH_RAISE_ON_ERROR` (type `bool`, default `True`)
61+
62+
Whether or not the middleware will raise an exception if an error occurs while downloading
63+
or decoding a response. If `False`, a warning will be logged and the raw upstream response
64+
will be returned upon encountering an error.
5965

6066
* `CRAWLERA_FETCH_DOWNLOAD_SLOT_POLICY` (type `enum.Enum` - `crawlera_fetch.DownloadSlotPolicy`,
61-
default `DownloadSlotPolicy.Domain`).
67+
default `DownloadSlotPolicy.Domain`)
68+
6269
Possible values are `DownloadSlotPolicy.Domain`, `DownloadSlotPolicy.Single`,
6370
`DownloadSlotPolicydefault` (Scrapy default). If set to `DownloadSlotPolicy.Domain`, please
6471
consider setting `SCHEDULER_PRIORITY_QUEUE="scrapy.pqueues.DownloaderAwarePriorityQueue"` to
65-
make better usage of concurrency options and avoid delays.
72+
make better usage of concurrency options and avoiding delays.
6673

6774
* `CRAWLERA_FETCH_DEFAULT_ARGS` (type `dict`, default `{}`)
75+
6876
Default values to be sent to the Crawlera Fetch API. For instance, set to `{"device": "mobile"}`
6977
to render all requests with a mobile profile.
7078

7179
### Spider attributes
7280

73-
* `crawlera_fetch_enabled` (type `bool`, default `False`). Whether or not the middleware will be enabled.
81+
* `crawlera_fetch_enabled` (type `bool`, default `False`)
82+
83+
Whether or not the middleware will be enabled.
7484
Takes precedence over the `CRAWLERA_FETCH_ENABLED` setting.
7585

7686
### Log formatter

0 commit comments

Comments
 (0)