Skip to content

Commit e04bf8a

Browse files
committed
Use Googlebot as User-Agent
1 parent b42ba43 commit e04bf8a

File tree

1 file changed

+6
-10
lines changed

1 file changed

+6
-10
lines changed

public/robots.txt

Lines changed: 6 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1,18 +1,14 @@
11
# See https://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file
22

3-
# Disallow everything by default
4-
User-Agent: *
5-
Disallow: /
6-
7-
# Allow Google crawler
8-
User-Agent: Googlebot
3+
User-Agent: GoogleBot
94
Allow: /
105

11-
# Allow DuckDuck crawler
126
User-Agent: DuckDuckBot
137
Allow: /
148

15-
# Allow Bing crawler
16-
User-Agent: Bingbot
17-
Allow: /
9+
User-Agent: bingbot
1810
Crawl-delay: 5
11+
Allow: /
12+
13+
User-Agent: *
14+
Disallow: /

0 commit comments

Comments
 (0)