Skip to content

Commit 387e1ad

Browse files
committed
Don't prevent robots from crawling removed content
Add trailing slashes to all directories.
1 parent 1114fee commit 387e1ad

File tree

1 file changed

+19
-29
lines changed

1 file changed

+19
-29
lines changed

static/robots.txt

Lines changed: 19 additions & 29 deletions
Original file line numberDiff line numberDiff line change
@@ -3,32 +3,22 @@ User-agent: *
33
Disallow: /riakee/
44
Disallow: /riakcs/
55
Disallow: /riakts/
6-
Disallow: /riak/1.0.0
7-
Disallow: /riak/1.1.0
8-
Disallow: /riak/1.1.4
9-
Disallow: /riak/1.2.0
10-
Disallow: /riak/1.2.1
11-
Disallow: /riak/1.3.0
12-
Disallow: /riak/1.3.1
13-
Disallow: /riak/1.3.2
14-
Disallow: /riak/1.4.0
15-
Disallow: /riak/1.4.1
16-
Disallow: /riak/1.4.10
17-
Disallow: /riak/1.4.11
18-
Disallow: /riak/1.4.12
19-
Disallow: /riak/1.4.2
20-
Disallow: /riak/1.4.5
21-
Disallow: /riak/1.4.6
22-
Disallow: /riak/1.4.7
23-
Disallow: /riak/1.4.8
24-
Disallow: /riak/1.4.9
25-
Disallow: /riak/2.0.0
26-
Disallow: /riak/2.0.1
27-
Disallow: /riak/2.0.2
28-
Disallow: /riak/2.0.3
29-
Disallow: /riak/2.0.4
30-
Disallow: /riak/2.0.5
31-
Disallow: /riak/2.0.6
32-
Disallow: /riak/2.1.1
33-
Disallow: /riak/2.1.3
34-
Disallow: /riak/2.1.4
6+
Disallow: /riak/1.0.0/
7+
Disallow: /riak/1.1.0/
8+
Disallow: /riak/1.1.4/
9+
Disallow: /riak/1.2.0/
10+
Disallow: /riak/1.2.1/
11+
Disallow: /riak/1.3.0/
12+
Disallow: /riak/1.3.1/
13+
Disallow: /riak/1.3.2/
14+
Disallow: /riak/1.4.0/
15+
Disallow: /riak/1.4.1/
16+
Disallow: /riak/1.4.10/
17+
Disallow: /riak/1.4.11/
18+
Disallow: /riak/1.4.12/
19+
Disallow: /riak/1.4.2/
20+
Disallow: /riak/1.4.5/
21+
Disallow: /riak/1.4.6/
22+
Disallow: /riak/1.4.7/
23+
Disallow: /riak/1.4.8/
24+
Disallow: /riak/1.4.9/

0 commit comments

Comments
 (0)