You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/infrastructure/configuration/data-retention/get-full-history-data-with-clio.md
+20-15Lines changed: 20 additions & 15 deletions
Original file line number
Diff line number
Diff line change
@@ -5,17 +5,15 @@ labels:
5
5
- Clio Server
6
6
- Data Retention
7
7
---
8
-
# Getting Full History Data with Clio
8
+
# Get Full History Data with Clio
9
9
10
10
This page describes how to access full XRP Ledger history data with the [Clio](../../../concepts/networks-and-servers/the-clio-server.md) server.
11
11
12
12
Clio stores the full XRP Ledger history in a publicly accessible [AWS S3](https://aws.amazon.com/s3/) bucket. This data enables Clio to serve requests that require complete historical context. Ledger data is uploaded automatically to the S3 bucket on a daily basis to ensure you always have access to the latest full history up to the current day.
13
13
14
14
## Prerequisites
15
15
16
-
To proceed with this tutorial you need:
17
-
18
-
- A Clio server installed. To learn more about how to build and compile Clio, see [Install Clio on Ubuntu Linux](../../installation/install-clio-on-ubuntu.md).
16
+
- Ensure you have the Clio server installed. To learn more about how to build and compile Clio, see [Install Clio on Ubuntu Linux](../../installation/install-clio-on-ubuntu.md).
19
17
{% admonition type="info" name="Note" %}
20
18
You need to add `-o snapshot=True` at the `conan install` build step to build the Clio server with the full history feature.
21
19
{% /admonition %}
@@ -24,20 +22,26 @@ To proceed with this tutorial you need:
24
22
25
23
## Download Full History Data from S3
26
24
27
-
To download the full XRP Ledger history data on your machine, run the following command in a terminal:
25
+
Before you can run the Clio server, you need to download the full XRP Ledger history data on your machine or environment. In a directory of your choosing, run the following command in a terminal:
The command uses the `--no-sign-request` flag since the bucket is **public**, so you don't need AWS credentials to access it. The data is stored in the [EU-West (eu-west-2)](https://docs.aws.amazon.com/global-infrastructure/latest/regions/aws-regions.html) region, and due to its large size, the sync process can be slow.
30
32
31
-
The command uses the `--no-sign-request` flag since the bucket is public, so you don't need AWS credentials to access it. The data is stored in the [EU-West (eu-west-2)](https://docs.aws.amazon.com/global-infrastructure/latest/regions/aws-regions.html) region, and due to its large size, the sync process can be slow.
33
+
To speed up the transfer, you can increase the number of concurrent AWS requests by setting the [`AWS_MAX_CONCURRENT_REQUESTS`](https://awscli.amazonaws.com/v2/documentation/api/latest/topic/s3-config.html#max-concurrent-requests) environment variable to an appropriate value for you system. For example:
32
34
33
-
{% admonition type="success" name="Tip" %}
34
-
To speed up the transfer, consider increasing the number of concurrent requests by setting `export AWS_MAX_CONCURRENT_REQUESTS=64`, or a request value that is appropriate for you.
35
-
{% /admonition %}
35
+
```sh
36
+
export AWS_MAX_CONCURRENT_REQUESTS=64
37
+
```
36
38
37
-
Once the sync completes, your local directory should contain a structure similar to the example shown below:
39
+
Once the sync completes, your directory should contain a structure similar to the example shown below:
38
40
39
41
40
42
43
+
Take note of the directory where you downloaded the ledger data, as you will need it for the next steps.
44
+
41
45
## Start Clio with Full History
42
46
43
47
To start the Clio server with full history:
@@ -50,9 +54,8 @@ To start the Clio server with full history:
50
54
--path=<path_to_full_history_folder>
51
55
```
52
56
53
-
This command starts a lightweight server that exposes the snapshot data over gRPC and WebSocket, and acts as a data sourceforClio when it's runningin full history mode.
54
-
2. Start the Clio server, along with the database backend (e.g., ScyllaDB or Cassandra).
55
-
3. Make sure to adjust your Clio configuration file to properly connect to the snapshot server, starting with the `etl_sources` field, which should match `grpc_server` address that you used in**step 1**:
57
+
This command starts a lightweight [Go](https://github.com/XRPLF/clio/tree/develop/tools/snapshot) server that exposes the snapshot data over gRPC and WebSocket, and acts as a data sourceforClio when it is runningin full history mode.
58
+
2. Make sure to adjust your Clio configuration file to properly connect to the snapshot server, starting with the `etl_sources` field, which should match `grpc_server` address that you used in**step 1**:
56
59
57
60
```json
58
61
"etl_sources": {
@@ -61,7 +64,7 @@ To start the Clio server with full history:
61
64
"grpc_port": "50052"
62
65
}
63
66
```
64
-
4. Set the `start_sequeunce` and `end_sequence` to define the ledger range that you want. For example:
67
+
3. Set the `start_sequeunce` and `end_sequence` to define the ledger range that you want. For example:
65
68
66
69
```json
67
70
{
@@ -75,3 +78,5 @@ To start the Clio server with full history:
75
78
{% admonition type="info" name="Note" %}
76
79
You can find the actual start and end sequence in the `manifest.txt` file located in the root of your synced `Full-History` directory. It is formatted as `start_sequence | end_sequence`.
77
80
{% /admonition %}
81
+
4. Start the Clio server, along with the relevant database backend (e.g., ScyllaDB or Cassandra).
0 commit comments