|
| 1 | +--- |
| 2 | +seo: |
| 3 | + description: Configure how far back your server should store transaction history. |
| 4 | +labels: |
| 5 | + - Clio Server |
| 6 | + - Data Retention |
| 7 | +--- |
| 8 | +# Get Full History Data with Clio |
| 9 | + |
| 10 | +This page describes how to access full XRP Ledger history data with the [Clio](../../../concepts/networks-and-servers/the-clio-server.md) server. |
| 11 | + |
| 12 | +Clio stores the full XRP Ledger history in a publicly accessible [AWS S3](https://aws.amazon.com/s3/) bucket. This data enables Clio to serve requests that require complete historical context. Ledger data is uploaded automatically to the S3 bucket on a daily basis to ensure you always have access to the latest full history up to the current day. |
| 13 | + |
| 14 | +## Prerequisites |
| 15 | + |
| 16 | +- Ensure you have the Clio server installed. To learn more about how to build and compile Clio, see [Install Clio on Ubuntu Linux](../../installation/install-clio-on-ubuntu.md). |
| 17 | + {% admonition type="info" name="Note" %} |
| 18 | + You need to add `-o snapshot=True` at the `conan install` build step to build the Clio server with the full history feature. |
| 19 | + {% /admonition %} |
| 20 | + |
| 21 | +- If you haven't already, install the [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) on your machine. |
| 22 | + |
| 23 | +## Download Full History Data from S3 |
| 24 | + |
| 25 | +Before you can run the Clio server, you need to download the full XRP Ledger history data on your machine or environment. From your terminal, run the following command in a suitable directory: |
| 26 | + |
| 27 | +```sh |
| 28 | +aws s3 sync s3://full-history-ledger-data/Full-History . --no-sign-request |
| 29 | +``` |
| 30 | + |
| 31 | +The command uses the `--no-sign-request` flag since the bucket is **public**, so you don't need AWS credentials to access it. The data is stored in the [EU-West (eu-west-2)](https://docs.aws.amazon.com/global-infrastructure/latest/regions/aws-regions.html) region, and due to its large size, the sync process can be slow. |
| 32 | + |
| 33 | +To speed up the transfer, you can increase the number of concurrent AWS requests by setting the [`AWS_MAX_CONCURRENT_REQUESTS`](https://awscli.amazonaws.com/v2/documentation/api/latest/topic/s3-config.html#max-concurrent-requests) environment variable to an appropriate value for you system. For example: |
| 34 | + |
| 35 | +```sh |
| 36 | +export AWS_MAX_CONCURRENT_REQUESTS=64 |
| 37 | +``` |
| 38 | + |
| 39 | +Once the sync completes, your directory should contain a structure similar to the example shown below: |
| 40 | + |
| 41 | + |
| 42 | + |
| 43 | +Take note of the directory where you downloaded the ledger data, as you will need it for the next steps. |
| 44 | + |
| 45 | +## Start Clio with Full History |
| 46 | + |
| 47 | +To start the Clio server with full history: |
| 48 | + |
| 49 | +1. Navigate to the `clio/build` directory and run the snapshot server using the following command: |
| 50 | + |
| 51 | + ```sh |
| 52 | + ./clio_snapshot --server --grpc_server="127.0.0.1:50052" \ |
| 53 | + --ws_server="0.0.0.0:6007" \ |
| 54 | + --path=<path_to_full_history_folder> |
| 55 | + ``` |
| 56 | + |
| 57 | + Replace `<path_to_full_history_folder>` with the path to the folder where you downloaded the ledger history data. |
| 58 | + This command starts a lightweight [Go server](https://github.com/XRPLF/clio/tree/develop/tools/snapshot) that provides the snapshot data over gRPC and WebSocket, and acts as a data source for Clio when running in full history mode. |
| 59 | +2. Edit your Clio configuration file to connect to the snapshot server. Start with the `etl_sources` configuration: |
| 60 | + |
| 61 | + ```json |
| 62 | + "etl_sources": [ |
| 63 | + { |
| 64 | + "ip": "127.0.0.1", |
| 65 | + "ws_port": "6007", |
| 66 | + "grpc_port": "50052" |
| 67 | + } |
| 68 | + ] |
| 69 | + ``` |
| 70 | + |
| 71 | + Ensure the `grpc_port` and `ws_port` values match the ones used in the `clio_snapshot` command from **step 1**. |
| 72 | + |
| 73 | +3. Set the `start_sequeunce` and `end_sequence` to define the ledger range that you want. For example: |
| 74 | + |
| 75 | + ```json |
| 76 | + { |
| 77 | + ... |
| 78 | + "start_sequence": 32570, |
| 79 | + "finish_sequence": 10000000, |
| 80 | + } |
| 81 | + ``` |
| 82 | + |
| 83 | + For the **full** ledger history, set: |
| 84 | + |
| 85 | + - `start_sequence`: `32570` (the oldest ledger version available in the production XRP Ledger history). |
| 86 | + - `end_sequence`: the latest ledger available in the downloaded history. |
| 87 | + |
| 88 | + {% admonition type="info" name="Note" %} |
| 89 | + You can find the actual start and end sequence in the `manifest.txt` file located in the root of your synced full history directory. It is formatted as `start_sequence | end_sequence`. |
| 90 | + {% /admonition %} |
| 91 | +4. Start the Clio server along with your chosen database backend (e.g., ScyllaDB or Cassandra). Once Clio is up and running, it automatically extracts data from the `clio_snapshot` server. To confirm that ledger data is being populated, you can monitor your database. |
| 92 | + |
| 93 | +This process may take some time, as Clio is ingesting and indexing the full history dataset into the database. You can tell the extraction is complete when the `clio_snapshot` server stops printing output to the terminal, which means all requested ledgers have been served. |
| 94 | + |
| 95 | +## See Also |
| 96 | + |
| 97 | +- **Concepts:** |
| 98 | + - [Ledger History](../../../concepts/networks-and-servers/ledger-history.md) |
| 99 | + - [The Clio Server](../../../concepts/networks-and-servers/the-clio-server.md) |
| 100 | +- **References:** |
| 101 | + - [Clio Github Repository](https://github.com/XRPLF/clio) |
| 102 | + |
| 103 | +{% raw-partial file="/docs/_snippets/common-links.md" /%} |
0 commit comments