How can I verify that Thanos Store has backed up all retained data in prometheus?

[pritster5]

I recently added Thanos Sidecar, Querier, and Store Gateway to an existing Prometheus deployment.
Prometheus was configured to retain data for 1 year.

Before reducing the retention period to 14 days and offload historical queries to Thanos Store Gateway, I would like to verify that all the data has been backed up from EBS volumes to the S3 bucket I'm using as an object store.

How can I do so?

[dosubot[bot]]

To verify that Thanos Store has backed up all retained data from Prometheus to the S3 bucket, you can use the verify command. This command checks all blocks in the bucket against specified issues, ensuring that the data is correctly backed up.

Here's how you can use the verify command:

thanos tools bucket verify --objstore.config-file=<path-to-your-s3-config>

Replace <path-to-your-s3-config> with the path to your S3 configuration file. This process might take some time as it involves downloading and checking all specified blocks ^[1].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[MichaHoffmann]

That looks all fine - weird - can you check locally to make sure that there are older blocks on this Prometheus instance?

[pritster5]

So, unfortunately I asked this question after having already reduced the prometheus retention period from 1 yr to 14days, so the answer is likely that those blocks are already deleted. However, the sidecar was for sure running and uploading data to S3 well before I made the change to retention.

But just in case, how would I check?

[pritster5]

I ssh'd into one of the prometheus-server replicas/instances and ran this command to see if I could check the disk usage of the mounted PersistentVolume

[pritster5]

That looks all fine - weird - can you check locally to make sure that there are older blocks on this Prometheus instance?

Any ideas on how to troubleshoot this?

[pritster5]

Claude mentioned this, is this accurate?

By default, Thanos Sidecar is designed to upload TSDB blocks that are created after its installation. It does not automatically retroactively upload existing Prometheus data that was present before the sidecar was deployed.
To back up existing historical data, you'll need to use the Thanos Repair (or Compactor) tool to upload pre-existing blocks. Here's how you can do this:

Manually Upload Existing Blocks:

thanos tools bucket backfill \
 --objstore.config-file=s3-config.yaml \
 --datadir=/prometheus/data/01234 \
 --no-upload-compacted

Alternative Approach:

Stop your Prometheus instance
Copy existing TSDB blocks to a separate location
Use the Thanos Repair tool to upload those blocks to S3
Restart Prometheus with Thanos Sidecar

Key points:

Thanos Sidecar by default only handles new data blocks
Existing blocks require manual intervention to be uploaded
You'll need to specify the exact block directories to upload
Ensure you have the correct object storage configuration

To verify existing blocks:

Check Prometheus data directory
Look for block directories (typically named with timestamps)
Use Thanos tools to inspect and upload these specific blocks