[FEATURE] Timeseries (Upvote if want)

[bitnom]

always useful

[madolson]

What exactly do people want with timeseries? I haven't heard all that much about it.

[robswc]

I currently use Redis Timeseries for pooling and accessing data that is streamed from an external source.

[madolson]

@robswc Would you be willing to say more. What type of aggregation are you doing? What is the size of the data? When do you truncate the data?

[robswc]

Sure! (sorry for the delay). Basically just signal processing. I don't use any aggregation or truncation, just setting an expiration. Something like weather data, recording the temperature every second, then pulling the last X and processing in an application.

[aniketawati]

We use it for real time calculation of customer attributes based on user behaviour. Like total purchase in last x hours/days.
We use different aggregation types, and also down sampling feature. This is currently used at a fairly large scale.

[songzelidd]

Would like to use redistimeseries to support both time-series range extraction and aggregation for a real-time feature store for ML prediction purpose.

[bgalvao]

+1!

[ccollie]

I'm working on a rust timeseries library that I had hoped to integrate into redis. If there is interest, I can devote some effort, but it would definitely need some external contributors.

[pior]

We're heavily using (plain) Redis to implement realtime rolling windows like "counting events in the last 30 days".
It works especially well with very sparse data on tens of millions time-series.

Something else that has been very useful is that incrementing the current time-series "bucket" returns the current count.
We've been using this to trigger other events.

[bgalvao]

I'd love to be pointed to a reference on how I could do this as well, as I am facing the same problem!

[ccollie]

I'd love to be pointed to a reference on how I could do this as well, as I am facing the same problem!

What are your requirements beyond what's in redis-timeseries ? I have code implementing parts of this, but my focus is slightly different. If you'd like some pointers, I'd be glad to help.

[ccollie]

If anyone is willing to collaborate on specs for an MVP, I can devote some time to getting something started.
For a first cut, i'd suggest

TS.CREATE
TS.DEL
TS.ADD
TS.MADD
TS.MGET
TS.RANGE
TS.MRANGE

Aggregations and compactions would come later.

[madolson]

You can review similar RFCs here: https://github.com/valkey-io/valkey-rfc. If you want to create a draft, people can update it later.

[ccollie]

First draft of the Valkey TimeSeries spec is posted here. Contributions and critiques welcome !

[wgordon17]

@ccollie is this intended to be API compatible with Redis's TimeSeries?

Edit: Reading comprehension is important 🤦‍♂️

the module is designed to be API-compatible with Redis Ltd.’s RedisTimeSeries.

[ccollie]

Actually the aim is to implement a superset or the API

[m2b]

@ccollie this is a great document. Threes things come to mind:

1. Does all timeseries data need to be stored in memory. We are using Redis TS and keeping up to two years of 5 min samples plus 30 days of high frequency samples (>=1sec) and we are hitting 64GB. Obviously the older the data the least frequently accessed but still needed for analysis of events.
2. I would be very interested in not only querying for time filters but for values (e.g. get me all the timestamps where the metric is less than X). This is quite useful in analyzing data. For example, all the times that weather temperature dropped below 0 or rose above 50 c.
3. Leveraging something like ZRAM in linux as a means to store the less frequently used data in compressed form.

[ccollie]

@m2b ,

1. Does all timeseries data need to be stored in memory. ... Obviously the older the data the least frequently accessed but still needed for analysis of events.

Currently yes. I'm using the infra provided by valkey itself, which has no analog of Redis On Flash. The only alternative at the moment is better compression. A method which I'm toying with is using a columnar block type and encoding with pcodec , which according to docs should offer near the theoretical limit of compression. In addition, there's configurable decimal/significant digits per timeseries which may help in cases where full precision is not needed.

2. I would be very interested in not only querying for time filters but for values (e.g. get me all the timestamps where the metric is less than X). This is quite useful in analyzing data. For example, all the times that weather temperature dropped below 0 or rose above 50

We support redis-style range queries which have value filters of the form FILTER_BY_VALUE min max

3. Leveraging something like ZRAM in linux as a means to store the less frequently used data in compressed form.

A stretch goal is to have configurable tiered storage where less frequently accessed data uses blocks with higher levels of compression. Any OS level support will have to happen in Valkey itself.

[m2b]

@madolson where does this feature stand?