I store millions of user feed records in Redis under one key. What's the best practice for structuring and retrieving large datasets efficiently?
Loading
I store millions of user feed records in Redis under one key. What's the best practice for structuring and retrieving large datasets efficiently?
Know the answer? Post it — somebody with the same question will find it here.
Sign in to answer this question
It is the same account you read, post and publish with — and you will come straight back to this page.
Tasadduq BurneyPosted Dec 8, 2025, 5:52 PM
Hey,
Hope you're keeping well.
Storing millions of records under a single Redis key is a scalability bottleneck because large
LRANGEorZRANGEcalls block Redis’ single-threaded event loop. In .NET, partition the feed into multiple keys—e.g., per user, per time window—using a sharding strategy so retrieval is scoped to smaller datasets. Use sorted sets for time-ordered feeds and apply pagination viaZRANGEBYSCOREorZREVRANGEto avoid loading the entire dataset. For serialization, prefer compact binary formats likeMessagePackwithStackExchange.Redisto reduce memory overhead and CPU cost. If hosted on Azure, consider Azure Cache for Redis with clustering enabled to distribute keys across shards for horizontal scaling.Thanks and regards,
Taz