Lens Substream
A Rusty way to stream Lens data using Substreams
About
The problem it solves Decentralized social media generates a huge amount of on-chain data. This on-chain data needs a low-latency interface for it to remain accessible for all, along with adding the ability to perform real-time analysis and monitoring. Substreams are a data solution developed for The Graph Network. They allow developers to write Rust modules composing data streams with low-cost caching and archiving of blockchain data, high throughput processing, and cursor-based reorgs handling. This project builds a (basic) Substream in Rust to monitor on-chain Lens protocol data and provide a low-latency data streaming interface. Potential uses of such a tool could be for real-time analysis and monitoring of Lens data. Since the data is streamed, it can be directly injected into a variety of databases (BigQuery, Clickhouse, Kafka etc.), opening up a lot of distinct analytics use cases. A few unrelated citations: 1. [Gizmodo: Here Are 30 of the Hundreds of Subreddits Going Dark to Protest Reddit's API Changes](https://gizmodo.com/reddit-subreddit-api-r-gaming-1850510180) 2. [Verge: Twitter just closed the book on academic research](https://www.theverge.com/2023/5/31/23739084/twitter-elon-musk-api-policy-chilling-academic-research) 3. [9to5Google: Reddit’s API changes kill Android’s best third-party apps, including RIF and Sync](https://9to5google.com/2023/06/08/reddit-android-third-party-apps/)
How it's made
Challenges we ran into - Substreams are an extremely new technology. The most actively developed implementation by the team (uniswap-v3) is still in beta. Therefore, the documentation is extremely limited. I worked around this by simple trial and error, playing around with things until they worked. - Lens protocol is complex and involves multiple moving parts, this implementation only covers a small fraction of it. However, given the modular nature of the substream, this functionality can be added in the future. - Given the novelty of working with Substreams, I would particularly like to thank Alex and Ørjan from The Graph for helpful discussions.