Sol Scraper
Scrape on-chain data (Solana) to create APIs with deep specificity
About
The problem it solves # Features * create APIs from on-chain data which is not limited to just the generics (hash, fees etc) * add dApp specific information to the APIs * filter data using custom queries # Design Details * Scraper: create/save queries which will run as background jobs to fetch and parse on chain data * Query: specify the contract, filters, and fields you would like to export * Storage: Centralised storage for now in the form of MongoDB to store query metadata and the results from the queries
How it's made
Challenges we ran into * Understanding the working of existing dApps including their transactions, programs, accounts to form meaningful queries * Parsing of queries : for extracting values from complex programs like Serum, there is a need to understand the structure of the data present in the event queues to find fill/bid trades