kamu-lilypad
A custom lilypad module for sql streaming through kamu.dev, this provides provenance, version control, and provenance of derivative datasets
Demo
About
This project puts kamu.dev into a bacalhau container and runs it by calling a lilypad v2 from the cli. It demonstrates that sql streaming can be performed through a lilypad module to help mange the provenance and veracity of a living dataset pulling from theoretically multiple sources. It takes a manifest from an external api, pulls it into kamu and than pushes it as a dag-pb to ipfs.
How it's made
A docker image that pulls ipfs and kamu is created, one provides a kamu manifest that instructions from where to pull data and how to store it in sql, once it pulls the data one uses the kamu system ipfs add <dataset> method to add the sql database into a ipfs dab-pb. One than sends the root cid into the lilypad outputs folder, pinning the dag