AvatarPose

Demo

About

The problem it solves ## Overview This project uses Machine Learning + zero-knowledge proofs to create NFTs for users. It uses [face landmark detection](https://ai.google.dev/edge/mediapipe/solutions/vision/face_landmarker) to detect facial features in the uploaded video. These poses can then be used to animate an SVG avatar. This approach to NFT generation aims to create NFTs from “lived experiences.” We choose to call these experiential NFTs. ## The problem 1. Current NFTs often lack a meaningful association to their owner or buyer, making them absent of any sentimental value. Without a personal experience tied to them, they feel more like digital assets than cherished memories. 2. Current NFTs lack a meaningful association with the creator. There is no trace of the creator's hand in the work. ## The solution (Benefits of using this approach to NFT generation) - **Value derivation.** Our NFT objects have a legitimate claim from where they derive value (unlike most of the stuff on the market now). This is the key value proposition for this project. If you know that an animation uses [poses objects](https://ai.google.dev/edge/mediapipe/solutions/vision/face_landmarker) based on a recording of Vitalik, then you'll surely be interested. “Using poses of a certain individual for your avatar allows you to share in their experience.” - **Market differentiation.** This approach is unique and brings much-needed innovation to the NFT space. User Interaction and Data Flow 1\. A user logs into the platform using a social login \- we use a **wallet abstraction** service. The user uploads a video (< 5s) to the DApp frontend. 2\. The [ezkl](https://app.ezkl.xyz) backend receives the video frames and processes them to detect human facial landmarks. We use ezkl to run our face landmark detection model **network.onnx** on the image **input.json**. Ezkl produces a **proof.json** file that contains the detected facial features and a zk proof. 3\. Our dapp backend then stores the following NFT metadata on IPFS: - **facial landmarks** - **svg** image - **original video** (optionally encrypted before storage) - Some extra info e.g. name of NFT Now, we mint the experiential NFT and then send our data to [NFT.Storage](https://app.nft.storage/v1/docs/intro) for enduring preservation. **NFT.Storage** is a Filecoin service specifically designed for the long-term preservation of NFT data with a one-time payment ($2.99 per GB). This service requires us to upload a CSV that contains a list of token IDs and CIDs for our NFTs. 4\. At the frontend, a user queries the detected **facial landmarks** plus **svg** avatar and uses these to generate an animation (**gif** or **video**). They can then list (or show off\!) this animation on an NFT marketplace. **NOTE on PRIVACY**: a) The main intended use case is for famous individuals to create NFT poses and list them. Users can then buy the facial landmarks and choose an SVG avatar for them. **This means that the videos are already public (no privacy concerns).** b) In case a user wants to have their data private, we can also do that. Ezkl provides for inference using a private ML model and private data. This is different from a) where we use a private model + public data. The project architecture and development process The main components are: 1. frontend (browser) 2. ezkl backend (via REST API) 3. blockchain (AttestSingleProof.sol and ERC721 contracts) Key differentiators and uniqueness of the project This project showcases how NFTs can be **meaningfully** associated with the creator or buyer. Trade-offs and shortcuts while building - **Model quantization**. The inference model uses half-precision and its current size of 360kB. We can quantize it to about 200 - 250kB. We didn't do this for the hackathon because the onnx quantization feature does not out-of-the-box produce a model compatible with tract. **Side note**: Ezkl uses the [tract](https://github.com/sonos/tract) onnx rust library Reducing the model size would reduce the pk size produced by ezkl by an estimated >30%. This means faster proof generation and less memory usage. - **Model splitting**. We can also split the model into sub-models for faster processing. At the moment, it takes ~7 minutes to generate a proof. This isn't bad, but if we need faster speeds we could use the option mentioned here. - **Single proof attestation**. In this demo, we are using Single proof attestation **AttestSingleProof.sol** (i.e the contract deployment can only attest for a single proof). This is because our **Halo2Verifier.sol** which would have verified for any number of proofs that use our vk and pk, is big (1MB) and can't be deployed at the moment. We still need to make modifications (mentioned above) to reduce its size.