CIP extended 721

CIP extended 721

Hello, I have been working on a new extended standard for the informational CIP 25. I would love to get some feedback!

Abstract

This standard proposes a fully on chain method to store and access metadata via the use of references or pointers to data.
This would allow users of 721 to increase or reduce the amount of data stored on chain.

Motivation

  • Large token mints that duplicated data could be dramatically reduced in size by pointing to one transaction payload that contains a ‘boilerplate’ structure.
  • 16kB is the upper limit of data in each transaction but If a user wanted more there is no alternative than to store that data off-chain using an external service such as ipfs.
  • The use of pointers to data is an extremely powerful tool. For example If a pointer, pointed to pointers you could soon minic something like the ext2 filesystem fully on chain.
  • The aforementioned is all currently possible but there is no informational standard.
    This CIP aims to describe is a standard so that if a user does want to reduce or increase there metadata size it can be queried by 3rd parties that look for the version 2 tag attached within the 721 metadatum transaction label

Specifically, this proposal aims to solve for the following:

  • Provide a mechanism to reduce duplicated metadata into one or multiple transactions that can be referenced
  • Provide a mechanism that allows on chain data to be referenced
  • Provide a mechanism to reference on chain data in a set order
  • Provide a mechanism to reference on chain data stored in a different policy

Specification

Please note I offer optional single char names to allow futher reduction in metadata size if desired

Inside the 721 metadata tag

reserved 721 metadata tag description example
references, refs, r contains data tag and src tag 'r':{ "data:"text/plain","src":[0,1] }
payload, p contains raw data 'p':{ "0":["h","e","l","l","o"], "1":"world" }
version this is described in 721 not sure if it should be version 1.1? instead of 2

inside the refences tag

tags that are reseved inside references description example
data contains the data type MIME for example if base 64, data:“text/plain;base64”
src contains a list of poitners to payloads "src":[0,1] or "src":["payload0","payload1"]
policy, p references another policy (optional). Default (if none) is current policy 'r':{"data:"text/plain","src":{[0,1]},"p":"<POLICY_ID>}"
txhash, tx references a specific tx hash (optional) 'r':{"data:"text/plain","src":{[0,1]},"tx":"<TX_HASH>}"
type the format/type of the data inside the payload 'r':{"data:"image/jpeg","src":{[0,1]},"type":"ipfs"
b, back, backup fallback, if a link to the data is used in the ‘src’ tag and is broken revert to a fallback link. 'r':{"src":{[0,1]},"b":{[2,3]}} or for multiple fallbacks use 'r':{"src":{[0,1]},"b":"{'0':{[2,3]}, '1'{[4,5]} }"

defintion of the different reference types

type desciption
ipfs data is stored at ipfs
url data is found a url
api data is found at api get request
utf-8 data is directly in payload as utf-8
b64 data is in base 64 encoded in payload
reference? advance feature, points to a reference
none assume data is raw and using the defined media type

?references in payloads could cause issues with circular dependencies, maybe this should be more like inode in ext2 or the depth defined

{
   "721":{
      "<POLICY_ID>":{
         "<NFT_NAME_B16>":{
            "project":"<PROJECT>",
            "name":"<NFT_NAME>",
            "references":{
               "type":"raw"
               "data":"text/plain"
               "src":[
                  0,
                  1
               ]
            }
         },
         "payload":{
            "0":["Hello"],"1":["World!"]
         },
         "version":"2"
      }
   }
}

Retrieve payload information using references

  1. Find all transactions with the 721 label
  2. Check the version tag
  3. If a payload is found append that to an map or some data structure
  4. Given a nft find there references
  5. Concatenate the data for all given references

In example

nft 0 has the payload "p":{"0":"Hello","1":"World" } and the references "r":[0,1]
nft 0 == "HELLOWORLD"
nft 1 has no payload and the references [0]
nft 1 == "HELLO"
nft 2 has no payload and the references [1,0] (note the order)
nft 2 == "WORLDHELLO"

Pseudo code

to get the metadata from payloads

for each transaction within the policy
   if metadatum tag is '721'
      if 'payload' or 'p' in json['721][<POLICY_ID>]
         for payloads in ['721][<POLICY_ID>]['p']
            for refrenceName, data in payload
               add data with referencename to datastructure    

to build the metadata for each nft

refs = json['721'][<POLICY_ID>][<NFT_NAME>]['references']['src']
dataType =  json['721'][<POLICY_ID>][<NFT_NAME>]['references']['data']

for ref in refs
   nft data += data from datastrucure indexed with ref

Backwards compatibility

Used the version tag described in CIP 25.

Further considerations

References

If we wanted to add more powerful referencing we could allow for references to references
"payload":{"r":{"src":{[98,99]},"p":"<POLICY_ID>}"}}
Like a block of pointers in the ext2 filesystem

Duplicate data

There could be issues with duplicate payloads. The solution I propose is just to use
the latest version of that payload.

In my opinion, this is not a good idea. A blockchain like Cardano is not for storing jpegs and to archive data files (books or internet pages).

Cardano already stores metadata on chain (CIP 25) and this is a slight extension to that standard which is completely possible to implement now. This proposal is about improving upon something that exists in an innovative way to allow more creative things to be done fully on chain.

I do agree that ‘congesting’ the chain would be bad, however with scaling and future development this will not be an issue. It’s also not likely to be implemented for the use of ‘heavy’ file formats such as .jpg as that would require a lot of cross referencing to say which pixel should be where and what color and most people don’t have the time (it would also be expensive).

As for the point on ‘Cardano is not for X’ it already does in some cases.

Metadata is small. This proposal is to store large files. This is not good. It is bad enough that the blockchain is storing all those smart contracts in every transaction, until the CIP33 will be implemented…

2 Likes

Ideally this would not be used for storing large files. If you look at the example that uses 16kb. This policy 1483b5476970146aef75579cf0e14be32c5fbea3f3bb0c7d48eabb46 on the testnet. You may notice that only 9 transactions contain the metadata to generate the asset. From this many permutations could be generated without any more payloads. Which is a small hit.

The user side implementation should be creative and transaction fees would stop abuse as it would make no sense to upload a full .jpg due to costs. This can already be done using just 721 I just want to formalize this as an informational CIP. For example I could implement this right now but sites like pool.pm and most wallets would not know what to do when the src tag points to references.

My goal here is to expand upon 721 using the version tag. I suppose I could even remove the 722 tag as a transaction metadatum label and embed it all within the 721 transaction metadatum label. However I believe a payload tag is cleaner.

Made a slight edit there is really no need for the transaction metadatum label.
(Added a section called Example usage only using 721)

In relation to congesting the chain issue I believe this CIP will have a minimal impact. It definitely should not impact Cardano in way that could cause major strain (I know nfts already can cause strain and this is obviously slightly more, but not enough to warrant dismissal imo).

As I’ve mentioned earlier it is not practical to mint jpegs with this method. It makes far more sense to use ipfs or some other decentralized off chain method for that usecase.

This CIP is designed to allow for creative usecases of cip25 and the 721 transaction metadatum label. The use-case I propose is with a mint size for 8192 nfts in kB would be the same as similar projects that mint 10,000 assets.

I will give an example usecase here.

  • A user wants to generate 5000 NFTs. They are not using a ipfs.
  • The data required to generate their art is 50kB of say javascript.
  • This is not possible currently…
  • Enter the new CIP the user can now split that 50kB into 4 transactions and segregate each line of code info a reference.
  • Because we can define the references we could reference a variable that makes the nft blue in one mint and red in another.
  • In this example the mint cost in kB is the same as any other project + a 50kB payload

You could image some quite powerful usecases with this CIP and I am looking for feedback or improvements to make this more of a standard.

You do know that there are already a lot of NFTs storing their content on-chain just using data:-URLs in the existing 721 metadata? I don’t really see, why we would need more.

Edit: Ah, reuse of JS across several different NFTs. … Hmm, that seems very application-specific. But could make sense.

1 Like

Yes I am aware of nfts using urls to link/pin their images :grinning:
This is more about providing an option to extend the metadata. It is not only for projects that use images. There maybe some blockchain use cases in which a little more than 16kB of data is required or data needs to be referenced from another nft or policy (which would be possible if we extend this CIP to include a tag such as policy reference)

edit* I’ve updated the initial post thanks for the feedback

I would also like to propose a way to reduce data on chain using extended 721

NFT 0
{
  "721": {
          "<POLICY_ID>": {
                "<NFT_NAME_B16>": {
                "name":"nft0"
                "traits":{"apple":"2", "pear":"1"    } // random data
                "references": {
                 "ext": ["0"]
                   }
          },
          "payload": { // this data is only stored in one tx
                  "0" : {
                                "project": "<PROJECT>", // data that will be duplicated for each nft
                                "website": "<WEBSITE>",  
                                "adahandle": "<handle>",
                                "discord": "discord",
                                "telegram": "<WEBSITE>",
                  }
                  },
          "version": "2.0"
          }
     }
}
NFT N
NFT 0
{
  "721": {
          "<POLICY_ID>": {
                "<NFT_NAME_B16>": {
                "name":"nft0"
                "traits":{"apple":"2", "pear":"1"    } // random data
                "references": {
                 "ext": ["0"] // references instead of duplicating data saving space on chain
                   }
          },
          "version": "2.0"
          }
     }
}

This implementation would save space on chain with nfts that duplicate a lot of data. Once such example is ada handle which contains duplicate data in every minted nft. If the reference tag is short enough for example “ext”:[0] instead of “references”:{“ext”:[0]} you could start to have a massive impact on reducing data on chain.

A rough saving calc could be determined as such:

100bytes for this data in each nft using standard 721
core:{"og":0,"termsofuse":"https://adahandle.com/tou","handleEncoding":"utf-",prefix:"$",version:0}

9 bytes if we use
"ext":[0]

saving 91 bytes per nft (probably around ~80 if we use the… refences:ext:0 option)

currently there are 108849 ada handles meaning this new CIP could have saved
9.905259 megabytes of data duplicate from being stored on chain.

I agree with george, I don’t think the blockchain is the place to store NFTs. My NFT, for example, has 5.1 MB in size, this would require lot’s and lot’s of transactions to store it on-chain. I do see the value, though, on preserving the integrity of the image and not rely on IPFS. In which case I think storing the hash of the base64 could be a better idea instead (maybe this could even be the token name).
Then we validate NFTs by making sure the image they hash of the base64 image they reference is equal to the one in the metadata or token name. To stop worrying about IPFS becoming suddenly corrupted somehow, we could consider the last metadata URL reference and, if something goes wrong, the user would simply send the NFT to him self in a new transaction with the new reference in the metadata matching the initial hash.

2 Likes

I like the idea of base64 hash verification to be an optional tag in the 721 data.
If you have 5MB of raw data why would you use this it would be a headache and storage like ipfs makes sense. The proposal allows a nft to use more than 16kB if needed but also the referencing could greatly reduce the size of large nft projects that duplicated data if use correctly.

The main point here is that this doesn’t define anything that veers from what is already implemented on chain. It’s an informational standard that should enhance 721. However, I do understand the concerns with putting more data than needed onto the chain being unnecessary whilst there are alternative options. After creating the initial proposal to allow me to store instructions to build a nft in a defined order I realised that this could also be used to reduce data see the 10th reply for an example.

  1. There is no concerns. If someone would like to fill up his/her transaction with anything then he/she pays the storage price as tx fee. You should be welcomed to fill up any transaction up to the max tx size to the 16kB. As this is how almost all blockchains are designed. Meaning, you use its storage then therefore you pay for it as tx fee. If anybody think it differently then they do not fully understand blockchains.

  2. Though there is a very-very big concern. How do you distinguish between two or more very similar 721s metadata e.g. same asset id (policy id and asset name are the same), but the traits are different, for example? As anybody can post a tx with any metadata and there can and definitely will be adversaries who would like to cheat and/or trick the system. What missing is some anchor, like signature etc., which proofs that the 721 metadata was created by those who are entitled.

1 Like

First thank you for actually looking into this and for the feedback!

  1. ace, you are the first person not to dismiss this due to that ‘concern’ :blush:
  2. To get the referenced metadata you must first query all transactions under your unique policy id (there should be only one) this first step should resolve the issue mentioned. I believe policy inside the 721 will have no impact

pr for reference look at the pseudo code section.

*edit: hopefully this isn’t a low quality reply, posted on my phone. If so I’ll answer in more detail tomorrow

You do know that a quite similar mechanism is already used for NFT metadata?

It is not a problem, since the metadata are only considered from minting transactions for the corresponding token. And minting transactions have to have a signature fulfilling the policy, so “those who are entitled”.

The existing CIP 25 is quite explicit that the last minting transaction with metadata is the relevant one, because there can be more than one.

1 Like

Here I mean that there should only be one policy id onchain to query… if there are duplicate metadata then the last one is the valid one.

explicitly i mean
1 policy_id → many transactions-> with many metadata

I’ll stop replying now until the morning (UK)

1 Like

Yes, but keep in mind in that case it’s not a real NFT (with updatable metadata) as it can be burned and minted again and again i.e. no difference with a bank or any other authority that promise I won’t do (mint more of) it. But, in other cases you are right that the original minting tx can be used for that anchoring but it can be very resource hungry.

1 Like

If the policy forbids it after a certain slot, it can’t be burned and minted (but then also the metadata can’t be updated anymore) … under the same policy ID.

Even “real” NFTs on other blockchains don’t give you guarantees that the minter had the copyright, that the minter won’t mint again under a different policy/contract, … It’s crypto space, it’s far less trustworthy than your average bank … except for the miniscule part that is ensured by mathematics.

1 Like

Indeed this does make it fungible until the policy locks (if the policy locks). However it is not so different to a NFT that contains a url link that points to data, that data could change at any given time. At least here once it’s locked it’s a true NFT.

In the PR I also suggest a tag called policy. If a user wanted they could generate the data on chain in a different policy that locks after creation. Then the current nft could point to a different policy that is already locked solving that issue.

Another benefit of the CIP proposal is the potential to reduce the total count of uploads to IPFS with minimal impact on the Cardano chain.

Example:

A user has a jpg nft set with 3 attributes each with 3 traits (total 9 traits). This would allow for 27 unique nfts

hats: tophat, cap, bald
eyes: sunglasses, sad, happy
body type: ape, fish, turtle

Instead of uploading 27 ipfs images the user uploads the 9 trait images.

What would this look like in the meta?
First lets build our references

{
   "721":{
         "payload":{
            "tophat" : "ipfs link to tophat ",
            "cap" : "ipfs link to cap !",
            "bald" : "ipfs link to bald!",
            "sunglasses" : "ipfs link to sunglasses!",
            "sad" : "ipfs link to sad!",
            "happy" : "ipfs link to happy!",
            "ape" : "ipfs link to ape!",
            "fish" : "ipfs link to fish!",
            "turtle" : "ipfs link to turtle!",
         },
         "version":"2.0"
      }
   }
}

Lets builld a nft of a sad fish wearing a tophat

{
   "721":{
      "<POLICY_ID>":{
         "<NFT_NAME_B16>":{
            "project":"<PROJECT>",
            "name":"<NFT_NAME>",
            "references":{
               "type":"ipfs",
               "mediaType":"image/jpeg",
               "src":['fish','sad','tophat']
            }
         }
      }
   }
}

We would need define a tag ‘type’ in references to tell the viewing application that the jpegs are using ipfs or links and need to be overlaid.

Before finalizing this CIP I would like to come up with other types that could be added to the standard. For example here are some types:

type desciption
ipfs data is stored at ipfs *probably deprecated by url
url data is found at url
api data is found at api get request
utf-8 data is directly in payload as utf-8
b64 data is in base 64 encoded in payload
none assume data is raw and using the defined media type

1 potential issue is that jpegs don’t overlap naturally. To solve this.
The client side should notice that there are more than 1 references defined and overlay each jpeg in the exact order defined in src. For example we used fish, sad, tophat… thus the client would render the fish, then the sad eyes then finally the tophat.

Obviously this scales if you have 4 attributes each with 10 traits (40 traits total) you can generate 10,000 unique nfts using only 40 ipfs uploads and the referencing method.

We could even define a way to define failbacks… by adding another tag like ‘src’ that points to the failback image if ipfs is corrupt

What if the link fails? Will I lose 10k nfts with hat type X… Not if you use the ‘backup’ (fallback) option.
More details here, in this commit