Managing a Shared Ledger: New CIP-Like Process?

Developers Node Development
1

We are quickly approaching a significant new era in our history: a multi-node network. There are a handful of node implementations in progress with new ones seemingly being announced every day.

While client diversity is a fantastic thing for the ecosystem, it necessitates clear communication and consensus in a way that we, as an ecosystem, have never really needed before. The ledger team has done amazing work including the community in the process, but has never had to support other teams implementing the node before.

At the node diversity workshop, Ori Pomerantz talked to us about testing in Ethereum in a multi-node ecosystem. He briefly mentioned the separation between EIPs and ERCs, where an EIP is a proposed improvement to the Ethereum protocol and an ERC is a proposed standard for users of the protocol.

I believe that it is worthwhile for us to explore a similar distinction in proposals in our own ecosystem. A CIP-like process is the perfect way for us to manage changes to the ledger, requiring collaboration and agreement from multiple node maintainers. It also provides, by default, a clear change log for hard forks. However, the CIP process itself, in it’s current form, does not feel like the best place for it. The CIP process is already hard to follow and encapsulates more than just the ledger. I believe we should design and implement a new process, specifically for changes to the ledger to streamline the process while ensuring that the ledger truly is a shared resource.

I would be happy to help design such a process, along with contributors from IO, Pragma, and other node implementors, if this is something we want. Ironically, I’d argue that this process should be specified in a CIP, even.

I’d love to hear feedback and ideas from the community!

3 Likes
2

@JSHyCS it’s great news that you’ve joined the forum after introducing the community to the rapid progress towards alternative nodes. Anyone wishing to follow the more technical details of this can bookmark this candidate CIP and follow the links to related CIP and Ledger discussion:

CIP-0154? | Orthogonal Certificates #1021

Most readers would interpret this statement to mean the CIP process is “hard to understand” or “difficult to observe”. It is neither of these things because

  1. absolutely everything about it, including everything editors do, is explained in nontechnical terminology here: Cardano Improvement Proposals (CIPs): Wiki
  2. anyone who wants to follow CIP statuses & evolution can subscribe at the top bar to get 100% notified of all activity in their own interests by email: more effectively & efficiently than X or any other means of social networking.

So @JSHyCS please correct me if I’m wrong but what you meant is: It’s practically impossible for architects, developers, innovators and subject matter experts to follow the CIP process for every change they make or plan. You are right about that and just last week this issue was posted, which just as you’ve said centers on how Ledger changes might be catalogued effectively under current circumstances:

The Ledger / (multi) Node specification is another application that would be served by a standards review process — likely with benefits from also using GitHub — that would resemble CIPs but provide a more lightweight & granular contribution mechanism.

This goal was already discussed late last year when CIPs were proposed for Governance propositions: and, despite the legitimacy it was suggested this would provide, was ruled out in practical discussion for these reasons: https://github.com/cardano-foundation/CIPs/issues/937#issuecomment-2476061813

… mainly due to the necessarily long, slow lifecycle of CIPs that you can see here:

… and so of course your own reasons above would also suggest a different process. The goal I think would be to accommodate the CIP Process’ goals of accountability without the lengths of time & documentation requirements for “improvements” that CIP peer review imposes for “standards”.

cc @ch1bo @arnaud @rkuhn @whatisRT @ryun1 @Crypto2099_Corp

3 Likes
3

I think it’s worth trying to categorize the various kinds of changes that one would want to make and try to figure out the needs of those categories. I can think of a few:

  1. Large new features
  2. Small tweaks
  3. Glue (i.e. how does X interact with Y)
  4. Security fixes

CIPs have so far served us quite well for number 1, and I’m not sure anything needs to be fixed there. You want slow moving and relatively high-level discussions to iron these out. For the others things tend to be a lot quicker, since they often happen relatively late during development, which means there is a lot of pressure to just do something. They are also usually quite technical, so there’s often not much value in discussing those from a high-level POV.

Another thing is that the CIPs aren’t a process for deciding how the next era will look like. Someone (probably a committee) needs to make a decision, first on large features, and later on all of the smaller stuff. Historically, the people who made the big decisions were not the same than those that made the small ones. There are lots of things that I designed and nobody really questioned the details (outside of security audits) so it made it in. I think a decision making body here would be great.

2 Likes
4

Thanks for engaging so quickly!

Yes, one of the takeaways from the workshop was that we should utilize the forum more to discuss technical matters. Not only is it more public than discord/slack (which can be private), it is a great way to also archive conversations. And it’s infinitely better than Twitter/X.

No, I think I would mean that it is “difficult to observe”. I think it’s very clear how the process works, but, probably due to user error, I often feel behind on CIPs. I know it’s not just me that feels this, as I’ve heard the same thing from many key contributors to the ecosystem, developers and non-developers alike. That being said, I really value the CIP process, I think it’s one of the most important pieces of our ecosystem. I think even something as simple as breaking the CIP categories out to their own repositories, each managed by editors that are most knowledgable on the subject, would greatly improve the process.

I did not mean to suggest that CIPs weren’t working as they should or that they aren’t valuable. I just think that a process that is focused only on ledger changes would be better suited for this.

3 Likes
5

Hey Andre! Thanks for engaging.

I totally agree that there are different kinds of changes and that they would need to be handled differently. I also tend to agree with your general categorization, and this is the kind of thing that would be ironed out while designing said process.

I totally agree! I think that if we were going to design a new process for some subset of the categories of ledger changes though, we should include all ledger changes in that process. No reason to have two different places to go to find the history of changes.

This is especially true in a single node environment. However, as we move to a multi-node ecosystem, I think it is vital that those things are still discussed, even if not from a “high-level”. It is important that we understand how each node would be impacted by said changes.

Well, I think that there could absolutely be a part of a new process that is reaching consensus on an upcoming era. I would argue that the CIPs would be the “large features” and “smaller stuff” that is debated. The changelog would, ideally, just be a list of CIPs (or whatever we call them in the new process), and each CIP would be clear enough that there is nothing else needed.

It sounds like you have some really good ideas that could be fleshed out and combined to create a robust process for ledger development.

Most importantly, I don’t expect any new process to be without it’s growing pains. New people directly contributing to Cardano in new ways along with the discomfort of sharing your (not you, in particular, IO in general) baby is bound to create some ripples.

1 Like
6

I largely agree on the categorization of changes to the ledger of @whatisRT and that the big deliberate changes (1) are well served by using CIPs. CIP-31, CIP-32, CIP-33, CIP-40, CIP-42, CIP-69 etc. are all good examples of that process working well.

However, things are different for changes of categories 2-4 because they happen during implementation of planned changes. Also note that IMO not all planned changes to the ledger are covered by CIPs, although the situation has became much better in that regards.

I’d argue that this process should be specified in a CIP, even

@JSHyCS Do you know about CIP-84 (and linked CIPs)? I didn’t and just read it now!

I think it does a very good job of characterizing how ledger changes need to be a CIP or not. See section for example What merits a ledger CIP?

The criterion for deciding if a change to the ledger merits a CIP is as follows: changes to the ledger require going through the CIP process whenever every implementation of the Cardano ledger needs to be standardized on the details. […]

Bug fixes are an exception to this criterion, they do not merit a CIP except in the case that the fix is substantially complicated. […]

Serialization changes are another possible exception to the criterion. Many serialization changes can be handled as a part of the normal development process without the need for a CIP […]

While the CIP could be updated to reflect the emerging reality of multiple ledger implementations, I think the key issue at hand is that changes not covered by a CIP are not easily available for standardization across multiple implementations.

Especially in crunch times leading up to a hard-fork, it is crucial we ensure that knowledge is shared across multiple teams to avoid surprises after the hard-fork. While it MAY be desirable to also provide ample lead time and options to “block consumers” like downstream components and DApps, we MUST make sure that all ledger implementations used on “block producers” to agree on rules and serialization.

I’d like to propose the Cardano Blueprint initiative to incubate an improvement to the non-CIP part of maintaining a ledger. While it is in it’s early stages (especially the ledger parts), the aim would be to have neutral ground to experiment with how we could try to avoid issues like CIPs#1028. So far I know of these two relevant issues:

What other steps could we take to create an accessible, consistent view of how the ledger should work in an implementation-indepent way?

1 Like
7

Yes, I am familiar with CIP-84, I reviewed it when submitting my own CIP candidate for a ledger change recently.

To be clear, I am not suggesting that we totally abandon CIPs for ledger changes. I think ledger changes can be documented as CIPs, I just think that the process for review/approval would be significantly different enough from non-ledger CIPs it may make sense to identify them separately.

In my ideal world, any change to the ledger would be covered by a CIP. In practice that may be hard to achieve, but I think we should strive to have hard fork change logs simply be a list of implemented CIPs (or whatever name we give a new process, if different enough).

Yes, please! I (as you know) am a big fan of the blueprint project and I believe we should really put a lot of effort into maintaining it as a central place for knowledge sharing.

That is an excellent question which I think will have constantly evolving answers. @arnaud has organized a poll to select a time for a first “Node Show and Tell” meeting where node maintainers can meet to discuss open-space style topics. I think this meeting is a great place to start to define a process for this kind of a process, as well as identifying other steps we can take to support node diversity. I hope anyone reading this that is passionate about node diversity attends!

1 Like
8

@COSDpool

We had our first “Show and Tell” node diversity meeting today and one thing that came out of it is that it is probably worth simply updating CIP-84 since ledger CIPs already have a slightly unique process.

I am happy to spearhead the updating process, but I’d love for it to be a collaborative effort. Perhaps @TheRealAdamDean, @whatisRT, @KtorZ, and others would have some opinions as well?

If anyone is interested, I’d love to schedule a call to discuss what should change to facilitate smooth and agreeable ledger upgrades. Otherwise, I can charge ahead on my own and receive feedback during the CIP process as usual.

1 Like
9

thanks @JSHyCS - I think we have enough documented in the tentative agreement above and in CIP issue Ledger rule changes: documentation & consistency? #1028 to put them into some edits for a CIP-0084 pull request.

If you can submit a PR with what you think should be there (and anything that’s become obsolete to be removed) I’ll go over it to double-check all issues pointed out so far have been addressed… and will flag anything we might not have consensus on for discussion here and/or there.

So I think we should be able to do this without a conference call (I would have nothing to contribute verbally): but if one is scheduled then I’ll be there.