The Cardano Foundation (CF) has published a Cardano Roadmap Tier List following an initial roundtable between Giorgio Zinetti (CTO, CF), Sebastien Guillemot (dcSpark, Paima Studios) and Matthias Benkort (Technical Director, CF) on the 28th March. A further roundtable was recently organised by CF on the 22nd April to continue the discussion.
It is great to see community initiatives like these and IOR welcomes discussion and debate from all actors in the Cardano ecosystem, particularly those which we wish to engage with more collaboratively and who are as notable as CF. IOR acknowledges these types of initiatives are much needed and we are working hard ourselves to improve how we engage with the Cardano community.
At the start of both roundtable discussions, it was stated that it is a fun TLDR exercise with people expressing their own opinions. However, we are seeing the tier list slowly becoming perceived as an official statement from CF and we would like to raise some observations we have made into the process and outcome (aka the Tier List) in light of this.
IOR Observations
The information used in the analysis for creating the Tier List was taken from several sources. One of the main sources was the Intersect Draft Bucket Budget Google sheet which was intended as a financial document, rather than a technical or scientific one.
The items grouped together include Research items proposed from IOR via the Intersect Product Committee, Engineering items proposed by the Intersect Technical Steering Committee, several items from the Intersect Product Committee’s roadmap, as well as from a few other sources.
The background of the evaluators in recent discussions has been weighted towards those with an engineering background, with several leaders of software labs within the ecosystem invited to take part. All feedback from technical/engineer community members is valuable. However, as far as we are aware, no one with a formal research background was involved in the tier list evaluation process. So we’d like to clarify a few points we think are important when assessing the relative merits of the different line items.
Software Readiness Timelines
It is important to recognise that research is a different discipline which operates on different time horizons to engineering. Maturing capabilities in emerging technologies and complex systems towards market readiness require considerable efforts that spans over multiple years.
The Software (also called Technology) Readiness Level was a guide consisting of 9 levels which was first developed by NASA, and has since been adopted by organisations such as the European Union, which outlines a development pathway with guidance on best practice.
Late stage SRLs 6-9 are product-orientated and near market, ideally with 12-18 month timelines, mid-stage SRLs 3-5 focus on innovative foundational engineering are 2-3 years out from mainnet deployment, whilst research functions at early SRLs 1-2 with a 3-5 year time horizon.
Engineering-Weighted Evaluation
When looking at the rankings from CF tier list, alongside those from other community driven initiatives (which we also applaud btw) such as the Cardano Roadmap Budget Workshop in Tokyo held in March, we notice a systematic bias in favour of product and engineering items over research.
It is perhaps not surprising that if you evaluate research items through an engineering lens, which considers near term impact, that research will rank poorly.
Yes, tactical short term research initiatives can support product and engineering to solve individual use cases. However, foundational research, by its nature, is not designed to solve challenges over a 12-18 month horizon. Foundational research solves tomorrow’s problems, not today’s, which we would argue requires a different scorecard to that of engineering.
Research Evaluation Timeline
So, how should you evaluate research over these longer time horizons? 5 years is a long time to wait, and could potentially be very costly, particularly if nothing comes of the research.
After all, early stage endeavours carry a high degree of risk. There are lots of unknowns, lots of things that can change (the market certainly will), and in many instances you are literally starting with a short problem statement. How do you predict the future?
You make bets. Strategic ones. Leading corporations do. Countries do. They all invest into scientific programmes. Like Cardano has done.
A scientific vision conducts foundational research in areas that, to the best of your knowledge, you believe the market will be in 3-5 years, and that are typically broadly applicable.
You do this, knowing that a number of these bets won’t come off, but the ones that do will create a source of value that is highly defensible, very hard to copy, provides competitive advantage and exceptional ROI. Ouroboros - one of the most cited and respected consensus protocols in the world - being a case in point.
A Research Evaluation Scorecard
Given this, what criteria should you use to evaluate research? Here are some suggestions from IOR:
Track record - what has the research group done before, what has been their impact?
Inputs - what is the team, what skills do they have, what networks are they in and what resources do they bring?
Proposed work - no-one can predict the future but does the vision, and the items under it, sound sensible and exciting?
Planning - do the project plans sound reasonable? Can the tasks be achieved? Are the deliverables realistic?
Flexibility - things change. Can the plan? Can it stay current and increase its chance for success?
Accountability - are there suitable checks and balances?
Impact - if a number of these research items are realized, what impact will they create?
IOR’s request to the community
We’re now entering a period where DReps will assess the many proposals put forward within the budget process. We’d encourage all DReps to consider the scorecard above as part of that assessment, or indeed any other they may choose.
It’s important that our proposal is assessed through a research lens - that recognises medium- and long term impact, not a product or engineering lens that looks within a much shorter time horizon.
If you are unfamiliar with evaluating research, or have any questions or concerns, please do reach out to us. We’d love to jump into calls and X Spaces with you.
We have a vision for the next 5 years, one that we believe will enable Cardano to lead in blockchain, and create the impact that’s much needed in the world.
We believe this vision is truly exciting. We sincerely hope you do too.