We’ve spent the past two weeks evaluating and testing a variety of automated server monitoring systems to run on all our global SL servers for AdaPools.io. This aspect is a critical part of ensuring that as a global pool, we never miss a slot election and thus reward for our members.
After running the two finalist via live trial this week, we’ve selected our winner, signed a deployment contract, and it’s now running live on the Singapore/Frankfurt/Amsterdam smart nodes (Chicago is being used for R&D right now).
What is really neat about the system and the reason it won, is that it uses machine learning/AI to evaluate the ‘normal’ parameters for the entire system (CPU, memory, network io, etc), brings in it’s own data from running on lots of other same OS servers, and then uses that to understand what is normal and intelligently monitor for when a problem is brewing and of course alert if one is happening - it already comes with a whole set of interdependent things to watch.
All the other systems we evaluated monitor things but you have to set arbitrary/dumb thresholds for alerts (i.e. CPU% > 80%), when in fact many things need to be looked at to understand if this is a problem or simply a short term change such as an automated backup being run, etc.
This system also solves the ‘how can we auto-monitor each and every node, 24/7/365’ and also see everything at a glance on the human side.
There are separate screens for each server, and more screens that show how the machine learning AI portion is viewing things.
Today and Monday we’ll be adding in the custom code to specifically monitor the Cardano SL node itself on each server as a process, currently the AI is just setup to monitor the entire server as a whole.
As a bonus, in talking with the different vendors, I’ve had a great chance to evangelize ADA. In every case it usually goes from ‘we are evaluating monitoring systems for our global server network’ to ‘its for a cryptocurrency pool’ to Ada and all the reasons we are doing this (Gen 1 BTC vs Gen 3 Ada, etc). Anyway, fun conversations and helps to spread awareness and interest in Cardano/Ada b/c people are interested in learning about what we are doing and why.
We’re adding a blog to the site to document our progress on the global pool network here - http://adapools.io
We’ll have more updates and screenshots as we continue to perfect the infrastructure for staking and await the Shelley release.