# Grafana Loki as centralized log solutions for pool operators

**URL:** <https://forum.cardano.org/t/grafana-loki-as-centralized-log-solutions-for-pool-operators/100824>\
**Category:** Operate a Stake Pool\
**Created:** [8 May 2022 10:46 UTC](https://forum.cardano.org/t/grafana-loki-as-centralized-log-solutions-for-pool-operators/100824 "2022-05-08T10:46:24Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![os11k](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.cardano.org/os11k/32/43985_2.png) [@os11k](https://forum.cardano.org/u/os11k)\
**Post date:** [8 May 2022 10:46 UTC](https://forum.cardano.org/t/grafana-loki-as-centralized-log-solutions-for-pool-operators/100824/1 "2022-05-08T10:46:25Z")

</div>

As far as some of us here are running several cardano nodes, at some point it might happen that we might need to go through some cardano node logs, maybe it is relay what misbehaves or maybe you missed block and now you need to find out what happened. I personally used to go through logs with tail, grep and etc. That doesn’t seems very productive way. How you can filter all logs from all nodes for 12:00-12:05 UTC 5th of May for example? You can off cause grep it on several machines and then piece them together, but what if you can have a central location were all logs are stored and you can search through all logs from all nodes simultaneously or on particular box only or on several particular boxes and even create alerts based on particular log entry using regexp and etc.

So here I would like to explain how I personally build centralized log solutions for all my cardano-nodes. I used for that grafana loki, what will receive logs and then you can see them in grafana. As far as grafana is widely used by SPOs, I think it a reasonable choice, you don’t need to add much to your existing infrastructure.

First we need to install Loki. I personally installed Grafana & prometheus some time ago using docker-compose, so to install Loki I just needed to update docker-compose file accordingly.

Grafana-prometheus-loki docker files what I personally use for Grafana, Loki and Prometheus can be found here:

> **[GitHub - os11k/grafana-loki-prometheus](https://github.com/os11k/grafana-loki-prometheus)**
>
> Contribute to os11k/grafana-loki-prometheus development by creating an account on GitHub.

If you are starting without existing grafana, or you want to use docker now, you can just run 5 following commands(assuming we are using debian based Linux distro) and you will have new and shine grafana, prometheus and loki running in docker:

```auto
apt-get update && apt-get upgrade -y && apt-get install docker-compose -y
mkdir /docker && cd /docker
git clone https://github.com/os11k/grafana-loki-prometheus.git
cd ./grafana-with-prometheus/
docker-compose up -d --build

```

Don’t forget to update accordingly to your setup `./etc-prometheus/prometheus.yml`.

If you are here just for Loki, you need to update `docker-compose.yml` and comment out all parts related to grafana & prometheus:

```auto
version: "3.5"

services:
# grafana:
# container_name: grafana
# network_mode: "host"
# image: grafana/grafana:latest
# restart: always
# volumes:
# - grafana_data:/var/lib/grafana
# logging:
# driver: "json-file"
# options:
# max-size: "200k"
# max-file: "10"
# prometheus:
# container_name: prometheus
# network_mode: "host"
# image: prom/prometheus:latest
# restart: always
# volumes:
# - ./etc-prometheus:/etc/prometheus
# - prometheus_data:/prometheus
# logging:
# driver: "json-file"
# options:
# max-size: "200k"
# max-file: "10"
  loki:
    container_name: loki
    network_mode: "host"
    image: grafana/loki:latest
    restart: always
    volumes:
      - ./etc-loki:/etc/loki
      - loki_data:/loki
    command: -config.file=/etc/loki/loki-config.yml
    logging:
      driver: "json-file"
      options:
        max-size: "200k"
        max-file: "10"
volumes:
# prometheus_data: {}
# grafana_data: {}
    loki_data: {}

```

There are some other ways to install Loki, but I personally would avoid that:

> **[Install Loki on Ubuntu 20.04](https://lindevs.com/install-loki-on-ubuntu/)**
>
> …

When Loki installed, you need to configure your nodes to push logs to Loki. In case if your cardano nodes are running in docker, you just need to install docker module and restart docker engine

```auto
docker plugin install grafana/loki-docker-driver:latest --alias loki --grant-all-permissions
systemctl restart docker

```

you can verify that everything is fine:

```auto
docker plugin ls

```

And you should see your newly installed plugin for docker:

![image](https://us1.discourse-cdn.com/flex023/uploads/cardano/original/3X/9/3/93cc61028732a0611d33a1364c98f893a77a2c5f.png)

Then you can either to configure each container separately, for example I just added those lines in my docker-compose(don’t forget to put your Loki IP instead of loki-ip):

```auto
    logging:
      driver: loki
      options:
        loki-url: http://loki-ip:3100/loki/api/v1/push

```

Or you can configure it once for all containers by creating `/etc/docker/daemon.json` file(again do not forget to change loki-ip to IP address of your Loki box):

```auto
{
    "debug" : true,
    "log-driver": "loki",
    "log-opts": {
        "loki-url": "https://loki-ip/loki/api/v1/push"
    }
}

```

Keep in mind that containers must be recreated so they can start send logs to Loki. As far as I utilize docker-compose, that what worked for me:

```auto
docker-compose down
docker-compose up -d --build

```

More details:

> **[Configuration | Grafana Loki documentation](https://grafana.com/docs/loki/latest/clients/docker-driver/configuration/)**
>
> Configuring the Docker Driver The Docker daemon on each machine has a default logging driver and each container will use the default driver unless configured otherwise.
> Installation Before configuring the plugin, install or upgrade the Grafana Loki...

> **[Collecting Docker Logs With Loki](https://yuriktech.com/2020/03/21/Collecting-Docker-Logs-With-Loki/)**
>
> Loki is a multi-tenant log aggregation system inspired by Prometheus.It is cost effective, easy to operate and allows viewing logs directly in Grafana.In this blog post, I will show how to setup a Lok

If you are running your nodes not in docker then you will need to install promtail, what is a client which will push logs to your Loki:

> **[Install Promtail on Ubuntu 20.04](https://lindevs.com/install-promtail-on-ubuntu/)**
>
> …

You will need to configure promtail to too, here is simple config file, what should work(never tried, though):

```auto
server:
  http_listen_port: 0
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

client:
  url: http://localhost:3100/api/prom/push

scrape_configs:
- job_name: system
  entry_parser: raw
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__ : /var/log

```

Keep in mind that you must put correct Loki IP, in above example it is localhost, additionally you need to change directory where are cardano-node logs are stored, in example above it is `/var/log`, so if your cardano node logs goes in different directory, you need to update it.

Config file was taken from here: [grafana-loki-demo/promtail-local-config.yaml at master · rongfengliang/grafana-loki-demo · GitHub](https://github.com/rongfengliang/grafana-loki-demo/blob/master/promtail-local-config.yaml)

Loki do not pull logs, but rather promtail or docker engine are pushing logs to Loki. So if you are using default 9100 port for Loki, then from clients(cardano nodes in our case) should be able to access Loki on 9100 port. You can check that with telnet command from cardano nodes `telnet loki-ip 9100`

When Loki is installed and Loki client is configured(docker driver or promtail) you should add Loki as source to Grafana, it is same process as you did for prometheus:

Configuration =\> Datasources:

![image](https://us1.discourse-cdn.com/flex023/uploads/cardano/original/3X/8/b/8b7e19220a6aaea9d99f1d35d35161a7b845013b.png)

Press add data source and then lesect Loki from list:

 ![image](https://us1.discourse-cdn.com/flex023/uploads/cardano/original/3X/8/6/86dfc30dde330529350a22c5e49e01859d2b465c.png)

You will see following screen:

 ![image](https://us1.discourse-cdn.com/flex023/uploads/cardano/original/3X/5/3/53ae96c4d57635a81251767c234ce23a5febaf17.png)

If you are running Loki on same box as Grafana or in docker as I described above, you will need just to put localhost in URL, as it proposes:

 ![image](https://us1.discourse-cdn.com/flex023/uploads/cardano/original/3X/f/2/f25d2b102d5cf082f6d5d15b5ee046a269a84528.png)

If you run Loki on different server, then those link should be updated accordingly.

Now we are ready to browse Loki data in grafana. Go to explore:

![image](https://us1.discourse-cdn.com/flex023/uploads/cardano/original/3X/5/b/5b08e4e0f1cb9a1fadbba1aa7db503747f723e81.png)

Select Loki:

![image](https://us1.discourse-cdn.com/flex023/uploads/cardano/original/3X/c/8/c8effb424668ddbb2a21c794cc8026a201604eee.png)

Click on Log browser:

![image](https://us1.discourse-cdn.com/flex023/uploads/cardano/original/3X/2/e/2ec66e6c4fccf0839dc66c1247cc6edcaf96675c.png)

There you should be able to see different labels. In my case it is compose\_projects and others, compose\_projects are my docker container names, with promtail it can be configured inside config file I believe, but in any case you must have something there.

![image](https://us1.discourse-cdn.com/flex023/uploads/cardano/original/3X/5/b/5b9dc50f0b017fc3c86f428c3fabe9699f5a012a.png)

If we select specific label in our case compose\_project =\> test-relay1

![image](https://us1.discourse-cdn.com/flex023/uploads/cardano/original/3X/3/5/35edadc573c676874c1bef602ce00e361110cc34.png)

If we press show logs, you should be able to see all logs from that box:

 ![image](https://us1.discourse-cdn.com/flex023/uploads/cardano/original/3X/2/c/2c2646609fb8abde2fb7e85fb1740e87888307cc.png)

So now you should be able to see all your logs in “explore” menu, if you like you can add that screen of logs to dashboard, if you press “add to dashboard” button on right top corner:

 ![image](https://us1.discourse-cdn.com/flex023/uploads/cardano/original/3X/4/c/4c5953e82a96143580d7c41c3c09c52d6fc432af.png)

I personally created one dashboard with 4 windows with logs from all my nodes, as shown here:

 ![image](https://us1.discourse-cdn.com/flex023/uploads/cardano/original/3X/2/6/266e28de55f8b31e7154b2201a1e2f6cd8b264bf.png)

To search in all logs simultaneously you can do it via “explore” or you can add following dashboard to your grafana:

> <https://gist.github.com/os11k/ffcc2a41862a8c35db7a30fd7f13ef09>

Keep in mind in my example dashboard cardano nodes has label named “compose\_project”, so if your nodes has different labels inside loki just substitute in above file word “compose\_project” to one you used. Your labels you can find in “explore” of grafana like here:

![image](https://us1.discourse-cdn.com/flex023/uploads/cardano/original/3X/9/5/95951c906ba37d205a6052d6ed757cb8efc76422.png)

At the end you should be able to have following dashboard:

 ![image](https://us1.discourse-cdn.com/flex023/uploads/cardano/original/3X/9/8/98ad5ec1a873d43f873fede8c06e42a451521964.png)

In “compose\_project” you can select nodes where you can search for logs and String match field is what you are looking for. For example I have p2p on my test pool, let search on relays, when peer status changed from Hot to Cold:

 ![image](https://us1.discourse-cdn.com/flex023/uploads/cardano/original/3X/5/1/51e2d1c63db701967c85608e713fd7282b4cfa38.png)

So that seems pretty nice working and no need to go login via SSH to your nodes and go through logs.

P.S.

In my github repository I used almost default Loki config file, only changes were directories where we store data I changed `/tmp` to `/loki`

I would like to add that currently latest versions of Grafana has very nice alerting out of the box, so called unified alerting. So if any of you are using some old Grafana versions, it is worth to consider to move to new version and maybe even a docker based Grafana, what will allow you to send alerts to telegram slack and etc, without need alert manager and if you will choice docker then updates of Grafana will become much easier. First I tried to use 8.3.3 version, what I consider quite new, but unfortunately I was not able to make unified alerting work with it, basically sending alerts from Grafana to me via Telegram, so I updated till 8.5.0(what is just one line code change in docker-compose) and it worked well.

Recommended resources:

> **[How To Forward Logs to Grafana Loki using Promtail | ComputingForGeeks](https://computingforgeeks.com/forward-logs-to-grafana-loki-using-promtail/)**
>
> Grafana Loki is a tool that gives you a panel for indexing of your systems' logs and visualizing them on a dashboard. Grafana Loki does not index the contents

> **[Installation | Grafana Loki documentation](https://grafana.com/docs/loki/latest/installation/)**
>
> Installation Installation methods Instructions for different methods of installing Loki and Promtail.
> Install using Tanka (recommended) Install using Helm Install through Docker or Docker Compose Install and run locally Install from source General...

[https://levelup.gitconnected.com/loki-installation-in-ubuntu-2eb8407de291](https://levelup.gitconnected.com/loki-installation-in-ubuntu-2eb8407de291)

[![](https://us1.discourse-cdn.com/flex023/uploads/cardano/original/3X/3/b/3bf65ba15ae6f7cf580480fca63c505a0eacdd7c.jpeg "Grafana Loki querying basics, log based metrics and setting alerts on logs") ](https://www.youtube.com/watch?v=VEGYgPiAazk)

[![](https://us1.discourse-cdn.com/flex023/uploads/cardano/original/3X/7/7/77aa3e4d48959defa4441f994c486ec8b4994589.jpeg "Log based metrics and alerting in Grafana Loki") ](https://www.youtube.com/watch?v=BvgLIsWNV-0)

[![](https://us1.discourse-cdn.com/flex023/uploads/cardano/original/3X/d/8/d8fa95f8cc19cded208b200b800ae92c79314818.jpeg "Unified Alerting Grafana 8 | Prometheus | Victoria | Telegraf | Notifications | Alert Templating") ](https://www.youtube.com/watch?v=UtmmhLraSnE)

---

<div class="post-metadata">

**Author:** ![BlocksWell](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.cardano.org/blockswell/32/45488_2.png) [@BlocksWell](https://forum.cardano.org/u/BlocksWell)\
**Post date:** [9 May 2022 03:13 UTC](https://forum.cardano.org/t/grafana-loki-as-centralized-log-solutions-for-pool-operators/100824/2 "2022-05-09T03:13:40Z")

</div>

Thank you for taking the time to write up this guide! 👍

---

<div class="post-metadata">

**Author:** ![adatainment](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.cardano.org/adatainment/32/81481_2.png) [@adatainment](https://forum.cardano.org/u/adatainment)\
**Post date:** [5 October 2022 11:41 UTC](https://forum.cardano.org/t/grafana-loki-as-centralized-log-solutions-for-pool-operators/100824/3 "2022-10-05T11:41:09Z")

</div>

@os11k Should we bring this into the stake pool category on [developers.cardano.org](http://developers.cardano.org)?

---

<div class="post-metadata">

**Author:** ![os11k](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.cardano.org/os11k/32/43985_2.png) [@os11k](https://forum.cardano.org/u/os11k)\
**Post date:** [5 October 2022 12:00 UTC](https://forum.cardano.org/t/grafana-loki-as-centralized-log-solutions-for-pool-operators/100824/4 "2022-10-05T12:00:34Z")

</div>

Yes, we should. I actually raised an issue if it is ok to put it there, but if you think it is ok, I will work on this on some weekends.

Cheers!

---

<div class="post-metadata">

**Author:** ![COSDpool](https://sea1.discourse-cdn.com/flex023/user_avatar/forum.cardano.org/cosdpool/32/21482_2.png) [@COSDpool](https://forum.cardano.org/u/COSDpool)\
**Post date:** [5 October 2022 12:35 UTC](https://forum.cardano.org/t/grafana-loki-as-centralized-log-solutions-for-pool-operators/100824/5 "2022-10-05T12:35:10Z")

</div>

> [@os11k](#):
>
> I actually raised an issue if it is ok to put it there

I only found this issue now (was posted before I started following the Dev Portal repo) and I think it’s a great idea to put it there 🤓 [developer-portal#616](https://github.com/cardano-foundation/developer-portal/issues/616)
