1.35.3 needs more heap memory

Yesterday I upgraded to Vasil 1.35.3 and today I was surprised how much memory it graps.
I run 1.34.1 with 19.9GB heap memory allocation over weeks. Now the producer needs 22.3GB.
The relays are just fine. I couldn’t find any differences. They have leveled off at 10GB as before.

What is your experience?

Relay → GHCRTSOPTS=“-N -T -I0 -A16m --disable-delayed-os-memory-return --nonmoving-gc”
Producer → GHCRTSOPTS=“-N”

pool1

My two relays and BP are still using normal amounts of RAM. Have you restarted since you upgraded? Any other services running?

What is your normal amount of RAM?

I restartet the producer and now it is at 10.7GB after 3 hours uptime.
pool1

Let’s see how it behaves after a while. It looks very good for now.

On the hardware server is just the producer and nothing else.

not running/ran cncli leaderlog? This service will double the ram consumption while is checking

Hi! What exactly metrics you are referring to?

This is my rts_gc_current_bytes_used/(1024*1024):

Pike is were I ran cncli leaderlog. This is BP.

I didn’t run cncli leaderlog. The high allocation was after the upgrade to 1.35.3 and the following block verification process. After a restart everything looks good for now.

For metrics I use:

  • cardano_node_metrics_RTS_gcLiveBytes_int (light blue line) / Grafana unit → bytes(IEC)
  • cardano_node_metrics_RTS_gcHeapBytes_int (dark blue line) / GRafana unit → bytes(IEC)

pool1

My light blue line looks similar to your green graph. :+1:

The dark blue line is slightly over 11GB which I also see in top:
pool1

Thanks to the restart, everything looks fine now.

The BP ist running on a hardwareserver - 12 core Intel i7 3.33 GHz with 24 GB Ram.

I’m out of my depth with all this. But I do understand that each successful implementation of node 1.35.3 brings us closer to the launch of Vasil.

1 Like

Hey @FrankR. If you are looking for a stake pool to stake some ADAs I would appreciate your support at the Hamster Stake Pool [HAMDA]. We are well prepared for Vasil :slight_smile: