Quote from: xkcdd on September 23, 2021, 03:02:37 PM

Quote from: gadado2 on September 22, 2021, 03:03:49 PM

Ah to break the silence... here just a quick update from my server auto-reset experience. Wink

That works perfect!!! No runaway CPU usage anymore.
CPU usage stays at around 10% on all nodes.
Instead weekly I choose to go more agressive with a daily reset.

The reason for daily: That way I don't have to check daily because it will reset anyway before any Pose can ever accumulate till up to a kick out.

Yes with that frequency it might hit 1 or 2 server reset quorum/pose collition a year a node..(guesswork. it could be caclulated) but that's nothing to worry about.
Pose is only a problem if your node does stay off for a few hours.
Observed no Pose till now or missed it in case there was already a collition.

I am very happy with that solution. Grin

Man, this is great, Grin

can you keep us posted on this? I might add it to the dash masternode zeus since it does seem like the dashd has a memory leak when left running for weeks at a time.

Well I have news but no good ones.
This morning on dashninja I saw that one of my MNs got 3800 Pose.
So I asumed by chance such a collision happened and checked the last pay-out block and when it was not within a sheduled Server-Reset. Some hours away.

In fact I did a calculation lately and came to a number of mayb 1 collision in 10 years a MN for a daly reset but then I am no more that strong in mathe as I was once and my calculation can be off. Still would be carefully and go with maybe 1 collision a year.

Anyway vultur showed CPU 100% and strange enough I saw a significant Disk I/O increase 11 hours bevor the cpu spike that keept stable on a high level but NO CPU load increase (still 10%) and NO networktraffic increase.

The Daily-Server-Reset was due intwo hours so I waited.
Dash-Ninja showed a decrasing POSe.. which is strange given the CPU was 100% load equals the node is not responding anymore.

Now even before the Server-Reset happened the Node became delisted from Dash-Ninja which useually indicates it became POSe banned (thought that's happening when it gets closer to the server count of 4800)

I waited 2 hours but the CPU stayed 100% and what's killing me is the Server-Reset did not happen!
I am 100% sure that the daily-resets are and were executed every day since I checked about them. But it did not happen while the CPU was on 100% load.

That's a killer because I have hopped that it autorecovers in such a case due to the daily-reset.
Also I checked with a dash-client and yes the node is pose banned with a score of 4800+ and it happened within 3 to 4 hours!!!

My previous experience is that you have at least 1 day time to do a reset before getting banned.

I asked vultr very specific about if there was a maintenance or something causeing this server go down and they said no.

This is frustrating.
I do not understand how the whole system can go down that not even that system contrab thing works anymore. Isn't that a system always work thingy?

Ah yes vultr gave my commands for postmortem analysis but I guess at that point it is better if I resolve and recreate the node with completly new collateral, BLS, IP and everything.