I'm wondering if it has something to do with the pool.  I switched from 2Miners to MiningPoolHub and decided to use 1.63 and get some samples of the miner stopping reporting, but it has been running 2 days and is still reporting stats properly.
Also, I decided to try putting the 2060S back into the TBM list (since it was conflicting with the 3090 and generating a lot of rejected shares when they were all running together), and I haven't had any rejected shares for 20 hours.   

Is there some significant difference between MPH and 2Miners that would cause such different behaviour?

Exactly like that.  Just Jobs and Submitted shares being reported.

The terminal stops reporting the card details, so there's just a long list of accepted jobs and submitted jobs but no 60 second block display of power usage or hash rate.  It ~seems~ random, but there may be some other issue that isn't being logged, like the momentary GPU glitch in the Nvidia driver which stops it from reporting the Memory Temp and fan speed.  Whatever it is, it works fine in 1.62, just not in 1.63.

It's been pretty dead around here for a while, but I just wanted to raise a slight issue that seems to happen randomly.

On the latest version in HiveOS, the miner will stop reporting GPU stats after a while.  It's still mining and providing solutions to the pool, but I have no idea what the cards are doing in terms of power usage and thermals (which is becoming more of an issue as summer temperatures start making it harder to keep the cards at a reasonable temperature.

You probably want to put the intensity to about 4000 on the A2000 (which has the 192bit memory bus) and 2800 on the A4000 (which has the 256bit memory bus).

It seems that no matter if I use ./start.sh "0" or "1", it will always show "All devices selected." and even when I have my 3090Ti idle (not mining) and set as the output device to the monitor, it will eventually corrupt the monitor and crash the server.

Getting a runtime error on HiveOS:
./TheBlackPill.x: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by ./TheBlackPill.x)
./TheBlackPill.x: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by ./TheBlackPill.x)

What version of Linux did you compile this on?  HiveOS is still using Ubuntu Version 16.04 LTS and I can't seem to upgrade the system to use any GLIBC_ past 2.29. 

There are probably some third party ppa: repos that would fix it, but I'm not comfortable using unverified external ppa: repos published by some random on the net.

Alternatively I could probably compile/build the GLIBC_## packages and go through the process of relinking things, but it might be at the expense of disabling part of the main kernel, since the libc is so tightly integrated with the kernel, any changes can mess a lot of things up -- and besides, as soon as an upgrade comes along that updates the kernel for the HiveOS distro, I'll have to do it all over again.

**** gcc-9 and using the ppa: for ubuntu-toolchain-r/r-test is where I can get GLIBC_2.29 functioning, installing gcc-10 -> gcc-11 don't make a difference and my entire package repo is up to date and upgraded.
 

So I ran this in Windows to see how it compares to other unlocks and it seems to perform very well, and a full 100% unlock for the 3070Ti card that I have.  There's some variability in the miner reported rate, but the pool reported rate is showing a normal range from 80-90 MH/s at 2miners and with their 30 minute rolling window.



I'll keep it running for a while longer and see how it maintains the unlock over the long term; so far so good!

If core drops when you raise the mem clock - you run into the powerlimit - increase it.
I have seen similiar things on 1 of my 2060S, all of this cards run at 45mh, on 1 card, if i raise the mem - the coreclocks drops to 750. For me it makes no sense because the card only use around 120w, but the powerlimit in the bios is a little bit lower than all other cards and i can´t increase the pl over 100% it is kept there.
[/quote]

When the core drops to 750 on that one card, if you're using HiveOS or linux, run "nvidia-info" and it will report if the card is being throttled and the reason for it.  If the card isn't reporting any throttle but you can't manually push it above 750 MHz then it most likely is a power problem and that particular card is sensing decreased amperage which will cause it to limit without reporting. I've seen 1000 watt PSUs that can only supply 700 watts consistently and will run into issues when loaded beyond that -- it can run 1000 watts transiently to provide power for a rapid spike load, but just not for more than a few seconds.

Update: 
I did get a timeout on the AMD GPU about 5 1/2 hours after a reboot.  Cards crashed (still showed as hashing for about 4 minutes, but the calculated Hashrate/W jumped up):

00:05:26 [2022-05-07 17:36:30.130] GPU5    3090     CUDA    2       6800    43/0    95      1500    11275   333   
00:05:26 [2022-05-07 17:36:30.130]                                                                          919   
00:05:26 [2022-05-07 17:36:30.130]
00:05:26 [2022-05-07 17:36:30.130] ID      BOARD    HASHRATE/W    HASHRATE      AVERAGE       SHARES       RATE   
00:05:26 [2022-05-07 17:36:30.130] GPU0    gfx1030  1947.58 kH/W  66.22 MH/s    66.14 MH/s    133/0/0      100.00%
00:05:26 [2022-05-07 17:36:30.130] GPU1    gfx1030  2285.92 kH/W  66.29 MH/s    66.29 MH/s    128/0/0      100.00%
00:05:26 [2022-05-07 17:36:30.130] GPU2    gfx1030  2269.47 kH/W  65.81 MH/s    65.82 MH/s    144/0/1      100.00%
00:05:26 [2022-05-07 17:36:30.130] GPU3    gfx1030  1882.83 kH/W  65.90 MH/s    65.90 MH/s    135/0/0      100.00%
00:05:26 [2022-05-07 17:36:30.130] GPU4    gfx1030  2456.14 kH/W  66.32 MH/s    66.28 MH/s    137/0/0      100.00%
00:05:26 [2022-05-07 17:36:30.130] GPU5    3090     399.57 kH/W   133.06 MH/s   133.05 MH/s   296/0/0      100.00%
00:05:26 [2022-05-07 17:36:30.130]                  1873.58 kH/W  463.59 MH/s   463.47 MH/s   973/0/1      100.00%
00:05:26 [2022-05-07 17:36:30.130]                                                                                 
00:05:26 [2022-05-07 17:36:30.130] SHARES PER MINUTE: 2.98            POOL HASHRATE: 434.08 MH/s


Then they posted the failure:


00:05:29 [2022-05-07 17:39:30.159] GPU2: has timed out, trying to restart..
00:05:29 [2022-05-07 17:39:30.159] GPU4: has timed out, trying to restart..
00:05:29 [2022-05-07 17:39:30.160]
00:05:29 [2022-05-07 17:39:30.160]
00:05:29 [2022-05-07 17:39:30.160] us-eth.2miners.com (ethash)    PING: 36ms    DIFFICULTY: 2.03     EPOCH: 491
00:05:29 [2022-05-07 17:39:30.160]                                                                                 
00:05:29 [2022-05-07 17:39:30.160]
00:05:29 [2022-05-07 17:39:30.160] ID      BOARD    TYPE    KERN    XINT    TEMP    FAN     CORE    MEM     WATT   
00:05:29 [2022-05-07 17:39:30.160] GPU0    gfx1030  OpenCL  0       693     27/0    89      1500    1069    114   
00:05:29 [2022-05-07 17:39:30.160] GPU1    gfx1030  OpenCL  0       693     25/0    89      1500    1069    104   
00:05:29 [2022-05-07 17:39:30.160] GPU2    gfx1030  OpenCL  0       949     26/0    89      1500    1059    120   
00:05:29 [2022-05-07 17:39:30.160] GPU3    gfx1030  OpenCL  0       660     24/0    89      1650    1069    124   
00:05:29 [2022-05-07 17:39:30.160] GPU4    gfx1030  OpenCL  0       963     25/0    89      1500    1069    109   
00:05:29 [2022-05-07 17:39:30.160] GPU5    3090     CUDA    2       6800    42/0    95      1500    11275   333   
00:05:29 [2022-05-07 17:39:30.160]                                                                          904   
00:05:29 [2022-05-07 17:39:30.160]
00:05:29 [2022-05-07 17:39:30.160] ID      BOARD    HASHRATE/W    HASHRATE      AVERAGE       SHARES       RATE   
00:05:29 [2022-05-07 17:39:30.160] GPU0    gfx1030  1947.58 kH/W  66.22 MH/s    66.14 MH/s    133/0/0      100.00%
00:05:29 [2022-05-07 17:39:30.160] GPU1    gfx1030  2367.56 kH/W  66.29 MH/s    66.29 MH/s    128/0/0      100.00%
00:05:29 [2022-05-07 17:39:30.160] GPU2    gfx1030  0.00 H/W      0.00 H/s      0.00 H/s      144/0/1      100.00%
00:05:29 [2022-05-07 17:39:30.160] GPU3    gfx1030  1830.53 kH/W  65.90 MH/s    65.90 MH/s    135/0/0      100.00%
00:05:29 [2022-05-07 17:39:30.160] GPU4    gfx1030  0.00 H/W      0.00 H/s      0.00 H/s      137/0/0      100.00%
00:05:29 [2022-05-07 17:39:30.160] GPU5    3090     399.57 kH/W   133.06 MH/s   133.05 MH/s   297/0/0      100.00%
00:05:29 [2022-05-07 17:39:30.160]                  1090.87 kH/W  331.46 MH/s   331.37 MH/s   974/0/1      100.00%
00:05:29 [2022-05-07 17:39:30.160]                                                                                 
00:05:29 [2022-05-07 17:39:30.160] SHARES PER MINUTE: 2.96            POOL HASHRATE: 430.56 MH/s
00:05:29 [2022-05-07 17:39:30.160]
00:05:29 [2022-05-07 17:39:33.624] Job d52cb
00:05:29 [2022-05-07 17:39:40.329] GPU5 Solution found in 0.19s (Submit every 21.29 seconds)
00:05:29 [2022-05-07 17:39:40.329] Job d52cc
00:05:29 [2022-05-07 17:39:40.361] Accepted (975/975/1) Reply: 32ms
00:05:29 [2022-05-07 17:39:47.034] Job d52ce
00:05:29 [2022-05-07 17:39:49.273] Job d52d1
00:05:29 [2022-05-07 17:39:51.508] Job d52d2
00:05:29 [2022-05-07 17:39:58.213] Job d52d4
00:05:29 [2022-05-07 17:40:04.919] Job d52d6
00:05:29 [2022-05-07 17:40:11.624] Job d52d9
00:05:30 [2022-05-07 17:40:13.859] Job d52dc
00:05:30 [2022-05-07 17:40:18.329] GPU5 Solution found in 4.60s (Submit every 21.31 seconds)
00:05:30 [2022-05-07 17:40:18.369] Accepted (976/976/1) Reply: 40ms
00:05:30 [2022-05-07 17:40:20.564] Job d52de
00:05:30 [2022-05-07 17:40:29.504] Job d52df
00:05:30 [2022-05-07 17:40:30.163] GPU0: has timed out, trying to restart..
00:05:30 [2022-05-07 17:40:30.163] GPU1: has timed out, trying to restart..
00:05:30 [2022-05-07 17:40:30.163] GPU3: has timed out, trying to restart..
00:05:30 [2022-05-07 17:40:30.163]



Then eventually the miner just stopped


00:05:32 [2022-05-07 17:42:30.171] us-eth.2miners.com (ethash)    PING: 36ms    DIFFICULTY: 2.03     EPOCH: 491
00:05:32 [2022-05-07 17:42:30.171]                                                                                 
00:05:32 [2022-05-07 17:42:30.171]
00:05:32 [2022-05-07 17:42:30.171] ID      BOARD    TYPE    KERN    XINT    TEMP    FAN     CORE    MEM     WATT   
00:05:32 [2022-05-07 17:42:30.171] GPU0    gfx1030  OpenCL  0       693     26/0    89      1500    1069    111   
00:05:32 [2022-05-07 17:42:30.171] GPU1    gfx1030  OpenCL  0       693     26/0    89      1500    1069    102   
00:05:32 [2022-05-07 17:42:30.171] GPU2    gfx1030  OpenCL  0       949     26/0    89      1500    1059    117   
00:05:32 [2022-05-07 17:42:30.171] GPU3    gfx1030  OpenCL  0       660     24/0    89      1650    1069    121   
00:05:32 [2022-05-07 17:42:30.171] GPU4    gfx1030  OpenCL  0       963     25/0    89      1500    1069    106   
00:05:32 [2022-05-07 17:42:30.171] GPU5    3090     CUDA    2       6800    43/0    95      1500    11275   333   
00:05:32 [2022-05-07 17:42:30.171]                                                                          890   
00:05:32 [2022-05-07 17:42:30.171]
00:05:32 [2022-05-07 17:42:30.171] ID      BOARD    HASHRATE/W    HASHRATE      AVERAGE       SHARES       RATE   
00:05:32 [2022-05-07 17:42:30.171] GPU0    gfx1030  0.00 H/W      0.00 H/s      0.00 H/s      133/0/0      100.00%
00:05:32 [2022-05-07 17:42:30.171] GPU1    gfx1030  0.00 H/W      0.00 H/s      0.00 H/s      128/0/0      100.00%
00:05:32 [2022-05-07 17:42:30.171] GPU2    gfx1030  0.00 H/W      0.00 H/s      0.00 H/s      144/0/1      100.00%
00:05:32 [2022-05-07 17:42:30.171] GPU3    gfx1030  0.00 H/W      0.00 H/s      0.00 H/s      135/0/0      100.00%
00:05:32 [2022-05-07 17:42:30.171] GPU4    gfx1030  0.00 H/W      0.00 H/s      0.00 H/s      137/0/0      100.00%
00:05:32 [2022-05-07 17:42:30.171] GPU5    3090     399.58 kH/W   133.06 MH/s   133.05 MH/s   299/0/0      100.00%
00:05:32 [2022-05-07 17:42:30.171]                  66.60 kH/W    133.06 MH/s   133.05 MH/s   976/0/1      100.00%
00:05:32 [2022-05-07 17:42:30.171]                                                                                 
00:05:32 [2022-05-07 17:42:30.171] SHARES PER MINUTE: 2.94            POOL HASHRATE: 427.55 MH/s
00:05:32 [2022-05-07 17:42:30.171]
root@red-ether:/var/log/miner/teamblackminer# cat teamblackminer.2.log |grep -i -B 15 -A 250 '1873.58' > ~/StillTimimgOut_163.txt

I've been running 1.63 with different cards running different intensities and the rig has been stable, no GPU timeouts. 

Overall average hashrate on the AMD cards is 62 MH/s, reported average is 65 Mh/s, so in general the reported and actuals are close -- how the peaks and valleys align with the 2miners reporting window has a more significant effect on the hashrate than anything -- I wish they had a more controllable window, but the ETH earned per day is aligning with the values coming out of various hashrate profit calculators based on the miner reported values.
 
I do notice that once in a while, one of the cards will drop its hashrate for a minute from 65-66 MH/s to 40-something and go back to its previous rate a minute later, but that's not really an issue, just some funny behaviour. 

I'm super impressed with how the 3090Ti is working with the miner software.  I am running 9992 intensity with zero stale or invalid hashes -- I set it to autotune and notice that it seems to switch between kernel 2 and kernel 13, any idea why it's only those two kernels that are putting up 100's vs. 99's for the others (aside from kernel 0 which is like 70)?  Performance is 136-137 MH/s from the miner and 135-136 at the pool.  Temperatures on the 3090Ti are great!!  probably the coolest card in all my rigs running no hotter than 52/58 Degrees (Core/Memory) @ 337 watts.
 

We need to shorten the time between when a card 'times out' and when the miner restarts.  I've never seen a card recover from a timeout and often times 3-4 will all timeout at the same time which drops my average hashrate significantly, which also affects my poolside share of a given round.

I mean ideally the AMD cards wouldn't just randomly time out, but if they do, I'd rather just have the process restart than wait for the inevitable crash while hashing at a low rate.

I posted an issue report in Github; will update as I find out more.  But it seems to only be happening across my Nvidia cards.

I don't know if it's the extra card or version 1.62 CUDA 11.5, but I'm getting a LOT of duplicate share rejects from 2miners now.

I've updated HiveOS to the latest kernel and all the other applications so that I'm running the full-spec HiveOS release (standard libraries and drivers), and I'm still getting random timeouts on the AMD cards.  I'm getting the same timeouts on windows using a 3070Ti and 6900XT with 1000 watt platinum PSU, so I can be pretty certain it's not a power problem, unless there's some issue in the power being supplied from the mains (I should probably check to make sure I'm getting consistent voltage and it's not browning out -- but it's not always the same cards at the same time -- but the wiring is kinda weird since it used to be 220/240V and was converted to 120V).

I'm running the 3090 Ti now too.  It takes different timings on the memory (higher stock values than the 3090), but can be run up to 11851+ MHz and still be stable (although this is +2700 in HiveOS offset which wouldn't break 10000 MHz on the 3090).
As for speed -- it's faster for sure, and with some tweaking of the clocks and voltages it's not any more power hungry than the 3090 in software reporting anyway).  Right now it's running between 138 and 155 MH/s, but the memory temperatures are WAY WAY better than the 3090 -- 58 vs. 84 Celcius at 328 watts for both cards, so they're much better for a farm since there's no memory on the back-side of the card to cool.

I'll continue to tweak the 3090Ti, but I'm currently running at 85% amperage capacity of my wall power, which is not advisable, although as I said, the original wiring was rated for 220/240, so it's more just an issue with the breaker possibly tripping if I try and push too much more power through the circuit.
 

I'm running the 5.11.1001 AMD drivers that HiveOS provides -- I've messed around with updating the AMD drivers before and while it's possible -- there's always something that breaks because of it.  I added 48G of swap and am running the code supplied by hive for version 1.61, so all the files are what is found within the distro.

I punched up the voltages and switched mining over to MiningPoolHub to see if there was any difference and I still had an AMD GPU timeout on me... Next step is to drop the intensity values on a couple of cards to see if they remain functional and only the higher intensity cards fail.

I'm also getting a 3090Ti delivered in a couple of days -- I'm curious to see how much I can crank out of that card!

Just as an update / FYI:

I'm still getting GPU timeouts on my AMD cards when mining on 2Miners.  The mining time is quite random and can be anywhere from 1-24 hours with 1 - 3 cards timing out at the same time in the mixed AMD/NVidia rig (4 AMD cards).  My AMD card running in windows will still crash occasionally, but it takes a while longer for it to happen (although the card quite often drops power to 15 watts in windows so there's a different pattern of disfunction which may hide the mining issue).

I waited on sending an update until I had updated the HiveOS kernel to the latest version, and it's still crashing.  Although, the new AMD drivers / Linux Kernel does seem to improve the performance on the AMD cards, or at least reduces the variability in performance, with the dips in hashrate being higher than before.

The new Linux kernel is letting the miner run CUDA 11.5 at a consistent 130+ MH/s, although with the temperatures climbing, I'm going to have to keep an eye on thermals vs. pure hash power; right now I can push the card to 134 - 137 MH/s average at the pool, but Memory Temps are creeping up into the 80's with ambient being 12-15 Celsius.

Assuming it's Windows -- use the Task Scheduler to create a task that is triggered on the computer startup - before user logon, with full admin permissions.  This will ensure that there are no user-specific apps that are running.  The caveat here is that if you're dependent on any overclocking software like MSI afterburner or Precision X1, you need to either set up scripts that execute these apps before you start the miner, or have the apps execute from within the batch file before running the miner.  I have to execute X1 to get my 3rd fan spinning on my EVGA cards, but I have the script execute and then kill the application after a few minutes so it's not always running in the background taking up resources (X1 is a resource hog).

Here's another example:

00:05:52 [2022-03-14 08:05:45.276] GPU1    gfx1030  575.87 kH/W   66.23 MH/s    66.22 MH/s    156/0/0      100.00%
00:05:52 [2022-03-14 08:05:50.586] GPU1 hashtime 20c
00:05:53 [2022-03-14 08:06:43.051] GPU1 hashtime 20c
00:05:53 [2022-03-14 08:07:12.860] GPU1: 1d
00:05:53 [2022-03-14 08:07:12.860] GPU1    gfx1030  575.85 kH/W   66.22 MH/s    66.22 MH/s    157/0/0      100.00%
00:05:54 [2022-03-14 08:07:35.516] GPU1 hashtime 20c
00:05:55 [2022-03-14 08:08:27.979] GPU1 hashtime 20c
00:05:55 [2022-03-14 08:08:47.132] GPU1    gfx1030  575.86 kH/W   66.22 MH/s    66.22 MH/s    157/0/0      100.00%
00:05:56 [2022-03-14 08:10:05.404] GPU1: 61
00:05:56 [2022-03-14 08:10:05.404] GPU1    gfx1030  2136.27 kH/W  66.22 MH/s    66.22 MH/s    157/0/0      100.00%
00:05:58 [2022-03-14 08:11:21.980] GPU1: ae
00:05:58 [2022-03-14 08:11:21.980] GPU1    gfx1030  2136.27 kH/W  66.22 MH/s    66.22 MH/s    157/0/0      100.00%
00:05:59 [2022-03-14 08:12:39.996] GPU1: fc
00:05:59 [2022-03-14 08:12:39.996] GPU1    gfx1030  2136.27 kH/W  66.22 MH/s    66.22 MH/s    157/0/0      100.00%
00:06:00 [2022-03-14 08:13:52.028] GPU1: 144
00:06:00 [2022-03-14 08:13:52.028] GPU1: has timed out, trying to restart..
00:06:00 [2022-03-14 08:13:52.028] GPU1    gfx1030  0.00 H/W      0.00 H/s      0.00 H/s      157/0/0      100.00%
00:06:01 [2022-03-14 08:15:10.236] GPU1: 192
00:06:01 [2022-03-14 08:15:10.236] GPU1    gfx1030  0.00 H/W      0.00 H/s      0.00 H/s      157/0/0      100.00%
00:06:03 [2022-03-14 08:16:26.844] GPU1: 1de
00:06:03 [2022-03-14 08:16:26.844] GPU1    gfx1030  0.00 H/W      0.00 H/s      0.00 H/s      157/0/0      100.00%
00:06:04 [2022-03-14 08:17:42.451] GPU1: 22a
00:06:04 [2022-03-14 08:17:42.451] GPU1    gfx1030  0.00 H/W      0.00 H/s      0.00 H/s      157/0/0      100.00%
00:06:05 [2022-03-14 08:19:01.088] GPU1: 279
00:06:05 [2022-03-14 08:19:01.091] GPU1    gfx1030  0.00 H/W      0.00 H/s      0.00 H/s      157/0/0      100.00%
00:06:07 [2022-03-14 08:20:19.228] GPU1: 2c7
00:06:07 [2022-03-14 08:20:19.228] GPU1    gfx1030  0.00 H/W      0.00 H/s      0.00 H/s      157/0/0      100.00%
00:06:08 [2022-03-14 08:21:33.532] GPU1: 311
00:06:08 [2022-03-14 08:21:33.532] GPU1    gfx1030  0.00 H/W      0.00 H/s      0.00 H/s      157/0/0      100.00%
00:06:09 [2022-03-14 08:22:47.305] GPU1: 35b
00:06:09 [2022-03-14 08:22:47.305] GPU1    gfx1030  0.00 H/W      0.00 H/s      0.00 H/s      157/0/0      100.00%
00:06:10 [2022-03-14 08:24:04.700] GPU1: 3a8
00:06:10 [2022-03-14 08:24:04.700] GPU1    gfx1030  0.00 H/W      0.00 H/s      0.00 H/s      157/0/0      100.00%
00:06:12 [2022-03-14 08:25:20.220] GPU1: 3f4
00:06:12 [2022-03-14 08:25:20.220] GPU1: GPU thread crashed exiting program.

The card failed at some point between 8:10 and 8:11 (you can tell by the increase in KH/W), three minutes later the card was officially pronounced dead and a restart was attempted, and 12 minutes after the restart was attempted, the card finally caused a restart of the miner.  I'm not sure what the debug output is supposed to identify, but the last hashtime value was posted at 8:07/8:08. 

The GPU1's stats just before the crash were stable and no values were out of anywhere near critical:

00:05:55 [2022-03-14 08:08:47.132] GPU1    gfx1030  OpenCL  0       693     27/0    80      1370    1069    115
00:05:55 [2022-03-14 08:08:47.132] GPU1    gfx1030  575.86 kH/W   66.22 MH/s    66.22 MH/s    157/0/0      100.00%

Then, just after:

00:05:56 [2022-03-14 08:10:05.404] GPU1    gfx1030  OpenCL  0       693     20/0    80      1370    1069    114 
00:05:56 [2022-03-14 08:10:05.404] GPU1    gfx1030  2136.27 kH/W  66.22 MH/s    66.22 MH/s    157/0/0      100.00%