Quote from: todxx on August 23, 2019, 01:31:23 AM

That is rather odd and should definitely not be happening.

I've read log files in detail and found relation:
after one or more GPUs stops giving shares, "total table" stops appearing further. In below example GPU3 and GPU4 are lost, so the blue table is not shown anymore, but rig continues hashing without two GPUs until manual restart. I'm sure you can catch totals disappearing in code and react as GPU watchdog shoud. I could write external tool for log monitoring, but it's a hack.

Quote

[2019-08-27 23:10:00] Pool xmr-eu1.nanopool.org share accepted. (GPU4) (a:47526 r:853) (139 ms)
[2019-08-27 23:10:03] Pool xmr-eu1.nanopool.org share accepted. (GPU7) (a:47527 r:853) (133 ms)
[2019-08-27 23:10:09] Pool xmr-eu1.nanopool.org share accepted. (GPU7) (a:47528 r:853) (151 ms)
[2019-08-27 23:10:10] Stats Uptime: 4 days, 05:59:51
[2019-08-27 23:10:10] GPU 0 [59C, fan 13%] cnr: 2.036kh/s, avg 2.034kh/s, pool 1.940kh/s a:5937 r:116 hw:0
[2019-08-27 23:10:10] GPU 1 [60C, fan 15%] cnr: 2.038kh/s, avg 2.036kh/s, pool 1.949kh/s a:5964 r:98 hw:0
[2019-08-27 23:10:10] GPU 2 [59C, fan 13%] cnr: 2.040kh/s, avg 2.039kh/s, pool 1.945kh/s a:5950 r:95 hw:0
[2019-08-27 23:10:10] GPU 3 [59C, fan 14%] cnr: 2.022kh/s, avg 2.037kh/s, pool 1.940kh/s a:5937 r:123 hw:0
[2019-08-27 23:10:10] GPU 4 [58C, fan 13%] cnr: 2.039kh/s, avg 2.037kh/s, pool 1.954kh/s a:5980 r:99 hw:0
[2019-08-27 23:10:10] GPU 5 [60C, fan 16%] cnr: 2.019kh/s, avg 2.033kh/s, pool 1.953kh/s a:5975 r:114 hw:0
[2019-08-27 23:10:10] GPU 6 [58C, fan 13%] cnr: 2.016kh/s, avg 2.035kh/s, pool 1.899kh/s a:5810 r:109 hw:0
[2019-08-27 23:10:10] GPU 7 [59C, fan 13%] cnr: 2.036kh/s, avg 2.034kh/s, pool 1.953kh/s a:5975 r:99 hw:0
[2019-08-27 23:10:10] Total cnr: 16.25kh/s, avg 16.29kh/s, pool 15.53kh/s a:47528 r:853 hw:0
[2019-08-27 23:10:14] Pool xmr-eu1.nanopool.org share accepted. (GPU0) (a:47529 r:853) (265 ms)
[2019-08-27 23:10:19] Pool xmr-eu1.nanopool.org share accepted. (GPU2) (a:47530 r:853) (201 ms)
[2019-08-27 23:10:26] Pool xmr-eu1.nanopool.org share accepted. (GPU3) (a:47531 r:853) (219 ms)
[2019-08-27 23:10:26] Pool xmr-eu1.nanopool.org share accepted. (GPU3) (a:47532 r:853) (127 ms)
[2019-08-27 23:10:29] Pool xmr-eu1.nanopool.org share accepted. (GPU7) (a:47533 r:853) (131 ms)
[2019-08-27 23:10:40] Stats Uptime: 4 days, 06:00:21
[2019-08-27 23:10:40] GPU 0 [59C, fan 14%] cnr: 2.036kh/s, avg 2.034kh/s, pool 1.940kh/s a:5938 r:116 hw:0
[2019-08-27 23:10:40] GPU 1 [60C, fan 15%] cnr: 2.039kh/s, avg 2.036kh/s, pool 1.949kh/s a:5964 r:98 hw:0
[2019-08-27 23:10:40] GPU 2 [60C, fan 14%] cnr: 2.040kh/s, avg 2.039kh/s, pool 1.945kh/s a:5951 r:95 hw:0
[2019-08-27 23:10:40] GPU 3 [59C, fan 13%] cnr: 2.026kh/s, avg 2.037kh/s, pool 1.941kh/s a:5939 r:123 hw:0
[2019-08-27 23:10:40] GPU 4 [60C, fan 12%] cnr: 2.039kh/s, avg 2.037kh/s, pool 1.954kh/s a:5980 r:99 hw:0
[2019-08-27 23:10:40] GPU 5 [59C, fan 16%] cnr: 2.020kh/s, avg 2.033kh/s, pool 1.953kh/s a:5975 r:114 hw:0
[2019-08-27 23:10:40] GPU 6 [61C, fan 13%] cnr: 2.015kh/s, avg 2.035kh/s, pool 1.899kh/s a:5810 r:109 hw:0
[2019-08-27 23:10:40] GPU 7 [59C, fan 14%] cnr: 2.036kh/s, avg 2.034kh/s, pool 1.953kh/s a:5976 r:99 hw:0
[2019-08-27 23:10:40] Total cnr: 16.25kh/s, avg 16.29kh/s, pool 15.53kh/s a:47533 r:853 hw:0
[2019-08-27 23:10:52] Pool xmr-eu1.nanopool.org share accepted. (GPU5) (a:47534 r:853) (134 ms)
[2019-08-27 23:10:57] Pool xmr-eu1.nanopool.org received new job. (job_id: 9111)
[2019-08-27 23:11:10] Stats Uptime: 4 days, 06:00:51
[2019-08-27 23:11:10] GPU 0 [59C, fan 13%] cnr: 2.036kh/s, avg 2.034kh/s, pool 1.940kh/s a:5938 r:116 hw:0
[2019-08-27 23:11:10] GPU 1 [57C, fan 15%] cnr: 2.038kh/s, avg 2.036kh/s, pool 1.949kh/s a:5964 r:98 hw:0
[2019-08-27 23:11:10] GPU 2 [59C, fan 14%] cnr: 2.040kh/s, avg 2.039kh/s, pool 1.945kh/s a:5951 r:95 hw:0
[2019-08-27 23:11:10] GPU 3 [59C, fan 14%] cnr: 2.024kh/s, avg 2.037kh/s, pool 1.941kh/s a:5939 r:123 hw:0
[2019-08-27 23:11:10] GPU 4 [60C, fan 13%] cnr: 2.039kh/s, avg 2.037kh/s, pool 1.954kh/s a:5980 r:99 hw:0
[2019-08-27 23:11:10] GPU 5 [59C, fan 16%] cnr: 2.021kh/s, avg 2.033kh/s, pool 1.953kh/s a:5976 r:114 hw:0
[2019-08-27 23:11:10] GPU 6 [61C, fan 14%] cnr: 2.016kh/s, avg 2.035kh/s, pool 1.899kh/s a:5810 r:109 hw:0
[2019-08-27 23:11:10] GPU 7 [59C, fan 15%] cnr: 2.036kh/s, avg 2.034kh/s, pool 1.953kh/s a:5976 r:99 hw:0
[2019-08-27 23:11:10] Total cnr: 16.25kh/s, avg 16.29kh/s, pool 15.53kh/s a:47534 r:853 hw:0
[2019-08-27 23:11:13] Pool xmr-eu1.nanopool.org share accepted. (GPU5) (a:47535 r:853) (178 ms)
[2019-08-27 23:11:14] Pool xmr-eu1.nanopool.org share accepted. (GPU2) (a:47536 r:853) (194 ms)
[2019-08-27 23:11:16] Pool xmr-eu1.nanopool.org share accepted. (GPU2) (a:47537 r:853) (139 ms)
[2019-08-27 23:11:19] Pool xmr-eu1.nanopool.org share accepted. (GPU5) (a:47538 r:853) (127 ms)
[2019-08-27 23:11:40] Stats Uptime: 4 days, 06:01:21
[2019-08-27 23:11:41] Pool xmr-eu1.nanopool.org share accepted. (GPU7) (a:47539 r:853) (132 ms)
[2019-08-27 23:11:44] Pool xmr-eu1.nanopool.org share accepted. (GPU7) (a:47540 r:853) (214 ms)
[2019-08-27 23:11:44] Pool xmr-eu1.nanopool.org share accepted. (GPU6) (a:47541 r:853) (174 ms)
[2019-08-27 23:11:49] Pool xmr-eu1.nanopool.org share accepted. (GPU0) (a:47542 r:853) (155 ms)
[2019-08-27 23:11:57] Pool xmr-eu1.nanopool.org received new job. (job_id: 9112)
[2019-08-27 23:12:03] Pool xmr-eu1.nanopool.org share accepted. (GPU2) (a:47543 r:853) (168 ms)
[2019-08-27 23:12:11] Pool xmr-eu1.nanopool.org share accepted. (GPU7) (a:47544 r:853) (203 ms)
[2019-08-27 23:12:19] Pool xmr-eu1.nanopool.org share accepted. (GPU2) (a:47545 r:853) (150 ms)
[2019-08-27 23:12:25] Pool xmr-eu1.nanopool.org share accepted. (GPU7) (a:47546 r:853) (179 ms)
[2019-08-27 23:12:25] Pool xmr-eu1.nanopool.org share accepted. (GPU0) (a:47547 r:853) (222 ms)
[2019-08-27 23:12:56] Pool xmr-eu1.nanopool.org share accepted. (GPU1) (a:47548 r:853) (167 ms)
[2019-08-27 23:12:57] Pool xmr-eu1.nanopool.org received new job. (job_id: 9113)
[2019-08-27 23:13:26] Pool xmr-eu1.nanopool.org share accepted. (GPU6) (a:47549 r:853) (149 ms)
[2019-08-27 23:13:38] Pool xmr-eu1.nanopool.org share accepted. (GPU0) (a:47550 r:853) (180 ms)
[2019-08-27 23:13:45] Pool xmr-eu1.nanopool.org share accepted. (GPU6) (a:47551 r:853) (136 ms)
[2019-08-27 23:13:57] Pool xmr-eu1.nanopool.org received new job. (job_id: 9114)
[2019-08-27 23:14:01] Pool xmr-eu1.nanopool.org share accepted. (GPU0) (a:47552 r:853) (169 ms)
[2019-08-27 23:14:02] Pool xmr-eu1.nanopool.org share accepted. (GPU1) (a:47553 r:853) (129 ms)
[2019-08-27 23:14:06] Pool xmr-eu1.nanopool.org share accepted. (GPU7) (a:47554 r:853) (127 ms)
[2019-08-27 23:14:12] Pool xmr-eu1.nanopool.org share accepted. (GPU1) (a:47555 r:853) (180 ms)
[2019-08-27 23:14:19] Pool xmr-eu1.nanopool.org share accepted. (GPU1) (a:47556 r:853) (128 ms)
[2019-08-27 23:14:20] Pool xmr-eu1.nanopool.org share accepted. (GPU2) (a:47557 r:853) (168 ms)
[2019-08-27 23:14:25] Pool xmr-eu1.nanopool.org share accepted. (GPU1) (a:47558 r:853) (146 ms)
[2019-08-27 23:14:29] Pool xmr-eu1.nanopool.org share accepted. (GPU5) (a:47559 r:853) (151 ms)
[2019-08-27 23:14:32] Pool xmr-eu1.nanopool.org share accepted. (GPU6) (a:47560 r:853) (155 ms)
[2019-08-27 23:14:49] Pool xmr-eu1.nanopool.org share accepted. (GPU1) (a:47561 r:853) (128 ms)
[2019-08-27 23:14:53] Pool xmr-eu1.nanopool.org share accepted. (GPU0) (a:47562 r:853) (188 ms)
[2019-08-27 23:14:56] Pool xmr-eu1.nanopool.org share accepted. (GPU2) (a:47563 r:853) (142 ms)
[2019-08-27 23:14:58] Pool xmr-eu1.nanopool.org received new job. (job_id: 9115)
[2019-08-27 23:15:11] Pool xmr-eu1.nanopool.org share accepted. (GPU6) (a:47564 r:853) (158 ms)
[2019-08-27 23:15:23] Pool xmr-eu1.nanopool.org received new job. (job_id: 9116)
[2019-08-27 23:15:46] Pool xmr-eu1.nanopool.org share accepted. (GPU6) (a:47565 r:853) (135 ms)
[2019-08-27 23:15:47] Pool xmr-eu1.nanopool.org share accepted. (GPU0) (a:47566 r:853) (159 ms)