Quote

Just had this happen:

14.10.2018 18:22:29 CUDA: Device: 3 Thread exited with message: "the launch timed out and was terminated"
14.10.2018 18:22:29 CUDA: Device: 13 Thread exited with message: "the launch timed out and was terminated"
14.10.2018 18:22:29 CUDA: Device: 12 Thread exited with message: "the launch timed out and was terminated"
14.10.2018 18:22:29 CUDA: Device: 8 Thread exited with message: "the launch timed out and was terminated"
14.10.2018 18:22:29 CUDA: Device: 4 Thread exited with message: "the launch timed out and was terminated"
14.10.2018 18:22:29 CUDA: Device: 0 Thread exited with message: "the launch timed out and was terminated"
14.10.2018 18:22:29 CUDA: Device: 6 Thread exited with message: "the launch timed out and was terminated"
14.10.2018 18:22:29 CUDA: Device: 11 Thread exited with message: "the launch timed out and was terminated"
14.10.2018 18:22:29 CUDA: Device: 1 Thread exited with message: "the launch timed out and was terminated"
14.10.2018 18:22:29 CUDA: Device: 9 Thread exited with message: "the launch timed out and was terminated"
14.10.2018 18:22:29 CUDA: Device: 2 Thread exited with message: "the launch timed out and was terminated"
14.10.2018 18:22:29 CUDA: Device: 7 Thread exited with message: "the launch timed out and was terminated"
14.10.2018 18:22:29 CUDA: Device: 10 Thread exited with message: "the launch timed out and was terminated"
14.10.2018 18:22:29 CUDA: Device: 5 Thread exited with message: "the launch timed out and was terminated"
14.10.2018 18:22:30 WARNING: Looks like GPU0 are stopped. Restarting...
14.10.2018 18:22:30 INFO: GPU0 are restarted.
14.10.2018 18:22:30 WARNING: Looks like GPU1 are stopped. Restarting...
14.10.2018 18:22:30 INFO: GPU1 are restarted.
14.10.2018 18:22:30 WARNING: Looks like GPU2 are stopped. Restarting...
14.10.2018 18:22:30 INFO: GPU2 are restarted.
14.10.2018 18:22:30 WARNING: Looks like GPU3 are stopped. Restarting...
14.10.2018 18:22:30 INFO: GPU3 are restarted.
14.10.2018 18:22:30 WARNING: Looks like GPU4 are stopped. Restarting...
14.10.2018 18:22:30 INFO: GPU4 are restarted.
14.10.2018 18:22:31 WARNING: Looks like GPU5 are stopped. Restarting...
14.10.2018 18:22:31 INFO: GPU5 are restarted.
14.10.2018 18:22:31 WARNING: Looks like GPU6 are stopped. Restarting...
14.10.2018 18:22:31 INFO: GPU6 are restarted.
14.10.2018 18:22:31 WARNING: Looks like GPU7 are stopped. Restarting...
14.10.2018 18:22:31 INFO: GPU7 are restarted.
14.10.2018 18:22:31 WARNING: Looks like GPU8 are stopped. Restarting...
14.10.2018 18:22:31 INFO: GPU8 are restarted.
14.10.2018 18:22:31 WARNING: Looks like GPU9 are stopped. Restarting...
14.10.2018 18:22:31 INFO: GPU9 are restarted.
14.10.2018 18:22:32 WARNING: Looks like GPU10 are stopped. Restarting...
14.10.2018 18:22:32 INFO: GPU10 are restarted.
14.10.2018 18:22:32 WARNING: Looks like GPU11 are stopped. Restarting...
14.10.2018 18:22:32 INFO: GPU11 are restarted.
14.10.2018 18:22:32 WARNING: Looks like GPU12 are stopped. Restarting...
14.10.2018 18:22:32 INFO: GPU12 are restarted.
14.10.2018 18:22:32 WARNING: Looks like GPU13 are stopped. Restarting...
14.10.2018 18:22:32 INFO: GPU13 are restarted.
14.10.2018 18:22:42 INFO: Detected new work: a70f
14.10.2018 18:22:50 Temp: GPU0 0C GPU1 0C GPU2 0C GPU3 0C GPU4 0C GPU5 0C GPU6 0C GPU7 0C GPU8 0C GPU9 0C GPU10 0C GPU11 0C GPU12 0C GPU13 0C
14.10.2018 18:22:50 GPU0: 41 Sol/s GPU1: 45 Sol/s GPU2: 41 Sol/s GPU3: 33 Sol/s GPU4: 42 Sol/s GPU5: 46 Sol/s GPU6: 44 Sol/s GPU7: 48 Sol/s GPU8: 44 Sol/s GPU9: 42 Sol/s GPU10: 41 Sol/s GPU11: 41 Sol/s GPU12: 44 Sol/s GPU13: 46 Sol/s
14.10.2018 18:22:50 Total speed: 598 Sol/s
14.10.2018 18:23:21 INFO: Target: 0013813813813814...
14.10.2018 18:23:21 Temp: GPU0 0C GPU1 0C GPU2 0C GPU3 0C GPU4 0C GPU5 0C GPU6 0C GPU7 0C GPU8 0C GPU9 0C GPU10 0C GPU11 0C GPU12 0C GPU13 0C
14.10.2018 18:23:21 GPU0: 0 Sol/s GPU1: 0 Sol/s GPU2: 0 Sol/s GPU3: 0 Sol/s GPU4: 0 Sol/s GPU5: 0 Sol/s GPU6: 0 Sol/s GPU7: 0 Sol/s GPU8: 0 Sol/s GPU9: 0 Sol/s GPU10: 0 Sol/s GPU11: 0 Sol/s GPU12: 0 Sol/s GPU13: 0 Sol/s
14.10.2018 18:23:21 Total speed: 0 Sol/s
14.10.2018 18:23:21 INFO: Detected new work: a710
14.10.2018 18:23:52 Temp: GPU0 0C GPU1 0C GPU2 0C GPU3 0C GPU4 0C GPU5 0C GPU6 0C GPU7 0C GPU8 0C GPU9 0C GPU10 0C GPU11 0C GPU12 0C GPU13 0C
14.10.2018 18:23:52 GPU0: 0 Sol/s GPU1: 0 Sol/s GPU2: 0 Sol/s GPU3: 0 Sol/s GPU4: 0 Sol/s GPU5: 0 Sol/s GPU6: 0 Sol/s GPU7: 0 Sol/s GPU8: 0 Sol/s GPU9: 0 Sol/s GPU10: 0 Sol/s GPU11: 0 Sol/s GPU12: 0 Sol/s GPU13: 0 Sol/s
14.10.2018 18:23:52 Total speed: 0 Sol/s
14.10.2018 18:24:16 INFO: Target: 001999999999999a...
14.10.2018 18:24:16 INFO: Detected new work: a711
14.10.2018 18:24:23 Temp: GPU0 0C GPU1 0C GPU2 0C GPU3 0C GPU4 0C GPU5 0C GPU6 0C GPU7 0C GPU8 0C GPU9 0C GPU10 0C GPU11 0C GPU12 0C GPU13 0C
14.10.2018 18:24:23 GPU0: 0 Sol/s GPU1: 0 Sol/s GPU2: 0 Sol/s GPU3: 0 Sol/s GPU4: 0 Sol/s GPU5: 0 Sol/s GPU6: 0 Sol/s GPU7: 0 Sol/s GPU8: 0 Sol/s GPU9: 0 Sol/s GPU10: 0 Sol/s GPU11: 0 Sol/s GPU12: 0 Sol/s GPU13: 0 Sol/s
14.10.2018 18:24:23 Total speed: 0 Sol/s
14.10.2018 18:24:31 WARNING: Looks like GPU0 are stopped. Restarting...
14.10.2018 18:24:36 ERROR: Looks like GPU0 are stuck.
14.10.2018 18:24:36 WARNING: Looks like GPU1 are stopped. Restarting...
14.10.2018 18:24:41 ERROR: Looks like GPU1 are stuck.
14.10.2018 18:24:41 WARNING: Looks like GPU2 are stopped. Restarting...
14.10.2018 18:24:46 ERROR: Looks like GPU2 are stuck.
14.10.2018 18:24:46 WARNING: Looks like GPU3 are stopped. Restarting...
14.10.2018 18:24:51 ERROR: Looks like GPU3 are stuck.
14.10.2018 18:24:52 WARNING: Looks like GPU4 are stopped. Restarting...
14.10.2018 18:24:53 Temp: GPU0 0C GPU1 0C GPU2 0C GPU3 0C GPU4 0C GPU5 0C GPU6 0C GPU7 0C GPU8 0C GPU9 0C GPU10 0C GPU11 0C GPU12 0C GPU13 0C
14.10.2018 18:24:53 GPU0: 0 Sol/s GPU1: 0 Sol/s GPU2: 0 Sol/s GPU3: 0 Sol/s GPU4: 0 Sol/s GPU5: 0 Sol/s GPU6: 0 Sol/s GPU7: 0 Sol/s GPU8: 0 Sol/s GPU9: 0 Sol/s GPU10: 0 Sol/s GPU11: 0 Sol/s GPU12: 0 Sol/s GPU13: 0 Sol/s
14.10.2018 18:24:53 Total speed: 0 Sol/s
14.10.2018 18:24:57 ERROR: Looks like GPU4 are stuck.
14.10.2018 18:24:57 WARNING: Looks like GPU5 are stopped. Restarting...
14.10.2018 18:24:58 INFO: Detected new work: a712
14.10.2018 18:25:02 ERROR: Looks like GPU5 are stuck.
14.10.2018 18:25:02 WARNING: Looks like GPU6 are stopped. Restarting...
14.10.2018 18:25:07 ERROR: Looks like GPU6 are stuck.
14.10.2018 18:25:07 WARNING: Looks like GPU7 are stopped. Restarting...
14.10.2018 18:25:12 ERROR: Looks like GPU7 are stuck.
14.10.2018 18:25:12 WARNING: Looks like GPU8 are stopped. Restarting...
14.10.2018 18:25:17 ERROR: Looks like GPU8 are stuck.
14.10.2018 18:25:18 WARNING: Looks like GPU9 are stopped. Restarting...
14.10.2018 18:25:23 ERROR: Looks like GPU9 are stuck.
14.10.2018 18:25:23 WARNING: Looks like GPU10 are stopped. Restarting...
14.10.2018 18:25:24 Temp: GPU0 0C GPU1 0C GPU2 0C GPU3 0C GPU4 0C GPU5 0C GPU6 0C GPU7 0C GPU8 0C GPU9 0C GPU10 0C GPU11 0C GPU12 0C GPU13 0C
14.10.2018 18:25:24 GPU0: 0 Sol/s GPU1: 0 Sol/s GPU2: 0 Sol/s GPU3: 0 Sol/s GPU4: 0 Sol/s GPU5: 0 Sol/s GPU6: 0 Sol/s GPU7: 0 Sol/s GPU8: 0 Sol/s GPU9: 0 Sol/s GPU10: 0 Sol/s GPU11: 0 Sol/s GPU12: 0 Sol/s GPU13: 0 Sol/s
14.10.2018 18:25:24 Total speed: 0 Sol/s
14.10.2018 18:25:28 ERROR: Looks like GPU10 are stuck.
14.10.2018 18:25:28 WARNING: Looks like GPU11 are stopped. Restarting...
14.10.2018 18:25:33 ERROR: Looks like GPU11 are stuck.
14.10.2018 18:25:33 WARNING: Looks like GPU12 are stopped. Restarting...
14.10.2018 18:25:38 ERROR: Looks like GPU12 are stuck.
14.10.2018 18:25:39 WARNING: Looks like GPU13 are stopped. Restarting...
14.10.2018 18:25:44 ERROR: Looks like GPU13 are stuck.
14.10.2018 18:25:45 WARNING: Looks like GPU0 are stopped. Restarting...
14.10.2018 18:25:50 ERROR: Looks like GPU0 are stuck.
14.10.2018 18:25:50 INFO: Exit...

took almost 3 minutes to exit after error.

I think i found the gpu causing the problem