Getting crashing on my cuda 9.1 workstation
cuda error: unspecified launch failure in cduMem::read (58)
[12:42:50] ERROR - GPU2 stopped working due to a critical error
[12:42:50] INFO - Attempt to restart after 10 seconds
cuda error: unspecified launch failure in cduMem::read (58)
[12:42:50] ERROR - GPU1 stopped working due to a critical error
[12:42:50] INFO - Attempt to restart after 10 seconds
cuda error: unspecified launch failure in cduMem::free (36)
cuda error: unspecified launch failure in cduMem::free (36)
cuda error: unspecified launch failure in cduMem::free (36)
cuda error: unspecified launch failure in cduMem::free (36)
cuda error: unspecified launch failure in algWorkerStd::finalise (141)
cuda error: all CUDA-capable devices are busy or unavailable in cduMem::create (45)
[12:43:00] ERROR - GPU2 unable to recover from error
cuda error: unspecified launch failure in duMem::free (3936m)
cuda error: unspecified launch failure in cduMem::free ********************************************************************************
_____ _ _____ _
/ ____| | | | __ \ | |
| | _ __ _ _ _ __ | |_ ___ | | | |_ __ ___ __| | __ _ ___
| | | '__| | | | '_ \| __/ _ \| | | | '__/ _ \/ _` |/ _` |/ _ \
| |____| | | |_| | |_) | || (_) | |__| | | | __/ (_| | (_| | __/
\_____|_| \__, | .__/ \__\___/|_____/|_| \___|\__,_|\__, |\___|
__/ | | __/ |
|___/|_| |___/
********************************************************************************