Firstly no it's not really a problem LOSING 80% of the network hash, it's a good test of how resilient the code is. Without that happening I won't be able to point to it in the future as evidence. It's an issue all new chains will face gaining support from miners. They come and go when it's "immediately profitable" or "immediately not profitable".
Let's assume we lose almost all the rest of the miners. What's the longest time that can pass until the next difficulty adjustment?