I think I made a mistake buying at $23,500 (obviously), so when should I really start to panic now that we dropped to $18K?
Market conditions have improved several times since the price dropped $18K. The increase is not very high, only a price change of $2k, or with a price increase of $20k, then you can already make a profit. But if you just leave Bitcoin in your wallet, of course to get back $23k, it will have to wait a long time. Because the current market conditions are bearish.
The price has dropped to $18,000 levels already, then bounce back above $20,000 and now we are in $19,000. So it's not that huge decreased if the OP bought at $23,000, just a few thousands dollars and if the price goes sideways then just hold on and don't sell.
Again, it might be take several years to recover, but once the market enter the bull run, you won't regret your decision to hold or even buy at the current price.