Then you most likely:

- don't have a common ground or
- you cut supply to the USB chip as well or
- Your power supply doesn't supply a stable enough voltage at low startup current.

If you use a 'standard' ATX computer power supply, they don't really work well at very low loads. See here for a description and solution:

https://mysite.du.edu/~etuttle/electron/elect62.htm