Some of the features on X11 clones and subsequent version of 'X' don't have all the API calls of Scrypt or SHA-256 coins. The cloning guides are pretty old now and don't cover these eventualities.

Built-in CPU mining is one of these features so when trying to mine the genesis block with setgeneratetrue command, it won't work. If you haven't got a mining rig, you will have to download and compile a version of CPUminer for your system/VPS before you can start the blockchain running, which I see you've already done though.

You need to check if your yourcoin/makefile.osx has all the correct paths to the hash algorithms. If you compare the Linux makefile that works to the Mac one you may notice some of the hash paths are missing i.e. obj/fugue, obj/shavite etc.