It is possible to use both Nvidia and AMD cards simultaneously.
For AMD cards, changing RAM timing with atiflash has a major role in GPU tuning.
Afterburner setting is also possible, but it has a little effect and you could ignore afterburner if it makes some problems.
Go mining with 580.
You can mix cards, if it's necessary, but it's not recommended at all.
It will work in Windows, but you can forget about any mixing in linux.