Surely they are someone out there that has Silverfish miners and have asked the same question I have. Now the only thing is finding that person with the golden knowledge.
The issue that your having is that it seems the silverfish only support stratum while mining to a wallet only supports getwork iirc.
You may be able to overcome this with a stratum proxy --
https://github.com/Stratehm/stratum-proxyPoint the silverfish at the proxy, then the proxy at the wallet