Been done before - fact is, end user will always have the power, because they have physical control over the machine. Any code that executes, they can modify - this includes firmware code - and all of it is open for disassembly and inspection.
I'm sure your x-algo asic kernels can be scrutinized in very much the same way.

jk.. Or am I ?...

In both senarios, we the general public can only imagine who has that code and what their intentions are.
EDIT
And.. how powerful it really is.
They can indeed. If you can grok AMD ISA code for GCN, you can do that. And the code is pretty decent - I wouldn't call it excellent or anything, nowhere near - but it's far better than the copy+paste of CPU code done by people who barely knew what a GPU is.