Been done before - fact is, end user will always have the power, because they have physical control over the machine. Any code that executes, they can modify - this includes firmware code - and all of it is open for disassembly and inspection.
I'm sure your x-algo asic kernels can be scrutinized in very much the same way.

jk.. Or am I ?...

In both senarios, we the general public can only imagine who has that code and what their intentions are.
EDIT
And.. how powerful it really is.