Is it possible to build this algorithm using something like a TEE/secure enclave/etc. such that I can generate a proof that my result was in fact produced by my unaltered (open-source) algorithm running on hardware following the secure protocol?
Yes, you can run it inside an Intel software enclave on the CPU, but this is only available on fairly recent Intel processors with the SGX instruction set, and AMD doesn't have an equivalent (and actually this instruction set was deprecated by Intel recently!)
If so, how long would such a proof be? Would it be possible to encode the proof into the Bitcoin blockchain so that someone could later verify it?
That would depend on the ZK proof algo you are using. If you choose
bulletproofs then the size is going to be "8 + 2 log n group elements and 5 scalars". Here, group elements represent elliptic curve points.