This thread seems pretty active again, so I figured I'd post some recent findings in hopes we can all solve this thing. ...
That's really interesting. I coded up my own version of this, and it's definitely there.
XOR'ing the key against the heights makes a lot of sense. Was there any rationale to the order you picked the flames? Or what led you to drop one of the flames to make it work (other than to make the math work out)?
The order was found through a search. As I mentioned before, I'm skeptical this is a real message. If anything I believe we may have stumbled upon a collision of the intended cipher, but not the true decryption method.
Initially the order seemed strange but, we rationalized it with the symbolism from the knight and bishop. Knights make jump moves in chess, hence the every 5th bit approach, and bishops move diagonally, hence the diagonal connection of the segments.
The glyphs we believed symbolized the modulus operation required to "unwind" the data. Figuratively, you are "spiraling" around the border of the painting. While this is all very well and good, we've been unable to make any progress on this theory. It was my belief that the message properly decoded would read "thecolouriskeyfile", which would indicate the inner and outer colour tracks would be decrypted by following the same steps as with the height track. The British spelling of "colour" seemed reasonable because the Rob Myers guy is a filthy Canadian, and CoinArtist seemed to have EU heritage, although I couldn't find anything definitive.