dark8011: lookup-gap is only used with Scrypt. I think thread-concurrency also is only used with Scrypt, but in any case it is not used to calculate buffer size for non-Scrypt. I think your cards unfortunately just do not have enough VRAM for these algorithms. The only thing you can try is set gpu-threads to 1 if you don't already have that. Other than that I don't think there's much you can do.

For developers: Perhaps X11/X13... buffer size can be lowered? I remember that some miners had a formula to calculate it, while we use hardcoded values in sgminer 5.0.