It depends on the algorithm you intend to mine.

For ETH, there's a hard cutoff between cards with sufficient graphics RAM to handle the DAG. (Right now, 2GB cards cannot mine ETH without modifications (such as loading the DAG in chunks), and such modifications have no guarantee of working, or of being worth doing. 3GB cards are the minimum.) However, beyond that hard cutoff, generally cards with additional graphics RAM outperform those with less. For example, the 6GB 1060s tend to outperform their 3GB counterparts by about 30% or more. (Cards with additional graphics RAM also tend to have other unrelated differences, such as default clock settings, due to them being considered higher-end cards than those with less graphics RAM.)

There are additional considerations, such as the ability of your hardware to overclock. Generally cards with Samsung or Micron memory are superior to those with Hynix chips, but note that overclocking abilities vary unit-by-unit.