They better hurry up with that algo change

If they use a common algorithm like scrypt with such a small market cap, what is stopping say, a litecoin mining farm from one day 51% attacking the coin?  Honestly I feel like such an attack could have happened at any point in the last year, especially now considering how much LTC's hashrate has increased, only reason it hasn't happened already is probably that no one cares enough.