Este tópico tem uma solução: link
in your xorg.conf file where it says
DefaultDepth 24
change this to readDefaultDepth 16
, maybe even try making it 8, but I know 16 works. The card doesn't have enough memory to run any higher than 800x600 at 24 depth, so by changing the depth you will free up enough memory to run at higher resolutions