I don't see an error in the Xorg log file. If the grey screen apears, it is correct and you need just to wait longer. xfce4 needs more than 30 minutes to start from internal hard disc. If you start from USB, it will take much more time. Without a swap partition there is no chance to start it.
When you reduce the video memory to the minimum, it will be faster.
For 640x480-32 it is 1228800 Bytes.
For 640x480-16 it is 614400 Bytes.