Here's my five cents.
I think you are basically supposed to tick every mode that your display device supports. That way PS3 knows if it can present the current media in its native format or if it needs to be rescaled (not necessarily upscaled) to another resolution, higher or lower. To my knowledge the XMB always uses the highest ticked resolution, in my case 1080p which is my TV's native resolution, therefore giving the best picture quality.
If you have not selected the full upscaling option from the A/V settings the PS3 will change the resolution to 576p or 480p when playing DVD. Otherwise it will upscale the material to the XMB resolution.
The upscaling is not just about streching or zooming the video, it actually means that the video is converted to high definition resolution. The final quality of this process is determined by the quality of the used upconverting algorithm/hardware and the media itself. PS3 does a very nice job converting DVD video up to FullHD resolution. This usually means slightly better video quality but still it isn't comparable to true high definition content because one simply can't 'magically' create actual picture information with these conversion processes. That is an important fact to remember.
The games on the other hand, are different. I haven't fiddled around too much with the game resolutions, but I have found a few issues. Most PS3 games are natively 720p, but not all of them agree to use 1080p resolution if the native one is unticked from the PS3's picture options - they just ignore the settings and force 720p resolution. I don't know if this would cause the used display device to actually go blank if it really didn't support 720p but one might assume it might just do that.
Then, are the graphics of native 720p games that support 1080p mode upscaled/upconverted to the higher resolution? I think they are just zoomed/stretched, but I'm not sure.
-= It's hard to be first class when you're white trash =-
PS2 v7 / Matrix Infinity / Seagate 320GB / Datel MAX Memory 64MB
PS3 60GB CECHC04