If I understand you correctly: You are saying that you don't get RGB output if you boot into the hacked OSD, but you get RGB output if you boot into the OSD without FMCB?
If yes, it could be like I thought: FMCB has broken NVRAM/EEPROM configuration handling.
The RGB setting is only used and carried out by Sony software - AFAIK it's only used by the OSD. Most homebrew software will not display video in RGB mode out of the box, regardless of the value of that setting.
SCPH-10000 S. MINOKAMO v1.01 (Defunct)
SCPH-10000 S. KISARAZU v1.00 (faulty)
SCPH-15000 S. KOHDA (With warranty seal!) :D
DESR-5100 S. EMCS