Sounds like the game is not compatible. Loads of games have this effect when forcing to 480p.
In my experience it is useless to try with hdxloder if you get blocky results with gsm.
Only games that bsod can be tried with xploder (and vice versa) and maybe if you´re lucky it will work.
gsm still has the best compatibilty.
gsm works with about 25% of my games (rough estimate).
xploder works with about 10%. 5% of which also works with gsm. The other 5% only works with xploder.
No use to make percentages because it really has to do with the game types, developers, etc.
gsm+opl is i.n.c.r.e.d.i.b.l.e.. This power combo has the highest compatibilty. Several games that did not work with gsm 0.36 work with the opl+gsm pack. Thanks.
PAL SCPH-50004 HDD freemcbooted widescreen set-up: free mcboot, ps2rd, opl. gsm 480p results: http://psx-scene.com/forums/f19/gs-mode-selector-development-feedback-61808/index183.html#post815266