There's been a way to play custom songs on guitar hero 3 on PC version, but since ps3 is almost wide open this solution might work on ps3 too. I have not tested it yet since I don't own psgroove device, but you may want to test it out by yourself.
Here is how to do it on PC:
MundayWeb.com - Adding Custom Songs to Guitar Hero III (PC)
I am eagerly waiting for anyone letting me know if that method works on ps3.
I keep my fingers crossed