Does Shadow hearts 2 covenant work with any OPL + HDD?

[yoshi314]

so, it would seem that a actual iop module patch is required this time around.

[Udosatiz]

Shadow Hearts 2 depends on the timing of certain function calls from the CDVDMAN module. Technically, this is a bug in the game's IOP modules; it is pure coincidence (or luck ) that it does not trigger on real PS2 drive hardware, just in emulation (or, in some cases, on backwards-compatible PS3s ).

Some technical details: the game uses a custom file I/O library (FIO.IRX), and that one has a bizarre race condition. When attempting to read sectors, it first performs a seek; when the seek completes, the callback triggers a state change in the underlying automaton, which then performs the read. The problem is now that the automaton state is set after sceCdSeek returns; so in case the CD callback runs before (which may happen in case of an emulated CD, depending on the timing and overall IOP load) the automaton permanently remains in the "seeking" state.

The code looks roughly like this (names are from FIO.IRX, as it still contains debug information):

[in HeadSeek]
if ((res = sceCdSeek(lsn)) != 0) { FioCommand = 0x2f; } else ...

[in CD_Read_Callback]
if (func == SCECdFuncSeek) { FioCommand = 2; WaitTime = 0; } else ...

As sceCdSeek is an asynchronous function, the call usually returns long before the callback is invoked, so it has all the time it needs to set FioCommand to 0x2f ("seeking"). If the callback runs afterwards, FioCommand ends up being set to 2, and the main automaton loop (in main_thread) continues to invoke Read_Order to read the disc. This is the usual case, and the game runs fine as long as this happens.
However, it is also possible that the callback runs before FioCommand is set in HeadSeek. In this case, FioCommand is first set to the correct value (2) in the callback, which is then almost immediately overwritten by 0x2f. Once this happens, it appears the disc drive is stuck seeking, and the game freezes.

It is possible, albeit cumbersome, to "un-freeze" the game. If a debugger is attached to the IOP and the address of the FioCommand variable is known, one can manually set it to 2, which continues the game. I have testet this on a T10k with my own CDVD emulator (Kermit/ODEM), and it works reliably.

A fix for this problem is to delay the invocation of the command completion callback. On my setup, I didn't get a single freeze in ~10h by using a timeout of 2ms (instead of the minimum value of 200us). However, please note that I'm using a custom emulator (ODEM) on development hardware, so results may be different for OPL.

By the way: I played the PAL version of SH:C, and it definitely allows saving right before the disc change.

Wow , very useful and interesting detail. Thanks for sharing.

[Udosatiz]

Just received the network adapter for hope to play Shadow hearts covenant but no luck. Tested with HDD IDE and SMB but it still have random *freeze* on OPL 0.5/0.6/0.7/0.8, tried with serveral setting.

SilverBull, could you disassemble the code in DVD seeking and loading module or you have any clue to hex edit the neccesary files?

[SP193]

Shadow Hearts 2 depends on the timing of certain function calls from the CDVDMAN module. Technically, this is a bug in the game's IOP modules; it is pure coincidence (or luck ) that it does not trigger on real PS2 drive hardware, just in emulation (or, in some cases, on backwards-compatible PS3s ).

Some technical details: the game uses a custom file I/O library (FIO.IRX), and that one has a bizarre race condition. When attempting to read sectors, it first performs a seek; when the seek completes, the callback triggers a state change in the underlying automaton, which then performs the read. The problem is now that the automaton state is set after sceCdSeek returns; so in case the CD callback runs before (which may happen in case of an emulated CD, depending on the timing and overall IOP load) the automaton permanently remains in the "seeking" state.

The code looks roughly like this (names are from FIO.IRX, as it still contains debug information):

[in HeadSeek]
if ((res = sceCdSeek(lsn)) != 0) { FioCommand = 0x2f; } else ...

[in CD_Read_Callback]
if (func == SCECdFuncSeek) { FioCommand = 2; WaitTime = 0; } else ...

As sceCdSeek is an asynchronous function, the call usually returns long before the callback is invoked, so it has all the time it needs to set FioCommand to 0x2f ("seeking"). If the callback runs afterwards, FioCommand ends up being set to 2, and the main automaton loop (in main_thread) continues to invoke Read_Order to read the disc. This is the usual case, and the game runs fine as long as this happens.
However, it is also possible that the callback runs before FioCommand is set in HeadSeek. In this case, FioCommand is first set to the correct value (2) in the callback, which is then almost immediately overwritten by 0x2f. Once this happens, it appears the disc drive is stuck seeking, and the game freezes.

It is possible, albeit cumbersome, to "un-freeze" the game. If a debugger is attached to the IOP and the address of the FioCommand variable is known, one can manually set it to 2, which continues the game. I have testet this on a T10k with my own CDVD emulator (Kermit/ODEM), and it works reliably.

A fix for this problem is to delay the invocation of the command completion callback. On my setup, I didn't get a single freeze in ~10h by using a timeout of 2ms (instead of the minimum value of 200us). However, please note that I'm using a custom emulator (ODEM) on development hardware, so results may be different for OPL.

By the way: I played the PAL version of SH:C, and it definitely allows saving right before the disc change.

Wow. That's some high-quality analysis done there.
Thanks for the information.

If we were to NOP the instruction that comes after sceCdSeek() that sets FioCommand to 0x2f, won't the problem be solved?

[zero35]

Wow. That's some high-quality analysis done there.
Thanks for the information.

If we were to NOP the instruction that comes after sceCdSeek() that sets FioCommand to 0x2f, won't the problem be solved?

I'm concern this principle maybe help solved SRT Impact's timing bug also？
because i found sio2man.irx from cdrom0:\IRX,maybe it's SRT Impact's custom I/O file ???

[SP193]

I'm concern this principle maybe help solved SRT Impact's timing bug also？
because i found sio2man.irx from cdrom0:\IRX,maybe it's SRT Impact's custom I/O file ???

No.
SRW IMPACT's timing problem does not seem to originate from any of the IOP modules loaded by itself, but from the EE side (The CPU of the PS2).

The SIO2MAN module is Sony's driver for the SIO2 interface (Used to connect to the Memory Cards and controller ports).

[SilverBull]

Wow. That's some high-quality analysis done there.
Thanks for the information.

You're welcome.

If we were to NOP the instruction that comes after sceCdSeek() that sets FioCommand to 0x2f, won't the problem be solved?

That could be part of a solution, but I doubt it'll be sufficient. FioCommand is checked in a loop in main_thread, and will be invoked again during each iteration as long as it stays at its old value (0x2e).

Because we don't know whether this is the only game that makes implicit assumptions about timing of n-cmd invocation and the completion callback, I'd suggest not to do a game-specific patch here. As I wrote, I was able to get rid of the hangs completely by increasing the delay after which the completion callback runs to 2 milliseconds. This should not have any adverse side effects on other games, as it can take almost forever on real hardware as well; sceCdSeek may never terminate in reasonable time if the target sector is unreachable (within the TOC, but scratched/destroyed), so games already need to take that into account. On the other hand, premature termination of the command is not possible on the PS2, so I'd think this is the reason this bug was not found earlier during QA.

If you would still like to have a game-specific fix, I recommand this one (which I haven't tested myself): set the FioCommand before calling sceCdSeek, then do not touch it at all if the function succeeds:

FioCommand = 0x2f;
if ((res = sceCdSeek(lsn)) != 0) { /* do nothing */ } else ...

The "else" case should be meaningless for OPL, or is it possible that n-cmds fail (sorry, I haven't looked at the code for ages )? Anyway, it is not necessary to change anything in that branch; FIO.IRX simply sets FioCommand to the error state, but that cannot interfere with the callback, as there won't be an outstanding CDVD command.

On a side note: I'd assume this could also fix the problems with this game on backwards-compatible PS3s, if (and only if) we had a chance to inject a modified IRX .

No.
SRW IMPACT's timing problem does not seem to originate from any of the IOP modules loaded by itself, but from the EE side (The CPU of the PS2).

I cannot comment on that specific game, but I remember something from debugging SH:C that may be interesting. When FIO.IRX has entered the bad state and hangs, it won't effect the EE at all, besides that its waiting for data that doesn't arrive. There is no exception, crash, tight endless loop or anything like that; the game's main loop continues as usual, and you cannot see anything (even in a DECI2 debugger). The game has two threads which it periodically switches around; the context switch counts of both threads increase with the same frequency, irrespective of whether the IOP hangs in FIO.IRX or not.

This is completely different to another game I had debugged quite some time ago, Xenosaga 2. You might remember that XS2 had a somewhat similar problem on HDL, as it would randomly hang as well (with the music still playing). In contrast to SH:C, though, XS2 did crash: right in the SIF RPC completion callback, when trying to free the completed RPC packet, as it passed a NULL pointer to the _rpc_pkt_free routine. I was never able to pinpoint why the SIF handler was being called with a wrong parameter, but I suspect it is a synchronization bug on the IOP.
The problem went away after I changed the CDVD emulator to cache a single directory sector for sceCdLayerSearchFile. Turned out XS2 was calling that routine for each access to its data files, whereas the requested file was not present in the ISOFS, but instead in a game-specific file system inside one of the big data files. Calling sceCdLayerSearchFile was completely superfluous (guaranteed to fail), but it somehow effected the timing and ultimately led to the defect SIF completion message.

[zero35]

Hi,SilverBull
Super Robot Taisen Impact is a random freeze game like Shadow hearts 2 on OPL,can you do some analysis on this game for help sp193 find out it's freeze principle?
it's disc-code is:Slps_251.04,a NTSC-J game

[SP193]

You're welcome.

That could be part of a solution, but I doubt it'll be sufficient. FioCommand is checked in a loop in main_thread, and will be invoked again during each iteration as long as it stays at its old value (0x2e).

Because we don't know whether this is the only game that makes implicit assumptions about timing of n-cmd invocation and the completion callback, I'd suggest not to do a game-specific patch here. As I wrote, I was able to get rid of the hangs completely by increasing the delay after which the completion callback runs to 2 milliseconds. This should not have any adverse side effects on other games, as it can take almost forever on real hardware as well; sceCdSeek may never terminate in reasonable time if the target sector is unreachable (within the TOC, but scratched/destroyed), so games already need to take that into account. On the other hand, premature termination of the command is not possible on the PS2, so I'd think this is the reason this bug was not found earlier during QA.

If you would still like to have a game-specific fix, I recommand this one (which I haven't tested myself): set the FioCommand before calling sceCdSeek, then do not touch it at all if the function succeeds:

FioCommand = 0x2f;
if ((res = sceCdSeek(lsn)) != 0) { /* do nothing */ } else ...

The "else" case should be meaningless for OPL, or is it possible that n-cmds fail (sorry, I haven't looked at the code for ages )? Anyway, it is not necessary to change anything in that branch; FIO.IRX simply sets FioCommand to the error state, but that cannot interfere with the callback, as there won't be an outstanding CDVD command.

On a side note: I'd assume this could also fix the problems with this game on backwards-compatible PS3s, if (and only if) we had a chance to inject a modified IRX .

That does clear more things up...
Thanks for the extra solutions too.

If I remember correctly, N-commands will only fail on OPL only if the N-command function was called while within an interrupt context (Otherwise, it will surely succeed).

I cannot comment on that specific game, but I remember something from debugging SH:C that may be interesting. When FIO.IRX has entered the bad state and hangs, it won't effect the EE at all, besides that its waiting for data that doesn't arrive. There is no exception, crash, tight endless loop or anything like that; the game's main loop continues as usual, and you cannot see anything (even in a DECI2 debugger). The game has two threads which it periodically switches around; the context switch counts of both threads increase with the same frequency, irrespective of whether the IOP hangs in FIO.IRX or not.

This is completely different to another game I had debugged quite some time ago, Xenosaga 2. You might remember that XS2 had a somewhat similar problem on HDL, as it would randomly hang as well (with the music still playing). In contrast to SH:C, though, XS2 did crash: right in the SIF RPC completion callback, when trying to free the completed RPC packet, as it passed a NULL pointer to the _rpc_pkt_free routine. I was never able to pinpoint why the SIF handler was being called with a wrong parameter, but I suspect it is a synchronization bug on the IOP.
The problem went away after I changed the CDVD emulator to cache a single directory sector for sceCdLayerSearchFile. Turned out XS2 was calling that routine for each access to its data files, whereas the requested file was not present in the ISOFS, but instead in a game-specific file system inside one of the big data files. Calling sceCdLayerSearchFile was completely superfluous (guaranteed to fail), but it somehow effected the timing and ultimately led to the defect SIF completion message.

I saw something like what you have observed in Xenosaga - the _rpc_pkt_free routine is passed a NULL pointer.

That game was SSX3, and it seemed to be either a bug or anti-HDL system:
1. The game calls one of the CD/DVD routines without calling sceCdInit() first (Possibly without initializing some of the other SIF RPCs too).
2. If the call succeeds, the EE will crash later on when a NULL pointer is passed to _rpc_pkt_free.
3. If the call was to fail (Which should happen), the game would initialize everything first before retrying.

----
About SRW IMPACT:

Modifying the RPC call to sceCdReadIOPm() to return without calling sceCdSync() solves the freezing issue.
However, the BGM will be messed up.

Plus the music is still playing when the game "freezes". The pad's analog light is on too, so the IOP is definitely still running.

[zero35]

That does clear more things up...
Thanks for the extra solutions too.

If I remember correctly, N-commands will only fail on OPL only if the N-command function was called while within an interrupt context (Otherwise, it will surely succeed).



I saw something like what you have observed in Xenosaga - the _rpc_pkt_free routine is passed a NULL pointer.

That game was SSX3, and it seemed to be either a bug or anti-HDL system:
1. The game calls one of the CD/DVD routines without calling sceCdInit() first (Possibly without initializing some of the other SIF RPCs too).
2. If the call succeeds, the EE will crash later on when a NULL pointer is passed to _rpc_pkt_free.
3. If the call was to fail (Which should happen), the game would initialize everything first before retrying.

----
About SRW IMPACT:

Modifying the RPC call to sceCdReadIOPm() to return without calling sceCdSync() solves the freezing issue.
However, the BGM will be messed up.

Plus the music is still playing when the game "freezes". The pad's analog light is on too, so the IOP is definitely still running.

Seems to need to find a more perfect solution