DMS4 Pro/ToxicBIOS+OS - How does it work
I've taken the task of figuring out what makes modchips tick and I've decided to attack the DMS4 Pro chip in the process.
From what I've "learned" so far looks like this:
-During the system startup the chip is monitoring the BIOS access and waits until the KERNEL is being copied from the BIOS. At that point it jumps in and steals the databus (forcing the signals overwriting what the BIOS-chip is outputting) to patch part of the KERNEL program code with its own.
-The system then executes the KERNEL, unaware that it has been tampered, since Sony didn't think someone would attack the system this way.
-When the execution of KERNEL reaches the patched code, it first checks which region (A/E/J) the system belongs, then fetches a little bootloader from the modchip, which it finally executes.
-The bootloader then makes some checks to and further patches the KERNEL in memory, and fetches the *ToxicBIOS* from the chip.
-The ToxicBIOS is responsible for reading out some config data from the chip and dealing with the user input to decide which method of boot (normal/mc.dev/hdd.dev/ToxicOS) to perform. It also does some further patching to the kernel (fe. disabling the modchip) and patches some extra functionality to the system for later use. ToxicBIOS is also what deals with bypassing the system copyprotection measurements.
-If the user chooses, or has configured the chip to do so, ToxicBIOS fetches the *ToxicOS* from the chip.
+Some extra information:
-I have no clue how the kernel patch and the bootloader are actually stored in the chip or in the DMS4 update ELF.
-Bootloader is encrypted/decrypted by simply XORing each byte with 0xb3.
-ToxicBIOS is compressed using sjcrunch and uses some relatively simple custom(?) encryption. It can be extracted from the DMS4 update ELF. Decryption is performed by the bootloader. One of the modules inside ToxicBIOS is also RNC compressed. Also several parts of the program use breakpoint traps with encryption for many of its segments (at start a hook is set which gets triggered by the breakpoint and it then decrypts the code segment before executing it normally).
-ToxicOS is also compressed using sjcrunch and uses RC4 encryption. The RC4 key is located inside ToxicBIOS. As with ToxicBIOS, several parts of the program code are encrypted and use the breakpoint traps. RNC compression is again used for several of its modules and graphics. The modules and graphics are also RC4 encrypted. The decryption keys are naturally inside ToxicOS itself, but in addition the module decryption keys are XORed using the MD5 sum of the ToxicOS itself.
-ToxicOS core has two RNC compressed / RC4 encrypted modules inside. Again the keys are also XORed using the MD5 sum of the core itself.
Included now are the ToxicBIOS 1.4 / ToxicOS 0.41 data I've extracted from the modchip in case someone wants to take a peek or join the "fun". :)
(The ToxicOS executable(s) seem to work. More testing needed.)