New modified SMAP driver with FULL DMA support + HDL Dump svr 0.8.7

[SP193]

HDL Dump server v0.8.7 is a modified version of HDL Dump server v0.8.6 with the server, protocol stack and PS2HDD driver from OPL.

I'm providing it as a proof-of-concept that full DMA support does exist for the SMAP.

It uses my own optimized version of PS2SMAP that has full (For transmission and receival) 128-byte block DMA transfers. Redundant code was stripped out, and less important checks were removed.
Right now, it can attain speeds of roughly 1.60MB/s with my SCPH-39006 Playstation 2 console and my PC (About 60% faster than the SMAP driver shipped with OPL).

I can see more ways for improvement though:

1. Port the DMA-support code to EEUG's SMAP driver (Found in OPL or SMS). His driver contains optimizations in MIPS assembly.
However, OPL's SMAP driver totally lacks DMA support and the SMAP driver of SMS only supports DMA transfers during data receival (Only one way).
2. Replace the PS2IP (LWIP) protocol stack with another stack that has less overhead or port the protocol stack to run on the EE instead.
3. Stick the DMA-support code into another SMAP driver; The system used by pktdrv seems good - on my systems, it can push data around at over 2MB/s - without DMA!

I could have ported my driver for use within OPL, but OPL cannot be compiled on my computers at the moment as my PS2SDK is a little too outdated (I have to stick with r1689 or I'll break some of my other homebrew).
Plus I'll like to spend some time on my other interests, my family and my school work (I need to maintain my grades) before I graduate and get comscripted into Singapore's army as part of my compulsary National Service (NS). D:

Additional Notes:
-----------------
1. I have tested the SMAP driver on my SCPH-39006, but not on my SCPH-10000 console. Compatibility cannot be guaranteed with the older console then, especially since I remember that an older version of my DMA-enabled SMAP driver didn't work well on that console.
2. I didn't do any tests on my SCPH-77006 console this time, but I'm sure that it'll work great with it.
3. The loader of this HDL dump server won't be compiled automatically at compile time as I had disabled that in it's makefile.
4. I didn't test the other functionality of the HDL dump server other than installing games. D:
5. I've re-added some of those checks and the intermediary buffer into the SMAP drivers. (See below for more details)

Downloads/Links
Modified SMAP driver source code: http://www.mediafire.com/?l1arl1d6bdjqhj5
(Unofficial) HDL Dump server v0.8.7 source code: http://www.mediafire.com/?xky0xc74d4f9dkx
(Unofficial) HDL Dump server v0.8.7 binary + README file: http://www.mediafire.com/?htonb3byy94bkvt

EDIT: Replaced all uploaded files with more stable versions.
The source code of a modified version of OPL's SMAP driver with DMA support in included in the archive containing the source code of HDL Dump server v0.8.7, under the "SMSMAP" folder.

I had re-added some of those checks and the intermediary buffer into the SMAP drivers in a bid to make them more stable (HDL Dump kept giving me error 10054 lol).

I don't know whether that error kept occurring because the OPL HDL dump server (And/or HDL dump) isn't stable at higher speeds, or the tweaks I've done to the SMAP driver had introduced bugs.

So, with those tweaks undone:

I think that by removing some checks, better performance can be achieved (E.g. modifying the LWIP stack to allocate memory of sizes rounded up to the nearest 128 bytes and modifying CopyToFIFO() to only perform 128-byte block DMA transfers to the SMAP).

The intermediary buffer might probably be removable too....

If you make all of these modifications, you will end up with the driver that I had originally uploaded.

Good luck and have fun!
-SP193

[yoshi314]

looks interesting, although i can get 2.1mb/s with classic hdl_dump0.8.* . does that mean that the solution has currently performance regressions, or i might achieve even higher speed on my setup?

[fresh]

Any movement is a win!
Will have a look at...
Respect and thanks!


Rgds.

[SP193]

looks interesting, although i can get 2.1mb/s with classic hdl_dump0.8.* . does that mean that the solution has currently performance regressions, or i might achieve even higher speed on my setup?

You will probably achieve higher speeds.

But I should probably mention that this driver is not entirely stable. I don't know why, but I believe that it has got something to do with a bug in the protocol stack or SMAP driver that occurs whenever data is transferred too fast (Or when the IOP isn't heavily loaded; In other words, it might be a synchronization issue).

On my setup, the OPL server only gives me ~1.00MB/s (lol... even OPL's network support doesn't want to work with me - not only it's USB support!).
Of course, the old HDL dump server gives me much worse performance than that...

In my post above, I had attached a modified version of the OPL SMAP driver, in case someone wants to stick it into OPL for tests (I cannot do that as OPL cannot be compiled on my setup again).

But I'll appreciate it if someone does manage to complete this driver though. Full DMA support was the only thing that all homebrew SMAP drivers lacked.

@fresh: Thanks.

[deba5er]

@SP193 - This has the potential to speed up several homebrew apps. Really excited to see this. If it is a drop-in replacement I'll try my OPL compile tonight. It would be neat if someone could contact EEUG and let him know about this. Even though he has mostly moved on, earlier this year he noted that he was interested in looking at his drivers again for SMS.

[SP193]

New update 2: It just hit the 5MB/s mark - and is stable. (!)


New update:

I think that I have just seen the PS2 reach unreal speeds - 4.75+MB/s!! @_@

OK, it didn't last more than 10-20 seconds at that rate before freezing up again, but I think that I have found the issue (Explained below).

I did take a screenshot of the HDL dumb screen, but lost it when I copied files. D:

I made another mod, but this time with the HDL Dump server v0.9.1.

With the original design (Data from the Rx FIFO is read from within the interrupt), ~4.75MB/s was reached (The HDD activity indicator of the PS2 shone bright orange while all that happened).

However, when the PS2 froze up, the light stayed on for quite some time - possibly an indication that DMA transfers from the SMAP conflicted with DMA transfers to the HDD

It's possible (After all, they do share the same DMA channel) - while within an interrupt, I cannot perform a DMA transfer from the SMAP via PS2DEV9 (by calling dev9DmaTransfer()) as WaitSema will return an error code and the transfer will not start.

So I initiated the transfer by calling dmac_request() followed by dmac_transfer() to initiate a transfer directly with DMACMAN.
Synchronization relied on checking the CHCR (CHannel Control Register) of the DEV9 DMA channel - so maybe the PS2 froze up because the HDD was transferring data when the transfer got interrupted.

(In other words, just because the TRansfer bit of the CHCR bit got cleared might not mean that the DMAC was done transferring data)

So I made a new version that uses threads - much like the SMAP driver of SMS does (Not OPL! I know that OPL was supposed to have used the SMS driver, but the design isn't exactly the same).

Since it uses a thread to read data from the Rx FIFO when data is received, dev9DmaTransfer() can be used and there are no freezes.

This uploaded version (Unofficial v0.9.2) uses a threaded design, and can reach speeds of around 2.6MB/s.
I know, maybe you can already reach speeds of about 2.2MB/s with the original v0.9.1 server, but this version proves that enabling DMA support doesn't cause issues as long as the IOP's resources are managed properly.

NOTE: If you had problems with the original v0.9.1 server, don't bother using this version either.

Upload linked to in the first post.

EDIT: Scratch that. It seems like this forum now doesn't allow me to edit the first post... so here it is:

EDIT 2: I did some more tests, and found that my suspending interrupts after polling for new packets to be handled, the stability issue has been resolved.

After that, I once even got about 5.15MB/s:



I was wrong - it probably has got nothing to do with the DMAC interrupts, but with the existing PS2 protocol stacks being unable to keep up with such high speeds.

This became evident when I noticed that the freezes occurred once the 3.5MB/s mark was hit.

The problem now is that I don't think that anything can be done to get the LWIP protocol stack to achieve such speeds unless it's turned into a single-threaded module - as suspending the interrupts also disables thread cycling.

Well, maybe the SMAP driver of this new v0.9.2A server will not be used for OPL in-game, but is definitely good for installing games.

Just in case some other developer is interested in finding a more elegant solution to this problem (Other than porting the protocol stack to run on the EE): Disabling the "NO_COPY" option in main.c() makes the driver stable too.

That probably indicates that the problem which the original author of PKTDRV had mentioned is manifesting itself once again, but only when higher speeds are in place.

Downloads/Links
Unofficial HDL Dump server v0.9.2 + source code: hdld_svr_092.7z
Unofficial HDL Dump server v0.92A + source code: hdld_svr_092A.7z

v0.9.2 is a modified version of HDL dump server v0.9.1 but uses threads. It is the original upload in this post and can achieve peek speeds of around 2.6 MB/s.
v0.9.2A is a modified version of HDL dump server v0.9.1 and still handles incoming packets within a interrupt handler, but will suspend interrupts while the HDD writes data.

None of them use the regular SMAP driver that Sony PS2 games, OPL and most other homebrew software use.

[RandQalan]

Wow this rocks

Just a ? thou if you get all bugs gone are you going to incorporate this into OPL and OPL virtual DVD reader

hdld svr 092A runs great at 4.86Mbs for me using hdl_dump_helper_gui_2.2.

I sincerely hope you add this to OPL soon

[SP193]

Wow this rocks

Just a ? thou if you get all bugs gone are you going to incorporate this into OPL and OPL virtual DVD reader

hdld svr 092A runs great at 4.86Mbs for me using hdl_dump_helper_gui_2.2.

I sincerely hope you add this to OPL soon

Wow. Great to hear that it worked well for you.

I don't think that I can integrate this into OPL anytime soon - the LWIP protocol stack (Use by OPL and other homebrew software) isn't stable with DMA support (Probably related to the same problem that I have encountered).

Even if I can get it stable by suspending interrupts (As I did for the HDL Dump server v0.9.2A), it probably isn't a good idea as some games load their own drivers and expect them to run as they did during development (Otherwise, the game might start misbehaving).

But if a new, stable protocol stack and SMAP driver with DMA support was created: YES, OPL will receive proper and stable SMAP DMA support.

[RandQalan]

Good to hear and you cut my time in half for putting games on my HD thanks for this even

[TnA]

Wow!
You got it working successfully on the FATs!


FINALLY we have an improved SMAP! Thank you so much!


Also finally someone brings some other causes and problems up...
The pktdrv-author already mentioned some of them like you said.

Now there is only one thing missing, beside an implementation into other Homebrew-Apps.
Another protocol stack,...

Anyway,... This alone is a heavy/big/cool contribution and improvement!
Can't wait to see this in other apps.