New modified SMAP driver with FULL DMA support + HDL Dump svr 0.8.7

[fresh]

Hi!

A little bit of feedback.
A quick test on a v7 shows me this speeds over 2 switches:

0.92


0.92a



Thanks for this work. I knew you´re a good one...




Rgds.


PS: Could anyone tell me why this board scales the attachments?...

[SP193]

@Fresh: Thanks.

So your connection to your PS2 goes through 2 switches and you got 3.87MB/s? That is already quite good.

Wow!
You got it working successfully on the FATs!


FINALLY we have an improved SMAP! Thank you so much!


Also finally someone brings some other causes and problems up...
The pktdrv-author already mentioned some of them like you said.

Now there is only one thing missing, beside an implementation into other Homebrew-Apps.
Another protocol stack,...

Anyway,... This alone is a heavy/big/cool contribution and improvement!
Can't wait to see this in other apps.

Wow. Haven't seen you here in a while. Welcome back.

Yes, I have gotten it working on my SCPH-39006 console (However, compatibility with my SCPH-10000 is still unknown as I have not tested it on that console yet).

It turned out that the FAT PS2 DMAC (On the IOP side) is a bit more picky when it comes to accepting parameters: The the target device does not have any more data to send (When more data than available is requested), it seems like transfers will somehow not get completed successfully.

DMA transfers MUST be in 128-byte blocks for some reason too (A bug in the DMAC block of the IOP?).
The solution was to manually copy out the remaining data from the FIFOs.

Yea, I heard of the issues with the PKTDRV, but I feel that it is the way forward. LWIP might be lightweight, but I don't seem to be able to port the latest version (v1.4.0) over to the PS2.

Even if I got it ported over, the 36MHz IOP will slow down transfers (Probably to sub-4MB/s) since the system used by the LWIP protocol stack requires buffers to be dynamically allocated and memcpy() to be used.

The concrete solution here would be to make a system with the protocol stack as tightly integrated to the SMAP driver as possible.
That will ensure that as little CPU time will be required for packets to be received and generated.

@All: If I do get some more time, I might attempt to improve PKTDRV. But I need a tester who has issues with using PKTDRV (In HDL Dump server v0.9.1) to help out.

If necessary, I will redesign the existing protocols used by the HDL dump server for better performance and stability.

(Of course, you have to be willing to do tests).

[TnA]

I'm really interested in an implementation into other Homebrew-Apps.

There are a lot possibilities.

-If uLE could have this and SMB-Support,... OMG!
-SMS might play back all DVD-VOBs through network without stutters. Now if it would only support IFOs too, that would be amazing (it doesn't need to handle the menu-stuff, but different subtitles, or atleast the playback of multiple VOBs)!
-OPL's GUI/HDL-Dump-Server obviously could improve a lot... The InGame-Core hopefully too!
-PS2-Linux/MegaMan's RTE! Can't wait to see PS2-Linux loaded through SMB/NFS/LAN with DMA-SMAP-Driver-Support!

[fresh]

@Fresh: Thanks.

So your connection to your PS2 goes through 2 switches and you got 3.87MB/s? That is already quite good.

Yes, but this is only possible with hq-toys (switches)...



As TnA stated before this driver could change the whole ps2 scene.
If i could help in any way, gimme a wink...



Rgds.

[yoshi314]

@All: If I do get some more time, I might attempt to improve PKTDRV. But I need a tester who has issues with using PKTDRV (In HDL Dump server v0.9.1) to help out.

If necessary, I will redesign the existing protocols used by the HDL dump server for better performance and stability.

(Of course, you have to be willing to do tests).

what kind of issues do you have in mind? my only issue is that network does not start up properly in 0.9.* and i have to init it with uLE first.

[SP193]

what kind of issues do you have in mind? my only issue is that network does not start up properly in 0.9.* and i have to init it with uLE first.

Frankly, I don't really know.

I heard that there were issues with using HDL Dump server v0.9.1, but didn't know the details. So I thought that I would wait for someone who has experienced issues with it to share his experiences with me.

You say that the only issue is that the NA isn't properly initialized with PKTDRV? That should be fixable (By switching over to the regular SMAP driver instead).

[dlanor]

New update 2: It just hit the 5MB/s mark - and is stable. (!)

That is great to hear ! But are you sure ?

I ask this mainly because you never mention fixing the bugs of HDL_Dump (both client and server) which made the original v0.9 corrupt games.

The basic problem is that TCP protocol was replaced by UDP, which has no long term connection stream, but only separate packets each of which establishes a brand new 'single-packet connection'. This eliminates some of the time-consuming overhead of TCP implementation on IOP, but requires that the client-server coder re-implements all necessary error detection, correction, and recovery methods that TCP protocol has built-in, but which UDP does not have any part of...

So HDL_Dump+HDLD_svr v0.9 gave very good speed, but with reliable results only in a LAN 100% free of packet errors.
I presume that the client and server changes caught some of the errors, but nevertheless lots of people got corrupted game installs.
And if those bugs are still present, then it doesn't matter how much faster the DMA methods have allowed the transfer to get.
Because if some packet errors are neither detected nor corrected the resulting game installs will be worthless.

Personally I'd have preferred you to do this work based on one of the traditionally stable versions of HDL_Dump+HDLD_svr, such as v0.8.6

Using v0.9 as basis means that it will be impossible to tell whether any problems with the installed games are due to the new code or due to the insufficient error recovery abilities of the v0.9 UDP implementation of the HDL_Dump methods.

EDIT: Scratch that. It seems like this forum now doesn't allow me to edit the first post... so here it is:

That is due to thread sabotage by some users involved in the recent 'hate attacks' against this site, where they attempted to mess up old threads by modifying or deleting all their old posts, so that the remaining threads would make no sense. As a result the admins found it necessary to set a time limit for post editing.

Editing as such is now allowed only within that limit (24 hours I think) of original posting, to allow short-term corrections of facts and spelling/grammar, but no long-term modifications to mess up old threads. It is very unfortunate that this strikes against all normal users, but there is no other safe way to handle it.

This editing limitation does not apply to admins, moderators, nor to developers working in an assigned forum of their own.
So you should check out the new 'sticky' thread about "Developer Status", to ensure that you can maintain your releases properly in future.

EDIT 2: I did some more tests, and found that my suspending interrupts after polling for new packets to be handled, the stability issue has been resolved.

That sounds promising in a way, though I'm still very sceptical about the whole idea of using 'loose' UDP packets instead of a proper TCP stream.

In this kind of file transfer, of executable code, nothing is more important than 100% secure error detection and correction.

I was wrong - it probably has got nothing to do with the DMAC interrupts, but with the existing PS2 protocol stacks being unable to keep up with such high speeds.

This became evident when I noticed that the freezes occurred once the 3.5MB/s mark was hit.

And the packet loss that then occurs is one that UDP has no way of recovering from, while TCP does... (though naturally at cost of some speed)

The problem now is that I don't think that anything can be done to get the LWIP protocol stack to achieve such speeds unless it's turned into a single-threaded module - as suspending the interrupts also disables thread cycling.

This is another area where UDP and TCP differ greatly. TCP has built-in safeguards to ensure proper 'ordering' of packet data, even when packets arrive out of order (which is one possible effect of multi-threading). But in UDP there is no relationship at all assumed between different packets, each one being considered a separate transaction independent of all others, so if they arrive out of order, then the data may be accepted with this erroneous order, with fatal results for the games. (Unless the client-server combo re-implements data-reordering, much like it is done by TCP protocol, and with similar timing costs.)

Well, maybe the SMAP driver of this new v0.9.2A server will not be used for OPL in-game, but is definitely good for installing games.

Even so, it will probably work better with TCP (like OPL should use in-game) than it does with UDP (as used by HDLD_svr v0.9x).

Like I've said above, I think much of the problems with HDL v0.9x installs are due to the fact that each HDL game install requires an extremely long-term data stream, making TCP the ideal protocol for it (from the viewpoint of safe transfer), whereas UDP is a very bad choice since it has no inherent support for any form of data stream, forcing the client and server software to implement all the data stream error detection and recovery that really belongs at protocol driver level.

Best regards: dlanor

[fresh]

what kind of issues do you have in mind? my only issue is that network does not start up properly in 0.9.* and i have to init it with uLE first.

I tested some more ps2 consoles and have this problem, too.

Failed:
Booting via fmcb into ule.
Starting elf.
No connection to hdl_dumb.

Worx:
Booting via fmcb into ule.
Starting network via ule.
Starting elf.
Booosted transfer.




Rgds.


PS: Quick tests give running, playable games. No problems in the first 30 minutes...

[SP193]

That is great to hear ! But are you sure ?

I ask this mainly because you never mention fixing the bugs of HDL_Dump (both client and server) which made the original v0.9 corrupt games.

Thanks.

When I made that comment, I was referring to the stability of the PKTDRV network driver within the HDL Dump server v0.9.2A (Not the data reliability issue that the HDL Dump server v0.9.1 always had).

During some of the earlier tests of older prototype versions, the PS2 would freeze up or get disconnected (Not because of my network, but because of a stability issue in the modified PKTDRV).

The basic problem is that TCP protocol was replaced by UDP, which has no long term connection stream, but only separate packets each of which establishes a brand new 'single-packet connection'. This eliminates some of the time-consuming overhead of TCP implementation on IOP, but requires that the client-server coder re-implements all necessary error detection, correction, and recovery methods that TCP protocol has built-in, but which UDP does not have any part of...

I do understand this, as I have studied networking before.

UDP does have corruption detection (It has a checksum in it's header), but does not have any system to prevent the lost of packets and the misordering of packets.

However, packet loss should not occur on reliable networks - like that one I have.

Yes, not everyone has such a connection (So HDL Dump server v0.9.2x isn't for everyone), but it's there for those who can use it.

So HDL_Dump+HDLD_svr v0.9 gave very good speed, but with reliable results only in a LAN 100% free of packet errors.
I presume that the client and server changes caught some of the errors, but nevertheless lots of people got corrupted game installs.
And if those bugs are still present, then it doesn't matter how much faster the DMA methods have allowed the transfer to get.
Because if some packet errors are neither detected nor corrected the resulting game installs will be worthless.

Agreed.

Personally I'd have preferred you to do this work based on one of the traditionally stable versions of HDL_Dump+HDLD_svr, such as v0.8.6

I did that in my earlier tests, but the LWIP protocol stack and SMAP driver combination of that server version wasn't good with DMA support.

Using v0.9 as basis means that it will be impossible to tell whether any problems with the installed games are due to the new code or due to the insufficient error recovery abilities of the v0.9 UDP implementation of the HDL_Dump methods.

You are right, but I had the intention of making a new driver with the design of PKTDRV - but with TCP support included too.

(Probably with the SMAP driver with a tightly integrated protocol stack)

I plan to create a server that supports dual modes - UDP and TCP modes. So that the user can choose between reliability and performance (Only if their networks are good enough to not suffer any packet losses).

That is due to thread sabotage by some users involved in the recent 'hate attacks' against this site, where they attempted to mess up old threads by modifying or deleting all their old posts, so that the remaining threads would make no sense. As a result the admins found it necessary to set a time limit for post editing.

Editing as such is now allowed only within that limit (24 hours I think) of original posting, to allow short-term corrections of facts and spelling/grammar, but no long-term modifications to mess up old threads. It is very unfortunate that this strikes against all normal users, but there is no other safe way to handle it.

This editing limitation does not apply to admins, moderators, nor to developers working in an assigned forum of their own.
So you should check out the new 'sticky' thread about "Developer Status", to ensure that you can maintain your releases properly in future.

Thanks for this piece of information. I will try to contact the appropriate moderator about getting the "developer status".

That sounds promising in a way, though I'm still very sceptical about the whole idea of using 'loose' UDP packets instead of a proper TCP stream.

In this kind of file transfer, of executable code, nothing is more important than 100% secure error detection and correction.

And the packet loss that then occurs is one that UDP has no way of recovering from, while TCP does... (though naturally at cost of some speed)

This is another area where UDP and TCP differ greatly. TCP has built-in safeguards to ensure proper 'ordering' of packet data, even when packets arrive out of order (which is one possible effect of multi-threading). But in UDP there is no relationship at all assumed between different packets, each one being considered a separate transaction independent of all others, so if they arrive out of order, then the data may be accepted with this erroneous order, with fatal results for the games. (Unless the client-server combo re-implements data-reordering, much like it is done by TCP protocol, and with similar timing costs.)

Even so, it will probably work better with TCP (like OPL should use in-game) than it does with UDP (as used by HDLD_svr v0.9x).

I'll try to commit some time (After my exams) to make something good - so that we can all get the best of both worlds.

Like I've said above, I think much of the problems with HDL v0.9x installs are due to the fact that each HDL game install requires an extremely long-term data stream, making TCP the ideal protocol for it (from the viewpoint of safe transfer), whereas UDP is a very bad choice since it has no inherent support for any form of data stream, forcing the client and server software to implement all the data stream error detection and recovery that really belongs at protocol driver level.

Agreed (Otherwise, why didn't UDP replace TCP? ).
But I feel that it should still be an option for users (At their own risk).

I tested some more ps2 consoles and have this problem, too.

Failed:
Booting via fmcb into ule.
Starting elf.
No connection to hdl_dumb.

Worx:
Booting via fmcb into ule.
Starting network via ule.
Starting elf.
Booosted transfer.

Interesting. After my exams, I'll build new drivers for the public to test.

PS: Quick tests give running, playable games. No problems in the first 30 minutes...

My games booted up fine too, so I guess that HDL Dump server v0.9.2A is stable for use - provided that your network never suffers a single packet loss.

[RandQalan]

Best used over crossover or closed network