New modified SMAP driver with FULL DMA support + HDL Dump svr 0.8.7

Quote:

---

Originally Posted by volca

Hey, sorry for the delay.

I left the effort sometime at the end of last year, solely for the lack of time. The state I left it was that I still didn't come up with a way to stop the freezes. I think the culprit is double sema locking, so disabling interrupts instead could help. The only combination that did work was using the SMS assembly fifo handling with the threaded SMAP code, which produced some 0.5 Mb/s speed gain, probably due to the fact that it does not block the IOP so much.

I thought about the way to get from the mess that it is currently in. I came up with a plan I currently have little time to realize:
1. Write a threaded SMAP.irx replacement, from scratch, that does not depend on SMSTCPIP.irx in any way.
2. Tune this on a piece of packet generating code - prepare a blocking read method that'd replace the one in SMSTCPIP at the end of the journey. In this state the PS2 should be able to reply to some hand made packets.
3. Rewrite the ARP, ICMP and UDP parts of the stack, so it will be able to communicate by UDP, reply to ping packets and manage ARP requests.
4. Write TCP stack on top of this.

By the end this should be capable doing client side of TCP.

I know jimmi has started such a rewrite, but without threading the SMAP - it may still be viable alternative, all we need is DMA code in single thread and working blocking read impl. Problem was doing DMA in interrupt context - namely dev9DmaTransfer which uses non-interrupt Sema calls, if not because of other problems.

Also, I think write operations should borrow buffers from the packet driver and write into those directly - should save some time.

---

Waow, nice answer ;)

I'll reply longer later, I must leave now !

Sure, no problem :)

Going a bit more into detail - the plan was to write a monolithic network irx module - the same concept jimmi started. Difference is that it should handle sockets and DMA (so it will probably have to be threaded - or we'd have to introduce something like "iDev9DmaTransfer", if it is even viable to do so...

From ground up there'd be:
phy - handling packet send/receive on SMAP
pkt - allocator for packets - circular buffer for input/output of packets - sequential FIFO with push/pull and notifications via callbacks.
que
- queue handling code - queueing ingoing and outgoing packets in pkt buffers - allocates packets in pkt, handles send/receive in phy.
- here we could implement localhost interface, but it seems it would be of no use...

- and a prototype for the blocking read/write calls that are handled via the queue (later, each socket would have it's own wait object - event flag)

after this is done and working, it'd be probably time to introduce sockets and start implementing the network stack...

Ok ... I try to correctly understand all these technical background :)

So we have two distinct part in this stuff (not directly related to your plan/explanation, it is just for me, to sum up):

1. First there is the device driver for the physical unit (network adapter), which is called SMAP. The current implementation depends on SMSTCPIP (is it related to SMS - Simple Media System - from EEUG ?).
2. Then we need a TCPIP stack. Current one used is lwip.

I like your plan, and hope you will find time to make progress on this, but I realize it involve lot's of work.

I just have a suggestion, about the lightweight TCPIP stack, as you know more on this subject as me, you could maybe read this article:

Inside the uIP Stack | Dr Dobb's

I've heard of this other TCPIP stack that may be promising, easier, lighter to implement for our goals.

To sum up the diff between lwip/uip:


uIP
Great if youre only wanting something basic like a bootloader
Small footprint.
Uses polling
No DHCP 'out of the box'
Poor UDP support

lwIP
Fully interrupt driven
Includes DHCP with failover AutoIP
UDP with multicast
Plus lot's of more

Oh dear me. So many things have happened while I was away. :lol:

Quote:

---

Originally Posted by hominem.te.esse

@volca, SP193

Any news, facts, story about this epic challenge, just to keep us in suspense ;) ?

---

Uh. No. :X
Maybe only that I am still occupied with trying to prevent my grades from sliding and my attention has been once again redirected somewhere else? :D

Quote:

---

Originally Posted by volca

Hey, sorry for the delay.

I left the effort sometime at the end of last year, solely for the lack of time. The state I left it was that I still didn't come up with a way to stop the freezes. I think the culprit is double sema locking, so disabling interrupts instead could help. The only combination that did work was using the SMS assembly fifo handling with the threaded SMAP code, which produced some 0.5 Mb/s speed gain, probably due to the fact that it does not block the IOP so much.

---

You found the problem? Wow. :applaud:

But why do the semaphores cause the freeze? Shouldn't they be preventing deadlocks from occurring to begin with? Unless... a circular wait occurred. :X

Quote:

---

Originally Posted by volca

I thought about the way to get from the mess that it is currently in. I came up with a plan I currently have little time to realize:
1. Write a threaded SMAP.irx replacement, from scratch, that does not depend on SMSTCPIP.irx in any way.
2. Tune this on a piece of packet generating code - prepare a blocking read method that'd replace the one in SMSTCPIP at the end of the journey. In this state the PS2 should be able to reply to some hand made packets.
3. Rewrite the ARP, ICMP and UDP parts of the stack, so it will be able to communicate by UDP, reply to ping packets and manage ARP requests.
4. Write TCP stack on top of this.

By the end this should be capable doing client side of TCP.

---

#3 can be ripped from PKTDRV, since I saw that it has uber simple implementations of the UDP and ICMP (and ARP?) protocols. :lol:

Quote:

---

Originally Posted by volca

I know jimmi has started such a rewrite, but without threading the SMAP - it may still be viable alternative, all we need is DMA code in single thread and working blocking read impl. Problem was doing DMA in interrupt context - namely dev9DmaTransfer which uses non-interrupt Sema calls, if not because of other problems.

---

You could copy what I did in my buggy DMA-enabled drivers. ;)

1. Keep polling the DMA channel's CHCR register until it's free.
2. Configure the DMA channel and SMAP hardware and set up DMA transfers (Remember that the size must be in multiples of whatever block size you set or it will fail on 99% of the PS2s out there if it still works!).
3. Copy out the remaining data with the IOP.
4. ???
5. Profit. :lol:

It should work, but this was probably not how Sony intended the DMA channel to be used in their PS2LINUX implemetation of the DEV9 driver.

Now to think about it, the original PS2SDK DEV9 driver polled the CHCR register of the DMA channel in a loop, instead of using semaphores... which make sense if the programmer would be invoking that function from within the interrupt context.

Quote:

---

Originally Posted by volca

Also, I think write operations should borrow buffers from the packet driver and write into those directly - should save some time.

---

Agreed. ;)

Why bother re-writing existing code that works?

I used to do that though, since I loved using everything that was made by myself. :lol:

Hey, all good points guys. Just a quick reply as I don't have the time...

Hmm, the dev9DmaTransfer only waits on dma_lock_sem, so polling CHCR alone is not enough - some other irx module could try to call dev9DmaTransfer causing unexpected behavior. Maybe dev9RegisterPreDmaCb could be used to introduce CHCR polling though - I'll have to verify that.

I could use the PKTDRV, I'll have to review if it is bendable to the usage we need - namely the blocking read and the introduction of sockets (really, the arp, icmp and udp code is easy to be written, it is tcp which is a bit of a nightmare).

@hominem.te.esse: i saw some mentions of uIp, I'll take a look, thanks.

I'm not promissing anything - I don't know if I'll find enough time for this :)

While it is true that dev9DmaTransfer() uses a semaphore to control the usage of the DMA channel, polling the DMA channel alone *should* be safe since the IOP will essentially become a single-threaded system while within an interrupt context.

The only thing left to do is to ensure that the DMA channel is free before attempting to send it commands.
(And of course, remember to ensure that it completes it's operations before returning from the interrupt context! So that the regular threads will be not even notice that the DMA channel was "hijacked")

However, yes, if something else causes the IOP to run another function that also tries to use the DMA channel (via dev9DmaChannel()), the result will be really unpredictable.

But, do try to get a truly threaded approach to work first. This is really like a hack... and anything can happen (E.g. the IOP can miss interrupt assertions) and might result in a freeze.

UPDATE: I've made the LWIP v1.3.2 stack as stable as the DMA-enabled LWIP v1.1.1 stack, with DMA transfer support. (But the throughput is terrible - 0.60MB/s at most)

One thing that seems to impact stability is how a Packet Buffer (PBUF) is allocated for incoming packets (Within the SMAP driver).

I believe that the PS2 can get spammed by the other computers on the network, and the IOP is too slow at clearing the received packets. The result would be that the PBUF pool will run out of space.

The solution? Add a while loop around the call to pbuf_alloc(), but remember to call DelayThread() if pbuf_alloc() returns NULL, so that the IOP can run the other threads (Otherwise, the PBUFs won't get cleared).

It worsens when there are more computers on the network, as that probably means that more packets are being sent to the PS2 (Even though the PS2 is not going to process those packets).

It seems like LWIP v1.1.1 might have a bug, hence pbuf_alloc() causes freezes under heavy traffic, when the PBUF PBUF_POOL type is specified. Changing it to PBUF_RAM or suspending interrupts before the call fo pbuf_alloc() seems to solve that problem.
The problem doesn't seem to (really?) exist in LWIP v1.3.2.

The problem now is that the old problem with the PS2 locking up (Sometimes, with the orange HDD-activity LED indicator shining brightly) is still there. There might be a bug in the DEV9 driver too.

I believe that the IOP is really too slow to operate the LWIP stack too reliability, so I'll probably try to run the stack on the EE instead.
When the PS2 is pinged, it seems like it's suffering from intense lag, as the latency can be as high as 350ms....

EDIT: Oh yea, I also stripped quite a lot of stuff from the SMAP driver, since quite a lot of the things that were removed were already done by the DEV9 driver.

Things there were removed included:
1. The interrupt-handling mechanism was replaced. Now, the individual interrupt events have their own interrupt handlers. That means that the IOP has less conditional statements to evaluate.
2. The SPEED won't be polled for interrupt event changes in the SMAP driver, since it's already done by the DEV9 driver (It will poll the SPEED device for interrupt events and call the respective interrupt handlers, so no interrupt events will be missed).
3. Interrupts that won't be handled by the SMAP driver are now disabled. These interrupts are the Descriptor Not Valid (DNV) interrupts (What do those interrupts signify anyway?).
4. The weird packet queueing mechanism was removed. Why did outgoing packets have to be queued in the SMAP driver? :confused:
5. <EDIT> Removed code that read data from the EEPROM. Replaced all that crap with one call to dev9GetEEPROM().

So you mean, that any DMA enabled version of latest lwip implementation are awfully slow, even slower that the "bulked" version used by default in OPL ? And that, not even with real condition (ingame) but with a simple environment test I suppose (sending/reading packets) ?

I can't understand why every attempt to replace some overbloated code (the current implementation of IP stack/SMAP/...) by some tweaked and optimized code (even with DMA) fails ... Long ago did Jimmi tried to enhanced the ETH performance with a complete rewrite and the results weren't what expected either ...

I'm not criticizing your task, really not, as I'll never be able to specialize myself into such low-level programming, it is just that the whole story is despairing ...

> Add a while loop around the call to pbuf_alloc(), but remember to call DelayThread() if pbuf_alloc() returns NULL, so that the IOP can run the other threads (Otherwise, the PBUFs won't get cleared).

Does this mean that we are constantly polling the device, to acquire packets ? There is no blocking read IO, saving cpu cycle ?

I thought that the whole point of using lwip was to not use polling, where as uip does ... Maybe you could have a look at uip (see my post above, the website for uip looks a bit scary) ?

Anyway, I hope you will continue your work on this, I think you can manage to successfully achieve this feat :)

Quote:

---

Originally Posted by hominem.te.esse

So you mean, that any DMA enabled version of latest lwip implementation are awfully slow, even slower that the "bulked" version used by default in OPL ? And that, not even with real condition (ingame) but with a simple environment test I suppose (sending/reading packets) ?

---

Yes. However, the tests weren't exactly simple data-transfer tests. They were actually real HDLoader game installation processes, done with the OPL HDLDump server.

There is actually a simple reason for that: The IOP is too slow.

Without DMA support, data used to be shoved out of the FIFOs within an interrupt context, which means that the SMAP driver has the entire IOP to itself (37MHz) to look for a nice spot in the Packet Buffer (PBUF) pool to store the packet and to copy the data from the FIFOs to that spot.

Now, with DMA support (by triggering DMA transfers via the DEV9 module), transfers have to be invoked outside of the interrupt context. That means that all threads will be running (And competing for CPU time)... and the DMA transfer might not start too soon.

We can cheat the system though: Don't use the DEV9 DMA function at all. Manually trigger and control the DMA channel with DMACMAN (Within the interrupt context), but then we still have to take care of the problem with the network stack running out of space. :(

If you do that properly (And assuming that the problems within the network stacks get fixed), you can get much better speeds than the traditional "PIO-only" SMAP drivers.

I just discovered not too long ago that Sony doesn't have their network stack on the IOP, which means that it probably wasn't designed to run something as heavy as a TCP/IP network stack (Otherwise, Sony probably would have done that).

Quote:

---

Originally Posted by hominem.te.esse

I can't understand why every attempt to replace some overbloated code (the current implementation of IP stack/SMAP/...) by some tweaked and optimized code (even with DMA) fails ... Long ago did Jimmi tried to enhanced the ETH performance with a complete rewrite and the results weren't what expected either ...

---

The IOP is really a simple beast that seems to be designed to handle only simple I/O operations. :(
Modifying modules to achieve tasks that require a bit more CPU power is uber difficult.

One reason why changing code bloated might cause things to stop working is.... the IOP is too slow. D:
It's so slow that synchronization is important, in order to prevent the IOP from causing crashes when buffers overflow.

It's also so slow that interrupts have to be handled and polled properly after an interrupt assertion is detected. I remember wasting WEEKS trying to get the i.Link driver stable on the IOP - and it kept missing interrupt assertions, especially under heavy load (And caused random freezes).

Since the IOP has to be always involved with a full IOP-side network stack, it's probably best to not expect too much.
(E.g. We got DMA support working with the i.Link driver, but we cannot hit 40MB/s with it because of the IOP. Since we need it to byte-swap data, data transfer rates have dipped to around 7MB/s.)

Quote:

---

Originally Posted by hominem.te.esse

I'm not criticizing your task, really not, as I'll never be able to specialize myself into such low-level programming, it is just that the whole story is despairing ...

---

I understand. :)

But it's really quite sad to know how we are always being limited by the IOP.... :mad:

Quote:

---

Originally Posted by hominem.te.esse

> Add a while loop around the call to pbuf_alloc(), but remember to call DelayThread() if pbuf_alloc() returns NULL, so that the IOP can run the other threads (Otherwise, the PBUFs won't get cleared).

Does this mean that we are constantly polling the device, to acquire packets ? There is no blocking read IO, saving cpu cycle ?

---

No. What I meant in that sentence was that we need to poll the network stack for space to store the packet. (I'll elaborate on why this is necessary below)

Acquiring packets from the hardware is not a problem. The problem is storing the packet for processing by the network stack.
(On a side note: For device interrupt assertions and DMA transfers, we have functions that can be used to put the thread to sleep without blocking other threads, until the appropriate event occurs... so no polling is required there)

Yes, the IOP runs so slowly that data can end up being read in from the SMAP faster than the stack can clear them. No, there is no way to fix that... and it's bad since Ethernet wasn't designed to have packet/host filtering in the hardware like IEEE1394 does (Which means that the hardware will accept all packets from all hosts, and it's up to the software to reject them one by one).

So even if it has 128MB (64x what the IOP has :lol:) of space to store packets, packets can stream in faster than the network stack can process them, causing the stack to eventually run out of space.

What happens then is that the SMAP driver usually drops the packet. :(

Quote:

---

Originally Posted by hominem.te.esse

I thought that the whole point of using lwip was to not use polling, where as uip does ... Maybe you could have a look at uip (see my post above, the website for uip looks a bit scary) ?

---

You are right. LWIP is certainly designed for use with systems that have an OS.

uIP? Isn't that the stack that is less-featured but smaller than LWIP? I thought of changing stacks... but then now I think that LWIP is the best for the complicated machine that the PS2 is (The IOP, although simple, has a lot of features that are found in real a OS like threading and semaphores).

I might take a look at uIP someday though, since the IOP needs a simple network stack that doesn't require too much CPU power.

Quote:

---

Originally Posted by hominem.te.esse

Anyway, I hope you will continue your work on this, I think you can manage to successfully achieve this feat :)

---

Thanks. :)

I hope that I haven't made my response too misleading this round. I spend quite a while looking through it, but I think that I need to go now. :(

It's probably best that I mention now that I'm going to try to do what Sony did first, and port the LWIP stack onto the EE... since game installations require as fast network connections as possible.

SP193,

I'm interested in trying your hdld_svr 0.9.2 as the reports of upwards of 5MB/s installation makes me salivate :P However, I've been unable to get any hdl_dump/hdl_dumb to connect to my 39001 running any hdld_svr ELFs, whether yours or not. Any ideas why? I can successfully connect to OPL's HDL server with no problems on the exact same setup (with roughly 1.5MB/s installation rates). I've tried launching PS2Net from within uLE prior to loading the hdld_svr ELFs with no change in behavior. Of course, if I need to install/remove a lot of games I'll choose to pull the drive, but I'd rather not do that as much as I can and can deal with a 66-75% reduction in speed for the convenience (although 15-20MB/s installs are really nice sometimes :P)