Open PS2 Loader Project Feature Requests

[izdubar]

Good work gilgamesh88 I think forcing the IO to background would really help your code.

Yes ! It would be better ...

... but I just tried some advice Jimmi gave me today, to tweak the smbman buffer size, and the result is just ... amazing !

You can't ever notice the cover loading now !

According to benchmark (loading N=100 covers), speed is 4.5X faster ...

From 54"50 secs to 12"05 secs ...

PS: just to avoid some expectations ... smbman is the module used only in the gui part of OPL, it will not change the loading time of the games. That concern is the future new LAN module of Jimmi.

[dlanor]

Well, 1 second delay to load the covers, but how many covers do you have? There are people out there with way more games than others.

It may be a really nice feature for people without many games, but for those with 120+ games, the wait may prove to be too much.

Indeed. And even more so when we want to avoid dependency on any USB drive.

I play games from LAN normally anyway, so no USB drive is needed or wanted, especially as I have to remove USB drives from my v15 PS2 in order to be able to use CodeBreaker with it.

But the 64MB MC in slot 1 of that PS2 is quite slow, so attempting to use even the present skins from it (even zipped to save both storage space and load time) makes the program unacceptably slow. So I use no skins at all with it. Having to load 163 icons as well (for the games installed on my LAN fileshare) would make the startup delay completely unacceptable.

Then there is also the matter of the necessary file size for icons of the scalable kind obviously needed here (to suit both static and dynamic menu). Even if only a few kB are needed per icon, multiplying that size by a high game count can still waste tons of MC space as well as loading time. So using such icons must definitely be optional both as a whole and per game.

Best regards: dlanor

[izdubar]

Having to load 163 icons as well (for the games installed on my LAN fileshare) would make the startup delay completely unacceptable.

Then there is also the matter of the necessary file size for icons of the scalable kind obviously needed here (to suit both static and dynamic menu). Even if only a few kB are needed per icon, multiplying that size by a high game count can still waste tons of MC space as well as loading time. So using such icons must definitely be optional both as a whole and per game.

Best regards: dlanor

As said, the icons are loaded dynamically "on-the-fly", only when needed. It would have been suicidal to try loading all games art at start. Neither the devs nor the users would have accepted that ... even not the reduced PS2 memory

So in dynamic mode it will load 3 icons, then only one per move, and in static mode, it load the 10 icons per page in a row, then nothing until you go another page. And same for the cover, it will load it only when the user stop a few millis on a game (so if he's constantly navigating, nothing is loaded for covers).

With Jimmi's tweak on smbman all these are almost unnoticeable.

Indeed. And even more so when we want to avoid dependency on any USB drive.

It tries to load icons and covers from the same media were your games are stored. So if you are in USB mode it will load them from the USB, in ETH from the lan, so no additional dependency between each others mode.

(Nothing either is loaded from the MC, that would have been too much mess on these little thing)

Hope it is more clear now.

Regards.

PS: if you (anyone) want sources, it is more practical by seeing it in action

[JNABK]

PS: if you (anyone) want sources, it is more practical by seeing it in action

Im interested in checking it out, could you send me a PM please.

I've been busy testing IGR for Polo35's newest fixes, so i missed all the new eye-candy!

[dlanor]

So in dynamic mode it will load 3 icons, then only one per move, and in static mode, it load the 10 icons per page in a row, then nothing until you go another page. And same for the cover, it will load it only when the user stop a few millis on a game (so if he's constantly navigating, nothing is loaded for covers).

Even so, the very existence of such icon files should be optional, on a per-icon basis, so that no error state of any kind can come to exist, just because one or more of those icons is missing, nor is it permissible that absence of such icons should lead to any extended search for them, as that too could lead to intolerable delay (when the icons truly do not exist, by user choice).

So far not one word has been said about it being optional to even have those files, which is how I think it MUST be.

Possibly these needs too have been assumed by you as well, but since they have never been explicitly stated someone has to do so, at some point (and better now than later).

With Jimmi's tweak on smbman all these are almost unnoticeable.

On that topic, I'm getting a little worried. Here you mention these tweaks as already in use, even though those changes have not yet made it into any beta sources on the bitbucket site.

Did I miss something ? Is there now some alternate beta site that I have no access to ?
Or did jimmi just hand it over to you requesting some specific tests related to your code ?
I have no objections either way, really. I just want to know what the situation is on this.

I expect a lot from those speed tweaks of jimmi, considering the potential for improvement we discussed earlier. But the real impact of this should not be for the GUI though, but rather for the SMB loader core in-game. And that is of course where speed improvements matter most.

It tries to load icons and covers from the same media were your games are stored. So if you are in USB mode it will load them from the USB, in ETH from the lan, so no additional dependency between each others mode.

So on a USB drive, already quite messy from having to have ALL the game files in the root directory (an awful method) we are now also going to add one icon file to that same root directory for every installed game.

On SMB it is a little better, since that at least does not have to use a root directory, but it will still get rather messy if I have to add in 163 icon files in the same folder as the 559 split-iso files and the ul.cfg file of my 163 game fileshare folder.

And then there is the case of IDE HDD, where it is evidently the intention that we use some method specific to the HDLoader partition system, so that the icons for that medium are not stored as proper files at all, in any normal file system. I suppose this will work 'as advertised', but I don't get a good feeling from this kind of inconsistency. (In fact I'm kind of hoping I misunderstood that part.)

(Nothing either is loaded from the MC, that would have been too much mess on these little thing)

That is good.

Using MC is good for config and setting files that are static in size, but not for file collections that grow with the installed game content, like the icon collection does.

PS: if you (anyone) want sources, it is more practical by seeing it in action

I sure would like having them, especially for the speed-tweaked smbman (hopefully included), which is not available anywhere else as yet (to my knowledge).

Best regarded: dlanor

[izdubar]

Even so, the very existence of such icon files should be optional, on a per-icon basis, so that no error state of any kind can come to exist, just because one or more of those icons is missing, nor is it permissible that absence of such icons should lead to any extended search for them, as that too could lead to intolerable delay (when the icons truly do not exist, by user choice).

So far not one word has been said about it being optional to even have those files, which is how I think it MUST be.

I'm not sure to understand every requirement. So I'll try to reply how I can ...

First as stated in previous post, the whole extra Art feature can be disabled with a new option in the settings "enable Cover Art", along with another one "wide screen" (for which I choose the easy way, that is applying a simple horizontal scale ratio, I should also answer you on the corresponding post, but I miss some courage). In this case there will be no impact on OPL, no searching/loading, no errors, no delays.

Then, in the case the option is enabled. When OPL first tries to display the icons/cover for the current page and selected game, it tries to load the pictures ONCE. Then it will cache the result (either successful or not) and there will be nothing more for each subsequent frame drawing.

On that topic, I'm getting a little worried. Here you mention these tweaks as already in use, even though those changes have not yet made it into any beta sources on the bitbucket site.

Did I miss something ? Is there now some alternate beta site that I have no access to ?
Or did jimmi just hand it over to you requesting some specific tests related to your code ?
I have no objections either way, really. I just want to know what the situation is on this.

No, nothing has been committed, I have no repository, it is only code I do on my side for fun. I read request that others people post, then evaluate what I could do (because I have no PS2 knowledge I can only do thing that are pure C dev, nothing about CORE features related to PS2 API), and try my best on it.

I expect a lot from those speed tweaks of jimmi, considering the potential for improvement we discussed earlier. But the real impact of this should not be for the GUI though, but rather for the SMB loader core in-game. And that is of course where speed improvements matter most.

Completely agree with you. I too have a lots of hope for Jimmi new LAN module. And that's why I said in the concerning post:

PS: just to avoid some expectations ... smbman is the module used only in the gui part of OPL, it will not change the loading time of the games. That concern is the future new LAN module of Jimmi.

The tweak we speak about for the covers is only a very little change (the buffer size), but however the impact is really impressive. I still have no idea if other program could benefit from it.

So on a USB drive, already quite messy from having to have ALL the game files in the root directory (an awful method) we are now also going to add one icon file to that same root directory for every installed game.

On SMB it is a little better, since that at least does not have to use a root directory, but it will still get rather messy if I have to add in 163 icon files in the same folder as the 559 split-iso files and the ul.cfg file of my 163 game fileshare folder.

I'm not sure this is really a justified reproach. I don't mind handling thousand of files (but yes I don't use the sdtandard windows file explorer ). In any case, we can add the use of a "cov/" and "ico/" subfolders if needed. Hmmm that's may be a good idea, moreover in some special case (another mod I think about). My goal is to be able to share easily the Art from one user to another, and to ease the "setup" process.

And then there is the case of IDE HDD, where it is evidently the intention that we use some method specific to the HDLoader partition system, so that the icons for that medium are not stored as proper files at all, in any normal file system. I suppose this will work 'as advertised', but I don't get a good feeling from this kind of inconsistency. (In fact I'm kind of hoping I misunderstood that part.)

That is related to the post with Polo. I've looked at the code for HDD and didn't find as you said a way to write to the "filesystem". At least in the pre-gui part of OPL. I didn't look very deep, but the code in ps2hdd (and hdd_fio) doesn't seems to me to do what I need.

So the only way currently to implement this feature for HDD would be to use Polo suggestion. As I said earlier, I prefer to hope my code got merged one day, and then someone with better knowledges as me on this topic could easily add the missing loading method for Art.

Regards.

PS: sources sent to you

[JNABK]

Well, i finially got my "Gilgamesh88 BETA" working properly.

I created some game icons and placed them into my USB HDD, which i use for both SMB and USB.

Icons:

No lag or any type of load issue have been noticed, and i have been testing many revisions this week. If there was any such concerns about the icons causing a 'lag' or 'excessive loads', then you all can relax.....there is none i could see.

Cover Art:

I have only made one for testing....so i'll get back when i make more.

APPs folder:

This option is also working great, all apps displayed and booted just fine.

Shutdown Option:

Works great.....nothing else to report.

[dlanor]

First as stated in previous post, the whole extra Art feature can be disabled with a new option in the settings "enable Cover Art", along with another one "wide screen" (for which I choose the easy way, that is applying a simple horizontal scale ratio, I should also answer you on the corresponding post, but I miss some courage). In this case there will be no impact on OPL, no searching/loading, no errors, no delays.

Good. That is of course the minimum 'requirement' I intended.

Then, in the case the option is enabled. When OPL first tries to display the icons/cover for the current page and selected game, it tries to load the pictures ONCE. Then it will cache the result (either successful or not) and there will be nothing more for each subsequent frame drawing.

This sounds OK, though it does mean that at least one failed 'fioOpen' attempt will be made for every game on a page that lacks any corresponding icon file. But given the low number of games per page (which really should be raised IMO), the extra delay should be acceptably small.

----- re: new SMB tweaks ('izdubar' quoting 'gilgamesh88' as being himself)
I find it a bit confusing that you keep using two different user names, a practice normally 'frowned' upon by most forum sites, including this one...
I think you should pick one, and then stick with that instead.

The tweak we speak about for the covers is only a very little change (the buffer size), but however the impact is really impressive. I still have no idea if other program could benefit from it.

That's very hard to say as so few programs use SMB at all to date, and these all using different versions of those drivers. (AFAIK, OPL, SMS and myPS2 all use different drivers).

But I have long wanted to add SMB capability to uLE as well, though that has been held back by two things. Inaccessibility of the latest SMS drivers, and the total lack of write access to PC files with those drivers. But with the new work of Jimmi in this area the first problem should be eliminated, giving us fully open-source and up-to-date SMB drivers at long last. Even without write ability (which may perhaps be added later) that would still be a good complement to the current networking abilities of uLE.

In any case, we can add the use of a "cov/" and "ico/" subfolders if needed. Hmmm that's may be a good idea, moreover in some special case (another mod I think about). My goal is to be able to share easily the Art from one user to another, and to ease the "setup" process.

I'm sure that most people will find it easier to handle things with a proper folder structure to keep different things separate, such as the icon and cover files.

----- re: HDD-specific storage methods

That is related to the post with Polo. I've looked at the code for HDD and didn't find as you said a way to write to the "filesystem".

True. While HDL games are falsely flagged as being PFS filesystems, they are not accessible as such. They are fully HDL-specific raw sector chunks without any individual file access possible at all, not even to the whole chunk as such, as such a chunk has no normal PFS directory system (but only an HDL-specific header struct).

At least in the pre-gui part of OPL. I didn't look very deep, but the code in ps2hdd (and hdd_fio) doesn't seems to me to do what I need.

Since the HDL partition structure is completely HDL-specific, the only standard lib functions useful are those for raw sector access. Interpreting the content must be done using functions and data structs defined specifically for HDL (or the HDD core of OPL) and the HDL game installers.

So the only way currently to implement this feature for HDD would be to use Polo suggestion.

Not at all. That only applies if you insist on using the same partitions for the graphics as used for the games, which I do not find appropriate, due to the total inconsistency of that storage method as compared with those used for other media.

The most natural way of all would be to create cover and icon subfolders in some standard PFS partition, where all of the cover and icon files can be stored using the normal PFS filesystem methods (standard fileXio functions only).

As I said earlier, I prefer to hope my code got merged one day, and then someone with better knowledges as me on this topic could easily add the missing loading method for Art.

Sure, but doing it the way I suggest for HDD should be very easy for you to do as well, using methods almost identical to what you use for the USB and SMB media. The only real difference is that you must use fileXio instead of fio functions, and will need to mount an extra partition to access the HDD graphics. But the choice of partition is not in doubt, as the only logical choice is the same "HDLoader Settings" partition already used for every HDL installation in existence.

So all you need do is to define a couple of subfolders for the cover and icon files in that partition, and then use those in almost exactly the same way as for USB and SMB.

PS: sources sent to you

Yes thank you, I have them now and will proceed with some testing tomorrow.

Best regards: dlanor

[dlanor]

@izdubar:
Despite what I said at the end of my previous post, about waiting until tomorrow, I decided to go ahead tonight and test your modification to rev 239, with game-specific icon and cover graphics display. It worked pretty well overall, defaulting properly without noticeable delays when files were absent (like I've been nagging about ).

The only negative result was when I had made an erroneously scaled icon file, accidentally using 140x140 instead of 64x64. (I simply missed the final scaling step before conversion to raw.) With that icon file present in the SMB fileshare folder OPL simply froze, requiring a reset to regain control of the console. And since the icon belonged to a game that was to be displayed in the first page of the collection, and SMB was the default game list for display, OPL would also freeze up whenever restarted with this icon present. (Unless overriding the SMB access somehow.)

It would be much better if the icon handling/drawing routines could be recoded such that erroneous filesize does not result in the program freezing, but instead merely causes that icon file to be rejected as invalid without even trying to process it as graphic data.

After all, it only takes a simple 'seek' check for filesize and comparison to the known valid size, which will be the same for all image files with the same height and width as the 'raw' format has no compression.

Best regards: dlanor

[izdubar]

----- re: new SMB tweaks ('izdubar' quoting 'gilgamesh88' as being himself)
I find it a bit confusing that you keep using two different user names, a practice normally 'frowned' upon by most forum sites, including this one...
I think you should pick one, and then stick with that instead.

Yes, short story: I now made my choice, and it is Izdubar
Sorry for the inconvenience, all the old "quotes" wil still reflect my previous account.

But with the new work of Jimmi in this area the first problem should be eliminated, giving us fully open-source and up-to-date SMB drivers at long last. Even without write ability (which may perhaps be added later) that would still be a good complement to the current networking abilities of uLE.

I thought Jimmi is rewriting the low layers part of the network API (that is tcp/ip, smap, ...), he is also working on the smb part ?

I'm sure that most people will find it easier to handle things with a proper folder structure to keep different things separate, such as the icon and cover files.

Yes, I agree, and it is now changed to your suggestion.

The most natural way of all would be to create cover and icon subfolders in some standard PFS partition, where all of the cover and icon files can be stored using the normal PFS filesystem methods (standard fileXio functions only).

Sure, but doing it the way I suggest for HDD should be very easy for you to do as well, using methods almost identical to what you use for the USB and SMB media. The only real difference is that you must use fileXio instead of fio functions, and will need to mount an extra partition to access the HDD graphics. But the choice of partition is not in doubt, as the only logical choice is the same "HDLoader Settings" partition already used for every HDL installation in existence.

That were the informations I was looking for when I asked

I'm already using fileXio, but I didn't know the HDD layout, and how to build a valid path. Should it be:

"hdd0:/+HDLoader Settings/..."

EDIT: Ok, we definitely not use fileXio. I thought fileio was for standard device access (the one by default, so mc, cdfs), and fileXio for extended device, with custom modules that got registered, like smb: and mass:, but now I understood.

I think I found all the code needed in ps2sdk\ee\loader\src\loader.c (it is also in SMS, ULE, HD Project, ...), it is not that hard to add it, but it would require a little refactoring. Another problem is that we don't have the game code for HDD, we must either stick to the game name, or the partition name. Both are defined by the user I fear ?! That mean no standard icons/covers naming

Is the game code saved somewhere for the HDD games, and can it be retrieved ...?

/EDIT

The only negative result was when I had made an erroneously scaled icon file, accidentally using 140x140 instead of 64x64. With that icon file present in the SMB fileshare folder OPL simply froze

It would be much better if the icon handling/drawing routines could be recoded such that erroneous filesize does not result in the program freezing, but instead merely causes that icon file to be rejected as invalid without even trying to process it as graphic data.

After all, it only takes a simple 'seek' check for filesize and comparison to the known valid size, which will be the same for all image files with the same height and width as the 'raw' format has no compression.

Yes, I too thought about this, and Volca too, because there is a comment on this subject.

I even wanted to implement the filesize check you mentioned, doing simple math (width x height x 4), but then I discovered that some of my (working) custom icons were not doing 16384 bytes for some reason !! Until I know why, I put this check on a side ...