Dummy files

[agalloch]

Hey all,
Can removing dummy files increase performance with open loader?

And is there an automated way to remove dummy files while re ripping my collection? My NAS is filling up quick and i'm sure at least 50GB is in dummy files.

EDIT: I also use a 120gb usb harddrive when i cant access my network. (different room)

[nemesis01]

I have only done this myself once, this week actually with Soul Calibur 2 as although I have about 900gb free space I don't like to waste it. Managed to lop around 3gb off by replacing dummy files with 1kb files, didn't seem to improve performance but it sure did save space. I did all this manually, not sure about an automatic method.

[dlanor]

Hey all,
Can removing dummy files increase performance with open loader?

No it shouldn't.

The dummy files of an ISO should never be accessed by the game, and therefore the OPL routines will never transfer them. OPL only transfers the file content from the ISO that the game code requests to be accessed (by normal CDVD access functions). So the only effect of dummy files with OPL is to increase the storage space required for those games. This is a regrettable loss, but one I think we should accept.

And is there an automated way to remove dummy files while re ripping my collection?

No.

The only safe way to do it is to know EVERYTHING about how a game handles all file location. If even a single file on an ISO uses fixed LBA addressing then the removal of a dummy file may break that game. And there is NO way to automatically find all instances of fixed LBA references, in order to change them.

For games that others have already developed 'ripkits' for it may be done safely, but those kits are usualy focused on making games fit on CD, and for most games this means reducing the quality of various game features or even eliminating some (FMVs etc). And in any case all such kits are specific to one game or just a small group of related games. There is no universal ripkit, and there never can be any, as that just isn't possible.

My NAS is filling up quick and i'm sure at least 50GB is in dummy files.

I'm sure there is a lot of dummy file space used in my own OPL fileshare too, currently using 640 GB (of a 750 GB partition), but I have no intention of patching ISOs to change that.

Doing it safely would require intensive research on every single ISO to be patched, and the savings possible just aren't worth that kind of work investment for me.

Best regards: dlanor

[TnA]

The HD, where my share is stored at is formatted in NTFS, so I set the folder-permissions to simply compress all files, which I store in it. ;-)

My server is only a Pentium III (3) 733/766MHz with 100MBit/s-LAN and it serves enough data, to simultaniusly play from 2 PS2s, so that fits well for me, not sure if it will help you.

[dlanor]

The HD, where my share is stored at is formatted in NTFS, so I set the folder-permissions to simply compress all files, which I store in it. ;-)

Hmm. I didn't think of that.
Normally I never use such compression, but for the PS2 stuff it really does make sense.

The PS2 networking speed is so relatively slow, compared to any modern PC, that there should be no problem serving the ISO and VMC data even in compressed form, unless the same computer is doing some other CPU-intensive tasks at the time.

And it is of course also possible to set compression to be used only for the subfolders "ART", "CD" and "DVD", leaving "VMC" unaffected. Dynamic compression of files opened for read_write is something I really don't like, so I'd rather avoid that, and use it only for the read_only ISO files. And since all except one game of my setup now use normal ISOs, I should gain tons of space this way.

The partition I use for my fileshare is rapidly filling up (and I still have uninstalled originals), with now only 27GB remaining of the 750GB total in that partition, 643GB of which is already used by the PS2SMB fileshare. So this kind of compression could really help.

I just wonder how long it will take to do the initial compression of almost 640GB...?

Best regards: dlanor

[TnA]

Hmm. I didn't think of that.
Normally I never use such compression, but for the PS2 stuff it really does make sense.

Indeed. I only have a few games installed, but came down from ~40GB to 17-18GB.

The PS2 networking speed is so relatively slow, compared to any modern PC, that there should be no problem serving the ISO and VMC data even in compressed form, unless the same computer is doing some other CPU-intensive tasks at the time.

I could only test it with 2 PS2s as of yet (were my testgame for videos and multiple accesses to one ISO, was FFX), but the loading-time didn't decrease. But it might do so, in some compressed parts.
However, I didn't noticed it as of yet on my equipment.

And it is of course also possible to set compression to be used only for the subfolders "ART", "CD" and "DVD", leaving "VMC" unaffected. Dynamic compression of files opened for read_write is something I really don't like, so I'd rather avoid that, and use it only for the read_only ISO files.

I'll handle it the same way, once I start using VMC.
Currently I do not use that feature, because both the SMB-Module and the VMC-Module are kind of experimental (especially their "write-features") and I avoid using it in itself, once it is considered "stable".

And since all except one game of my setup now use normal ISOs, I should gain tons of space this way.

I think so, too.

The partition I use for my fileshare is rapidly filling up (and I still have uninstalled originals), with now only 27GB remaining of the 750GB total in that partition, 643GB of which is already used by the PS2SMB fileshare. So this kind of compression could really help.

I'm really interested, how much the needed storage-space went down, if you might do it.

I just wonder how long it will take to do the initial compression of almost 640GB...?

Best regards: dlanor

Well, you don't need to do all at once, but only "a few" to test your equipment with it.

best regards TnA

[nemesis01]

Hmm. I didn't think of that.

I just wonder how long it will take to do the initial compression of almost 640GB...?

Since I only have 4 games installed totalling 15.8GB gave this a try, to see how much space it will save & to see how long it takes.

It took roughly 15 minutes to compress down my 4 ISO files, I saved 3.9GB which is not too bad considering. I guess it also depends on the game how well it would normally compress with 7Zip or WinRar, if the disk is filled with dummy files you would probably save a lot more space, Soul Calibur 2 would probably compress down very well using this method.

[dlanor]

Indeed. I only have a few games installed, but came down from ~40GB to 17-18GB.

That compression rate is quite impressive, but with only a few games it is hard to know if that is representative for the average game. Some of the more advanced games may already use compressed data to make room for everything on a normal DVD5 disc, and such games will give very bad NTFS compression ratios (like placing a RAR file inside a ZIP)

I could only test it with 2 PS2s as of yet (were my testgame for videos and multiple accesses to one ISO, was FFX), but the loading-time didn't decrease. But it might do so, in some compressed parts.
However, I didn't noticed it as of yet on my equipment.

I think it would take a rather slow PC not to keep up with the transfer rate of PS2 networking.

In running a single instance of FFX I have seen transfer burst peaks of around 12.5 Mbps at the very most, and that only for a brief fraction of a second, in complete reinitialization of game data, such as when loading a gamesave. For area transitions I see similar even briefer burst of around 6 to 8 Mbps, and for normal movement without special encounters the transfers needed are virtually none, as local background and soundtrack are already in memory.

Naturally this can vary greatly between games, and those less efficiently coded may need constant disc access (thus network transfer) for all kinds of stuff. But even then we can expect the absolute peak transfers to go no higher than around 12.5 Mbps due to limitations of the PS2.

I'm really interested, how much the needed storage-space went down, if you might do it.

I plan to try it, though I haven't quite decided how best to do it yet.

Well, you don't need to do all at once, but only "a few" to test your equipment with it.

Yes, I think the best way to do it is to keep the old "CD" and "DVD" folders uncompressed but rename them to something else, like adding "_old" as name suffix. Then I create new "CD" and "DVD" folders with the compression attribute set, and start moving some ISOs from "CD_old" and "DVD_old" into the new folders, which should cause them to be compressed as part of the move.

This way I can do it as gradually as I wish, guided by early results.

Since compression must be expected to take at least as long time as copying similar amounts of data would, I must expect compression of 640 GB to take considerable time. For normal copying from one drive to another I get appx 44MB/s (around 1.5 min for a 4GB ISO or 23 seconds per GB). But this compression will use the same drive for both source and destination which will slow things down, in addition to the compression work. But even if I figure on getting the same rate undiminished, that still comes to almost 15,000 seconds, which is over 4 hours. And it could take considerably longer...

So I'll try it just with a few ISOs at a time to begin with, to see how that goes.

Some tests later...:
Apparently normal moving does not work.

The compression only happens if I 'Copy' rather than 'Move' the files. So I am automatically forced to do this very gradually at first, as only 27 GB remain free on the drive at this time, so that is the largest 'chunk' of ISOs I can use initially, until some more space has been released as I delete originals of the already compressed ISOs. I'm a bit nervous about this deletion of original rips too, as it would mean a HUGE amount of work if I had to redo the ripping, in case something goes wrong with the compressed files. Oh well, nothing risked, nothing gained, so here I go...

Best regards: dlanor

[dlsmd]

That compression rate is quite impressive, but with only a few games it is hard to know if that is representative for the average game. Some of the more advanced games may already use compressed data to make room for everything on a normal DVD5 disc, and such games will give very bad NTFS compression ratios (like placing a RAR file inside a ZIP)

I think it would take a rather slow PC not to keep up with the transfer rate of PS2 networking.

In running a single instance of FFX I have seen transfer burst peaks of around 12.5 Mbps at the very most, and that only for a brief fraction of a second, in complete reinitialization of game data, such as when loading a gamesave. For area transitions I see similar even briefer burst of around 6 to 8 Mbps, and for normal movement without special encounters the transfers needed are virtually none, as local background and soundtrack are already in memory.

Naturally this can vary greatly between games, and those less efficiently coded may need constant disc access (thus network transfer) for all kinds of stuff. But even then we can expect the absolute peak transfers to go no higher than around 12.5 Mbps due to limitations of the PS2.

I plan to try it, though I haven't quite decided how best to do it yet.

Yes, I think the best way to do it is to keep the old "CD" and "DVD" folders uncompressed but rename them to something else, like adding "_old" as name suffix. Then I create new "CD" and "DVD" folders with the compression attribute set, and start moving some ISOs from "CD_old" and "DVD_old" into the new folders, which should cause them to be compressed as part of the move.

This way I can do it as gradually as I wish, guided by early results.

Since compression must be expected to take at least as long time as copying similar amounts of data would, I must expect compression of 640 GB to take considerable time. For normal copying from one drive to another I get appx 44MB/s (around 1.5 min for a 4GB ISO or 23 seconds per GB). But this compression will use the same drive for both source and destination which will slow things down, in addition to the compression work. But even if I figure on getting the same rate undiminished, that still comes to almost 15,000 seconds, which is over 4 hours. And it could take considerably longer...

So I'll try it just with a few ISOs at a time to begin with, to see how that goes.

Some tests later...:
Apparently normal moving does not work.

The compression only happens if I 'Copy' rather than 'Move' the files. So I am automatically forced to do this very gradually at first, as only 27 GB remain free on the drive at this time, so that is the largest 'chunk' of ISOs I can use initially, until some more space has been released as I delete originals of the already compressed ISOs. I'm a bit nervous about this deletion of original rips too, as it would mean a HUGE amount of work if I had to redo the ripping, in case something goes wrong with the compressed files. Oh well, nothing risked, nothing gained, so here I go...

Best regards: dlanor

let us know how it turns out

[dlanor]

One thing I can say already is that the variation in compression efficiency is HUGE.

For some few games over 50% of their total space can be regained by compression, meaning that they end up using less than half their original storage space, like TnA described.


But for some games more efficiently made, like FFX, it is an entirely different story...

My ISO of FFX(UK)==SCES_504.90 now shows the following file properties

Size: 4.28 GB (4,598,169,600 bytes)
Size on disk: 4.25 GB (4,572,221,440 bytes)

So the space regained was 25,948,160 bytes, which is slightly less than 25MB, and less than one percent of the original total size. (appx 0.56%)

So if all game discs were made that efficiently, there would be no point in compressing at all, but that is not the case.
FFX is quite unusual in that regard (as it is in many other ways too).

Best regards: dlanor