Dummy files

[agalloch]

Hey all,
Can removing dummy files increase performance with open loader?

And is there an automated way to remove dummy files while re ripping my collection? My NAS is filling up quick and i'm sure at least 50GB is in dummy files.

EDIT: I also use a 120gb usb harddrive when i cant access my network. (different room)

[nemesis01]

I have only done this myself once, this week actually with Soul Calibur 2 as although I have about 900gb free space I don't like to waste it. Managed to lop around 3gb off by replacing dummy files with 1kb files, didn't seem to improve performance but it sure did save space. I did all this manually, not sure about an automatic method.

[dlanor]

Originally Posted by agalloch

Hey all,
Can removing dummy files increase performance with open loader?

No it shouldn't.

The dummy files of an ISO should never be accessed by the game, and therefore the OPL routines will never transfer them. OPL only transfers the file content from the ISO that the game code requests to be accessed (by normal CDVD access functions). So the only effect of dummy files with OPL is to increase the storage space required for those games. This is a regrettable loss, but one I think we should accept.

And is there an automated way to remove dummy files while re ripping my collection?

No.

The only safe way to do it is to know EVERYTHING about how a game handles all file location. If even a single file on an ISO uses fixed LBA addressing then the removal of a dummy file may break that game. And there is NO way to automatically find all instances of fixed LBA references, in order to change them.

For games that others have already developed 'ripkits' for it may be done safely, but those kits are usualy focused on making games fit on CD, and for most games this means reducing the quality of various game features or even eliminating some (FMVs etc). And in any case all such kits are specific to one game or just a small group of related games. There is no universal ripkit, and there never can be any, as that just isn't possible.

My NAS is filling up quick and i'm sure at least 50GB is in dummy files.

I'm sure there is a lot of dummy file space used in my own OPL fileshare too, currently using 640 GB (of a 750 GB partition), but I have no intention of patching ISOs to change that.

Doing it safely would require intensive research on every single ISO to be patched, and the savings possible just aren't worth that kind of work investment for me.

Best regards: dlanor

[TnA]

The HD, where my share is stored at is formatted in NTFS, so I set the folder-permissions to simply compress all files, which I store in it. ;-)

My server is only a Pentium III (3) 733/766MHz with 100MBit/s-LAN and it serves enough data, to simultaniusly play from 2 PS2s, so that fits well for me, not sure if it will help you.

[dlanor]

Originally Posted by TnA

The HD, where my share is stored at is formatted in NTFS, so I set the folder-permissions to simply compress all files, which I store in it. ;-)

Hmm. I didn't think of that.
Normally I never use such compression, but for the PS2 stuff it really does make sense.

The PS2 networking speed is so relatively slow, compared to any modern PC, that there should be no problem serving the ISO and VMC data even in compressed form, unless the same computer is doing some other CPU-intensive tasks at the time.

And it is of course also possible to set compression to be used only for the subfolders "ART", "CD" and "DVD", leaving "VMC" unaffected. Dynamic compression of files opened for read_write is something I really don't like, so I'd rather avoid that, and use it only for the read_only ISO files. And since all except one game of my setup now use normal ISOs, I should gain tons of space this way.

The partition I use for my fileshare is rapidly filling up (and I still have uninstalled originals), with now only 27GB remaining of the 750GB total in that partition, 643GB of which is already used by the PS2SMB fileshare. So this kind of compression could really help.

I just wonder how long it will take to do the initial compression of almost 640GB...?

Best regards: dlanor

[TnA]

Originally Posted by dlanor

Hmm. I didn't think of that.
Normally I never use such compression, but for the PS2 stuff it really does make sense.

Indeed. I only have a few games installed, but came down from ~40GB to 17-18GB.

The PS2 networking speed is so relatively slow, compared to any modern PC, that there should be no problem serving the ISO and VMC data even in compressed form, unless the same computer is doing some other CPU-intensive tasks at the time.

I could only test it with 2 PS2s as of yet (were my testgame for videos and multiple accesses to one ISO, was FFX), but the loading-time didn't decrease. But it might do so, in some compressed parts.
However, I didn't noticed it as of yet on my equipment.

And it is of course also possible to set compression to be used only for the subfolders "ART", "CD" and "DVD", leaving "VMC" unaffected. Dynamic compression of files opened for read_write is something I really don't like, so I'd rather avoid that, and use it only for the read_only ISO files.

I'll handle it the same way, once I start using VMC.
Currently I do not use that feature, because both the SMB-Module and the VMC-Module are kind of experimental (especially their "write-features") and I avoid using it in itself, once it is considered "stable".

And since all except one game of my setup now use normal ISOs, I should gain tons of space this way.

I think so, too.

The partition I use for my fileshare is rapidly filling up (and I still have uninstalled originals), with now only 27GB remaining of the 750GB total in that partition, 643GB of which is already used by the PS2SMB fileshare. So this kind of compression could really help.

I'm really interested, how much the needed storage-space went down, if you might do it.

I just wonder how long it will take to do the initial compression of almost 640GB...?

Best regards: dlanor

Well, you don't need to do all at once, but only "a few" to test your equipment with it.

best regards TnA

[nemesis01]

Originally Posted by dlanor

Hmm. I didn't think of that.

I just wonder how long it will take to do the initial compression of almost 640GB...?

Since I only have 4 games installed totalling 15.8GB gave this a try, to see how much space it will save & to see how long it takes.

It took roughly 15 minutes to compress down my 4 ISO files, I saved 3.9GB which is not too bad considering. I guess it also depends on the game how well it would normally compress with 7Zip or WinRar, if the disk is filled with dummy files you would probably save a lot more space, Soul Calibur 2 would probably compress down very well using this method.

[dlanor]

Originally Posted by TnA

Indeed. I only have a few games installed, but came down from ~40GB to 17-18GB.

That compression rate is quite impressive, but with only a few games it is hard to know if that is representative for the average game. Some of the more advanced games may already use compressed data to make room for everything on a normal DVD5 disc, and such games will give very bad NTFS compression ratios (like placing a RAR file inside a ZIP)

I could only test it with 2 PS2s as of yet (were my testgame for videos and multiple accesses to one ISO, was FFX), but the loading-time didn't decrease. But it might do so, in some compressed parts.
However, I didn't noticed it as of yet on my equipment.

I think it would take a rather slow PC not to keep up with the transfer rate of PS2 networking.

In running a single instance of FFX I have seen transfer burst peaks of around 12.5 Mbps at the very most, and that only for a brief fraction of a second, in complete reinitialization of game data, such as when loading a gamesave. For area transitions I see similar even briefer burst of around 6 to 8 Mbps, and for normal movement without special encounters the transfers needed are virtually none, as local background and soundtrack are already in memory.

Naturally this can vary greatly between games, and those less efficiently coded may need constant disc access (thus network transfer) for all kinds of stuff. But even then we can expect the absolute peak transfers to go no higher than around 12.5 Mbps due to limitations of the PS2.

I'm really interested, how much the needed storage-space went down, if you might do it.

I plan to try it, though I haven't quite decided how best to do it yet.

Well, you don't need to do all at once, but only "a few" to test your equipment with it.

Yes, I think the best way to do it is to keep the old "CD" and "DVD" folders uncompressed but rename them to something else, like adding "_old" as name suffix. Then I create new "CD" and "DVD" folders with the compression attribute set, and start moving some ISOs from "CD_old" and "DVD_old" into the new folders, which should cause them to be compressed as part of the move.

This way I can do it as gradually as I wish, guided by early results.

Since compression must be expected to take at least as long time as copying similar amounts of data would, I must expect compression of 640 GB to take considerable time. For normal copying from one drive to another I get appx 44MB/s (around 1.5 min for a 4GB ISO or 23 seconds per GB). But this compression will use the same drive for both source and destination which will slow things down, in addition to the compression work. But even if I figure on getting the same rate undiminished, that still comes to almost 15,000 seconds, which is over 4 hours. And it could take considerably longer...

So I'll try it just with a few ISOs at a time to begin with, to see how that goes.

Some tests later...:
Apparently normal moving does not work.

The compression only happens if I 'Copy' rather than 'Move' the files. So I am automatically forced to do this very gradually at first, as only 27 GB remain free on the drive at this time, so that is the largest 'chunk' of ISOs I can use initially, until some more space has been released as I delete originals of the already compressed ISOs. I'm a bit nervous about this deletion of original rips too, as it would mean a HUGE amount of work if I had to redo the ripping, in case something goes wrong with the compressed files. Oh well, nothing risked, nothing gained, so here I go...

Best regards: dlanor

[dlsmd]

Originally Posted by dlanor

That compression rate is quite impressive, but with only a few games it is hard to know if that is representative for the average game. Some of the more advanced games may already use compressed data to make room for everything on a normal DVD5 disc, and such games will give very bad NTFS compression ratios (like placing a RAR file inside a ZIP)

I think it would take a rather slow PC not to keep up with the transfer rate of PS2 networking.

In running a single instance of FFX I have seen transfer burst peaks of around 12.5 Mbps at the very most, and that only for a brief fraction of a second, in complete reinitialization of game data, such as when loading a gamesave. For area transitions I see similar even briefer burst of around 6 to 8 Mbps, and for normal movement without special encounters the transfers needed are virtually none, as local background and soundtrack are already in memory.

Naturally this can vary greatly between games, and those less efficiently coded may need constant disc access (thus network transfer) for all kinds of stuff. But even then we can expect the absolute peak transfers to go no higher than around 12.5 Mbps due to limitations of the PS2.

I plan to try it, though I haven't quite decided how best to do it yet.

Yes, I think the best way to do it is to keep the old "CD" and "DVD" folders uncompressed but rename them to something else, like adding "_old" as name suffix. Then I create new "CD" and "DVD" folders with the compression attribute set, and start moving some ISOs from "CD_old" and "DVD_old" into the new folders, which should cause them to be compressed as part of the move.

This way I can do it as gradually as I wish, guided by early results.

Since compression must be expected to take at least as long time as copying similar amounts of data would, I must expect compression of 640 GB to take considerable time. For normal copying from one drive to another I get appx 44MB/s (around 1.5 min for a 4GB ISO or 23 seconds per GB). But this compression will use the same drive for both source and destination which will slow things down, in addition to the compression work. But even if I figure on getting the same rate undiminished, that still comes to almost 15,000 seconds, which is over 4 hours. And it could take considerably longer...

So I'll try it just with a few ISOs at a time to begin with, to see how that goes.

Some tests later...:
Apparently normal moving does not work.

The compression only happens if I 'Copy' rather than 'Move' the files. So I am automatically forced to do this very gradually at first, as only 27 GB remain free on the drive at this time, so that is the largest 'chunk' of ISOs I can use initially, until some more space has been released as I delete originals of the already compressed ISOs. I'm a bit nervous about this deletion of original rips too, as it would mean a HUGE amount of work if I had to redo the ripping, in case something goes wrong with the compressed files. Oh well, nothing risked, nothing gained, so here I go...

Best regards: dlanor

let us know how it turns out

[dlanor]

One thing I can say already is that the variation in compression efficiency is HUGE.

For some few games over 50% of their total space can be regained by compression, meaning that they end up using less than half their original storage space, like TnA described.


But for some games more efficiently made, like FFX, it is an entirely different story...

My ISO of FFX(UK)==SCES_504.90 now shows the following file properties

Size: 4.28 GB (4,598,169,600 bytes)
Size on disk: 4.25 GB (4,572,221,440 bytes)

So the space regained was 25,948,160 bytes, which is slightly less than 25MB, and less than one percent of the original total size. (appx 0.56%)

So if all game discs were made that efficiently, there would be no point in compressing at all, but that is not the case.
FFX is quite unusual in that regard (as it is in many other ways too).

Best regards: dlanor

[TnA]

Originally Posted by dlanor

That compression rate is quite impressive, but with only a few games it is hard to know if that is representative for the average game. Some of the more advanced games may already use compressed data to make room for everything on a normal DVD5 disc, and such games will give very bad NTFS compression ratios (like placing a RAR file inside a ZIP)

Agreed. One of my installed games is EyeToy-Play, which is compressed to a smaller size than CD-size, but this is due the game's "false/fake LBA".
The game-data itself is not compressed that much.

I think it would take a rather slow PC not to keep up with the transfer rate of PS2 networking.

Well, I could set-up a server on an old "AMD K6/2-350MHz" and try it out, or even on a very old "box", where "Bull" is on some gummi-label in the front.
Anyway,... This PC (I'm talking from my P3-Server now) only acts as a file-server, so I have no problems with it.
It doesn't do any other demanding tasks.

In running a single instance of FFX I have seen transfer burst peaks of around 12.5 Mbps at the very most, and that only for a brief fraction of a second, in complete reinitialization of game data, such as when loading a gamesave. For area transitions I see similar even briefer burst of around 6 to 8 Mbps, and for normal movement without special encounters the transfers needed are virtually none, as local background and soundtrack are already in memory.

I mainly tested streaming a video on both consoles from the same ISO, over multiple PS2s, to see if there are speed-issues.
Maybe there are, if the files where heavily fragmented in combination with compression.

Naturally this can vary greatly between games, and those less efficiently coded may need constant disc access (thus network transfer) for all kinds of stuff. But even then we can expect the absolute peak transfers to go no higher than around 12.5 Mbps due to limitations of the PS2.

I plan to try it, though I haven't quite decided how best to do it yet.

Yes, I think the best way to do it is to keep the old "CD" and "DVD" folders uncompressed but rename them to something else, like adding "_old" as name suffix. Then I create new "CD" and "DVD" folders with the compression attribute set, and start moving some ISOs from "CD_old" and "DVD_old" into the new folders, which should cause them to be compressed as part of the move.

Yes, moving doesn't work. It "holds" the old set file-permissions.
I think it compresses it, if you move it from another device to the compressed folder and while you make a new ISO from disc.

This way I can do it as gradually as I wish, guided by early results.

Since compression must be expected to take at least as long time as copying similar amounts of data would, I must expect compression of 640 GB to take considerable time. For normal copying from one drive to another I get appx 44MB/s (around 1.5 min for a 4GB ISO or 23 seconds per GB). But this compression will use the same drive for both source and destination which will slow things down, in addition to the compression work. But even if I figure on getting the same rate undiminished, that still comes to almost 15,000 seconds, which is over 4 hours. And it could take considerably longer...

So I'll try it just with a few ISOs at a time to begin with, to see how that goes.

Some tests later...:
Apparently normal moving does not work.

Yepp, so I would say moving between devices is no real "moving", but rather a copy&"delete old file"-function (reminds me a bit to uLE, but while moving a file from the same device just to another folder a PC handles it a bit more efficiently, just rewriting the indexes, or what... Just on a side-note it would be cool, if uLE could handle "moving" on the same device a bit more efficiently... I think, that would be the top-notch for file-handling on the PS2.).

The compression only happens if I 'Copy' rather than 'Move' the files. So I am automatically forced to do this very gradually at first, as only 27 GB remain free on the drive at this time, so that is the largest 'chunk' of ISOs I can use initially, until some more space has been released as I delete originals of the already compressed ISOs. I'm a bit nervous about this deletion of original rips too, as it would mean a HUGE amount of work if I had to redo the ripping, in case something goes wrong with the compressed files. Oh well, nothing risked, nothing gained, so here I go...

Best regards: dlanor

The "compressed-files" are pretty much "on-the-fly"-decompressed, so they behave exactly like the "originals" and Byte-wise (in RAM) they are the same then.
Only Byte-wise in HDD-Sectors it is in compressed state...

Best wishes, luck & regards TnA

[agalloch]

Amazing! Never thought of using the NTFS compression. I think i disabled it for all files on my pc because the (minute?) lag it may add while booting, gaming, etcetera. I'll need to compress my NAS tonight.

But, for my usb hd I'll need to rebuild the iso's. I was up all night yesterday trying to get cdgenps2 to import the filelist without success. Today I stumbled upon "Xpert" which makes rebuilding ISO's much faster. Saved 20gb so far with just 9 games, most being simple kids games.

[jimmikaelkael]

This thread has given me the idea to do the same under Linux, thank you guys I saved a lot some precious space on my laptop HDD.
So here's how I proceeded under Ubuntu Lucid Lynx 10.04 for those interested, I tried to keep things simple: I can still use my normal PS2SMB folder or mount in place of it the compressed one.

First of all you need to install "fusecompress":

$ sudo apt-get install fusecompress

On my setup I use a folder named "OPLSMB" in my home directory as the normal "PS2SMB" SMB share, so I created another folder named "OPLSMB-zlib" (fusecompress will use zlib compression algorithm by default and I suggest you to not change it):

$ cd
$ mkdir OPLSMB-zlib/

Now you need to mount "OPLSMB-zlib" directory, for this I created another directory in my home dir as to be able to have a mountpoint with my user permissions:

$ cd
$ mkdir foo/
$ fusecompress ~/OPLSMB-zlib ~/foo

Now you foo directory is transparently accessible with nautilus, and you can go it, create the required "CD", "DVD", "THM", "ART" and "VMC" folders, or do it with the command line if you prefer.
Once done, copy the ISO you want to compress to their respective "CD" or "DVD" directory (inside "foo" dir !).

You can check the size of ISO files "OPLSMB-zlib" folder to see their compressed size.

$ cd ~/foo
$ du -csh */*

or get their uncompressed size:

$ du -csh --apparent-size */*

Finally, you can unmount and delete the foo dir:

$ cd
$ fusermount -u ~/foo
$ rmdir foo/         (do this only if you had no error while unmounting, otherwise it will complaint for non-empty directory)

Time to play these zlib compressed games over SMB, on my side I use my usual PS2SMB directory as mountpoint (named "OPLSMB" on my setup).
Note this time I use the 'sudo' to get the compressed directory mounted with valid permissions that allow it to be accessed by Samba server.

$ sudo fusecompress -o allow_other,nonempty ~/OPLSMB-zlib ~/OPLSMB

Note from now what you see as OPLSMB directory on your system is in reality OPLSMB-zlib directory, the normal directory being hidden.

After you played, you want to unmount the compressed directory, but sometimes it refuses to unmount if Samba server isn't restarted first:

$ sudo restart smbd        (wait 3 seconds...)
$ sudo fusermount -u ~/OPLSMB

The compression speed can vary a lot depending on games, from 2MB/sec to 35MB/sec on my setup.
On a performance note the result are excellent, I didn't have noticed any difference between a compressed ISO and a normal one.

On 23 games I had 65% compression ratio (zlib), not so bad

jimmikaelkael@PC-DE-JIM:~$ cd ~/OPLSMB
jimmikaelkael@PC-DE-JIM:~/OPLSMB$ du -csh */S*
436M	CD/SCUS_971.59.ICO.iso
443M	CD/SLES_501.66.Star Wars StarFighter.iso
140M	CD/SLES_533.45.Shadow of Ganymede.iso
354M	CD/SLUS_200.01.Tekken Tag Tournament.iso
194M	CD/SLUS_217.66.Pipe Mania.iso
2,4G	DVD/SCES_525.82.Everybody's Golf.iso
896M	DVD/SCUS_971.24.Jak and Daxter.iso
1,1G	DVD/SLES_514.74.Blood Rayne.iso
2,3G	DVD/SLES_522.02.DOWNHILL Domination.iso
2,5G	DVD/SLES_522.37.Hack - Infection.iso
405M	DVD/SLES_525.32.Aces of War.iso
2,5G	DVD/SLES_537.77.Prince of Persia T2T.iso
1,6G	DVD/SLES_540.85.SFA Anthology.iso
2,2G	DVD/SLUS_201.99.Shaun Palmer's Pro Snowboarder.iso
395M	DVD/SLUS_202.73.Namco Museum 50Th Anniversary.iso
1,2G	DVD/SLUS_204.40.Kya Dark Lineage.iso
1,6G	DVD/SLUS_204.96.V-Rally 3.iso
546M	DVD/SLUS_206.66.Disgaea ISO.iso
1,3G	DVD/SLUS_207.03.Airforce Delta Strike.iso
3,0G	DVD/SLUS_210.05.Kingdom Hearts 2.iso
2,2G	DVD/SLUS_210.46.King Arthur (US).iso
260M	DVD/SLUS_214.32.NRA Gun Club.iso
976M	DVD/SLUS_217.93.DT carnage.iso
29G	total
jimmikaelkael@PC-DE-JIM:~/OPLSMB$ du -csh --apparent-size */S*
531M	CD/SCUS_971.59.ICO.iso
625M	CD/SLES_501.66.Star Wars StarFighter.iso
198M	CD/SLES_533.45.Shadow of Ganymede.iso
605M	CD/SLUS_200.01.Tekken Tag Tournament.iso
306M	CD/SLUS_217.66.Pipe Mania.iso
2,9G	DVD/SCES_525.82.Everybody's Golf.iso
1,4G	DVD/SCUS_971.24.Jak and Daxter.iso
1,6G	DVD/SLES_514.74.Blood Rayne.iso
2,6G	DVD/SLES_522.02.DOWNHILL Domination.iso
2,8G	DVD/SLES_522.37.Hack - Infection.iso
1,2G	DVD/SLES_525.32.Aces of War.iso
4,1G	DVD/SLES_537.77.Prince of Persia T2T.iso
4,4G	DVD/SLES_540.85.SFA Anthology.iso
2,5G	DVD/SLUS_201.99.Shaun Palmer's Pro Snowboarder.iso
1,7G	DVD/SLUS_202.73.Namco Museum 50Th Anniversary.iso
1,9G	DVD/SLUS_204.40.Kya Dark Lineage.iso
3,0G	DVD/SLUS_204.96.V-Rally 3.iso
1,2G	DVD/SLUS_206.66.Disgaea ISO.iso
1,6G	DVD/SLUS_207.03.Airforce Delta Strike.iso
3,7G	DVD/SLUS_210.05.Kingdom Hearts 2.iso
2,7G	DVD/SLUS_210.46.King Arthur (US).iso
1,2G	DVD/SLUS_214.32.NRA Gun Club.iso
1,2G	DVD/SLUS_217.93.DT carnage.iso
44G	total

As you have seen it on Windows too, the compression ratio also vary on every games.

Look at Ubuntu manpages for a little help on fusecompress:
http://manpages.ubuntu.com/manpages/...ompress.1.html