HDD problem caused by HDDFormat tool !?!

[dlanor]

The night before last I had a serious problem with HDDFormat, which may serve as a warning to others that this program must be handled with care, and even some skepticism... I, for one, will never, ever, entrust it to remove any one partition again, and my reason follows below.

The full story:
---------------
I wanted to test HDD transfers with Altimit and PS2Menu, and found that both of them had an identical limitation of being able to access only the 10 first partitions they find on the HDD. I'm not sure exactly how that 'finding' is done, but the limit is definite, and on my HDD it meant that only HDL partitions, including the HDL settings partition, were found, making the tests I wanted to make impossible. Somehow I had to get rid of one of those partitions, and make a new normal partition there instead.

Now, as I've learned from discussions about HDL game installers, the HDL settings partitions is not strictly necessary, as HDL will rebuild it if missing when HDL is launched. Thus I decided to erase that partition with HDDFormat, and then make a new one for future tests.

Btw: I do realize that I could simply have renamed the HDL settings partition, to achieve a similar result. But I wanted to be sure there was nothing HDL specific about that partition, to affect any tests.

Anyway, I never got around to making that replacement partition, because as soon as I had erased the HDL settings partition, HDDFormat started to show the disk as being empty and stating that most of its disk space was 'Free'. And yet, it was also unable to create any new partitions whatever... In fact the disk was not quite empty, although all standard partitions were gone. What remained could only be displayed through ExecFTPs browsing and consisted of the "__mbr" partition and a few "__empty" partitions. The latter apparently remnants from earlier fragmentation.

At this point it was clear that no tool I have for the PS2 could fix that HDD, so I used my PC to make a quick-format NTFS partition, before putting the HDD back into the PS2, so that HDL would reformat it on its next launch. This worked fine.

Next I used HDDFormat to create the 'normal' partitions I'd had before (avoiding my original mistake of installing games before the 'normal' partitions), and proceeded to transfer stuff back into them over the LAN. PS2Link+Altimit were very useful in this, so I can recommend that combination. Altimit has the only PS2 file browser I know of that allows simple switching between source and destination paths without risk of 'losing' the setting for either of them. However, The host: support it has is quite primitive, as it allows for only one folder level. Anything inside a subfolder is either lost (subfolders) or created with zero size (files). This limitation does NOT seem to be inherent to the 'host:' server on the PC (here 'ps2client') though, so other software could implement their use of it better.

Once I had my 'normal' partitions restored I decided (for the first time, ever) to use the PC to install the game partitions, with the aid of the latest version (v1.6.4) of WinHiip. So I used this to install some 24 games in a row (Yawn...), reconnected HDD to PS2, and was back to business as usual again.

Summary:
--------
1: HDDFormat can, without any apparent reason, erase a full disk when asked to erase a partition. The full disk erasure may then be done with errors, that prevent further use of the HDD, without new low-level partitioning, impossible (at present) to perform on the PS2.

2: Point 1 above is NOT disproven by tests to show that single partitions can be erased safely, as that has worked on my PS2 as well, with the same copy of HDDFormat, on previous occasions. And, like I said above, there was NO apparent reason for it to work differently this time, though it certainly DID.

3: One possible contributor to the risk is the fact that I used the patched version of HDDFormat. I don't really think this had anything to do with it, but the possibility can't be excluded.

4: Another possible risk contributor is that I had previously erased some of the partitions that are created by default when formatting the disk. Those partitions are not used by ANY software and have been described elsewhere as being not needed. (I believe they are created merely to match content with what a standard Sony HDD uses.) Possibly HDDFormat tests for their presence, but that's just a guess...

Conclusion:
-----------
From now on I will NOT allow HDDFormat to erase any partitions on my HDD, but will use HDL_Dump and its clients instead, or even future versions of WinHiip, as it appears to be aiming for full compatibility with standard partitions as well as game partitions. I heartily recommend that direction of development, combined with the inclusion of network support as well, though I suspect that a new version of HDL_Dump (or an entirely new server) may be needed, to make all operations work as well by remote as by local disk access.

Best regards: dlanor

[02cali]

I ended up with those __empty partitions on my hdd also. I can only see them with ftp. there are 5 of them and they appear to be takeing up alittle over 1.2g, so is there no way i can get rid of them without loosing the rest of the data i have saved on my hard drive? and how do i avoid getting more of these __empty partitions? just dont deleat any partitions with hddformat? Ok, you said to use hdl_dump to deleat? so how do i go about deleateing partitions with hdl_dump, im using the hdl_dump fruntend and have never seen anything in there about deleateing a partition

[dlanor]

02cali:
The __empty partitions are not in themselves errors, but are perfectly normal results of disk fragmentation. Suppose you originally had installed 3 games of 2GB each, and then you deleted the game in the middle. That would then become a fragmented part of free space, and would show up as __empty in the partition list.

The only reason I complained about them on my disk was because HDDFormat claimed that the disk was empty, in which case free fragments should have been merged in with the normal, unfragmented, free space. Since that never happened, I have to assume that this method of handling the fragments is specific to HDLoader, and not supported by other software (or at least not supported properly by HDDFormat).

As I understand it, the '__empty' fragments will be reused under various circumstances, depending on what installation methods you use.

HDLoader itself is rather reluctant to use fragments, and will probably only do so if an entire game can fit into a single fragment. I believe HDL_Dump uses the same method.

WinHiip, on the other hand, is quite eager to use fragments, and will happily let a single game use up any number of separate fragments needed.

When you get a message from HDLoader, or HDL_Dump, saying that there is no room for a new game, even though the total free area is much larger than the game, this means that your disk is badly fragmented. You can then still install games with WinHiip, but not with the other methods, as they refuse the small fragments available.

Summing up:
The "__empty" partitions are in fact fragments of free space for future use, so there is no need to get rid of them to gain room for new stuff.

What we do need is a way to defrag the disk so that all its free space is merged into one contiguous block, which all install methods could use equally well. But as yet we have no such program, so when an HDL disk has many small fragments, the only way to use those fragments (as far as I know), is by installing games with WinHiip.

As for deleting partitions with the frontends, I'm not familiar with all of them, but I think they should all allow you to remove the same stuff they allow you to install. That is certainly the case with the two programs I have used for installing HDL games. Those programs are WinHiip (with HDD mounted in PC), and the HDL_Dump GUI by LordBoGaMi (with HDD mounted in PS2).

Best regards: dlanor

[Zelius]

Not really related, but important anyway: dlanor, you said the HDD would be rendered useless without any low-level partitioning, which is impossible to perform on PS2. My question is, why is this impossible? Wouldn't it be possible to simply create a tool similar to HDD Format to do this, or mod the existing program?

The reason I ask this, is because my HDD doesn't seem to work in my PS2, even though it's compatible, and low-level partitioning\formatting would seem like a sollution.

[dlanor]

Not really related, but important anyway: dlanor, you said the HDD would be rendered useless without any low-level partitioning, which is impossible to perform on PS2. My question is, why is this impossible?

Because currently available tools won't do it.

Wouldn't it be possible to simply create a tool similar to HDD Format to do this, or mod the existing program?

Yes, of course that would be possible, although I seriously doubt that any major modifications to the existing HDDFormat program would be practical. It is NOT open-source...

The reason I ask this, is because my HDD doesn't seem to work in my PS2, even though it's compatible, and low-level partitioning\formatting would seem like a sollution.

Quite likely that would solve your problem, but you don't have to do it on the PS2 (which current tools don't allow anyhow). You can use a PC to do it, putting a non-PS2 filesystem on the drive (eg: NTFS). Then put the drive back into the PS2, whereafter HDLoader should automatically recognize that the drive needs reformatting to use the PS2 filesystem. It worked for me.

Best regards: dlanor

[alucard_x]

long story short, we need a ps2-side defrag utility for non-sony drives.

[dlanor]

alucard_x:
I certainly agree on the need for a good HDD defrag tool, but the main point of my original post was that we really can't rely on HDDFormat to work right when normal partitions coexist with HDL partitions. Not after the faulty behaviour it gave me! Others may feel differently about this, but I will never trust it again.

So a new general purpose HDD support tool, for formatting, partition control, and other stuff is needed (where 'other stuff' should hopefully include defrag).

Best regards: dlanor

[alucard_x]

to note, was it DeeMS HDDFormat tool?

[zabolyx]

we need a full frontal nudity HDD tool that strips it bare then rebuilds it, and allows new partitios to be made, FTPing, mounting and editing HDL partitions, and defrags all in one simpe to use and user friendly (so us 'tards don't screw up our drives) interface.

[dlanor]

alucard_x:
Yes, it was the DeeMS HDDFormat tool I used. The one with the 48-bit LBA patch.

zabolyx:
Yes, I agree that we need all of that, but it's a mighty tall order...
Expecting any single homebrew programmer to fill that order might be over-optimistic, at least in the short run. I would be happy to settle for getting this functionality piecemeal, as specialized utilities, and/or patches or additions for existing tools.

Best regards: dlanor