Fallout/Oblivion DDX files decoded! Custom textures in sight?

[mgreed]

EDIT: I should say FIRST that I can't create proper DDX files, only open them up and tear out the good parts! Please, if you have any relevent skills/knowledge, post or PM me so that we can get this figured out.

I've been a lurker for a while, but I think I finally have something to contribute...

Since folks first started modding Oblivion on XBOX, its been difficult or impossible to get textures working in Bethesda's Gamebryo engine games. This goes for Fallout 3 and New Vegas, too. XBOX and PS3 both use the same mysterious format. You could spend your whole day Googling DDX and come up with nothing but lots of folks with the same problems/questions.

Well, its been nagging at me for several months now, so yesterday I finally decided to take a real look at these files. I'm not much of a programmer, and I've never worked with DDS (or textures of any type, really). I'm just stubborn not too bad when it comes research/experimentation.

So... bored yet? Ok. I'll STFU and show you what I've got...

INTRODUCING MGREED'S EXCLUSIVE DDX EXPOSÉ!

Disclaimer: Though I go on at length about Oblivion and Fallout 3, I've actually only checked this with files from New Vegas...

Right!

So, as far as I can tell, all DDX files are two DDS files sharing the same header that are compressed separately (sans header) using something similar to ZLIB and then concatenated into one file.

Great, huh?

There are basically two copies of the same DDS in each file. The first is simply half the size of the second (they always seem to be in that order). So molerat.ddx has two dds files compressed inside it. The first is 512x512 and the second is 1024x1024. Please remember that they are different sizes, but only have one header. That comes into play later and complicates things a little bit. Update: I'm now not entirely certain about this part. See Post #4

If you open up any DDX file you will see a 136 bytes worth of header. NOW: I have to admit that I'm not sure what role the first 8bytes fill. So what? Neither do you! Stop judging me! We're moving on to the next 128 bytes before my embarrassment and shame prevent me from completing this post. In fact, if you have a DDX open in a hex editor right now, just go ahead and delete the first 8 bytes. We don't want em right now anyway.

The next 128 bytes are your standard DDS header, except that they appear to be big-endian (or in other words: bass ackwards). To get a standard little-endian header you could reverse the order of every 4 bytes manually, or you could just select that section (and ONLY that section, thank you) in a hex editor and byte flip the whole thing (treated as 32bit unsigned long). If you did it right, the first for bytes should be 44 44 53 20 (or DDS followed by a single space, eg. "DDS ")

If you happen to have a hex editor open and are following along, then you will now have a DDX file that is 8 bytes smaller and has a usable DDS header taking up the first 128 bytes. Great, huh? You like having that header? Want to explore it a little bit?

Too bad for you!

Cut those 128 bytes out and paste them in a new file, leaving your DDX without a header. Well... Almost with out a header. I think you could call the next 4 bytes a header, but its not the file's header. They represent the length (starting with the byte immediately following) of the first compressed DDS. Immedietly following that block (again, starting with the very next byte) is the second compressed DDS. As far as I can tell the second block runs right to the end of the file. The last byte of the block is the last byte of the file. Unless your BSA extractor is having trouble and is adding 20 or so extra bytes onto the end of the file. I've seen that a lot...

You might see something like this at the end of your DDX:

J..Zj.......%........a..9$textures\clutter\food\molemeat_n.d

If so, just remove "textures\clutter\food\molemeat_n.d" PLUS the one byte before it (which is "$" in this case)

So now what you have what look like two compressed DDS files. They are compressed individually, so you need to find out where the first block ends (by checking the 4 byte header) and splitting the blocks into two separate files (and go ahead and remove the last 4 bytes of header while you are at it). Run each of these files through using the zlib library, but with a tool that will ignore errors! Whatever method is used to compress these doesn't actually create a fully valid zlib stream. In fact, I couldn't decompress the files in python at first. I was using the top-level decompress() included in zlib. I had to use the decompressobj().decompress() because it (apparently) ignores the lack of end_stream in the file.

Once you have the decompressed DDS files, you can add the 128 byte header that you removed earlier back to the top of the file. The larger DDS should open fine, but the smaller one will wither fail to load or look a tiny bit crazy. Its easily fixable though...

--------TO BE CONTINUED--------

Anyway, I'm getting tired of typing, so for now I'll just post a my shitty python script to turn a DDX into two DDS files. (this script assumes windows! if you are on something else, you may have to change it to account for that whole "/" instead of "\" in filepaths thing.)

The script takes 1 argument: the file you want changed! for example:

python script.py molerat.ddx

Yay!

Again: I'm not a programmer, so feel free to clean up that script or offer an alternative:

import zlib, struct, sys, binascii

try:
    import psyco
    psyco.full()
except ImportError:
    pass

def main(fn):

    ddx = file(fn[0], 'rb')
    trash = ddx.read(8)
    sheader = ddx.read(128)
    header = ''

    for x in range(1,33):
        theader = sheader[(x*4-4):(x*4)]
        header = header + theader[::-1]

    length = int(binascii.hexlify(ddx.read(4)), 16)

    dds1c = ddx.read(length)
    dds2c = ddx.read()
    dds1d = zlib.decompressobj().decompress(dds1c)
    dds2d = zlib.decompressobj().decompress(dds2c)

    header2 = header
    hheader = binascii.hexlify(header)
    
    a1 = hheader[24:26]
    a2 = hheader[26:28]
    b1 = hheader[32:34]
    b2 = hheader[34:36]
    
    a = hex(int(a2 + a1, 16)/2)[2:].zfill(4)
    a = a[2:] + a[:2]
    b = hex(int(b2 + b1, 16)/2)[2:].zfill(4)
    b = b[2:] + b[:2]

    header1 = binascii.unhexlify(hheader[:24] + a + hheader[28:32] + b + hheader[36:])
    dds1 = header1 + dds1d
    dds2 = header + dds2d

    name = fn[0].split('\\')[-1:][0][:-1] + 's'
    file(name.split('.')[0] + 'SM.' + name.split('.')[1], 'wb').write(dds1)
    file(name.split('.')[0] + 'LG.' + name.split('.')[1], 'wb').write(dds2)



if __name__=='__main__':
    sys.exit(main(sys.argv[1:]))

oh... and i intend to add a credits section and links to my sources

[dronesplitter]

This is great to hear. I tried looking at these files back in December and never got anywhere (other than realizing that the header was reverse endian). Two DDS files in one? Hmmm.

[Juggahax0r]

This is indeed good news for those who would like to make ground up texture replacements aimed specifically at the Xbox 360 and if the PS3 also uses DDX then the PS3 , however a discovery was made by a buddy of mine on how to use archiveinvalidation correctly to replace them with what already exists out there for texture replacement.

I won't make a tut right now I have way too much other stuff to worry about. Essentially make an archiveinvalidation on the PC version of what you want and change the tag at the end of the file names to .ddx instead of .dds it will load the external textures. Another Idea I have been kicking around but haven't tried yet is too decompress the Xbox 360 bsa files into there respective folders , turn susearchives off in the ini file and run off the data folder , then remove the .ddx version of whatever file it is you are trying to use.

You will notice especially in Fallout that if you have a texture replacement in that is not working it will switch between the 2. Archive invalidation is actually a bug in the engine , the reason you need that text is to force it to invalidate the older file inside the archive , basically it's a hack , because it should do that automatically if you have the option set for it in the ini.

The information you provided will help with getting some of the mods I have added in but won't display properly working. I have issues with some textures showing black or purple , and then some not at all , those being built for the PC , the SDK documentation has info on the proper format DDS to use for the 360 and a nice little plugin for an ATI tool you can use to convert there format.

[mgreed]

This is great to hear. I tried looking at these files back in December and never got anywhere (other than realizing that the header was reverse endian). Two DDS files in one? Hmmm.

Yeah, though the more I look at it, the more I'm certain that I'm wrong, already. It must have something to do with the way the consoles handle mipmapping/LOD. If you add the filesizes of the two DDS files together, it will match the filesize of the PC's single DDS file. We're gonna need someone who knows a little more about DDS files, I think...

I won't make a tut right now I have way too much other stuff to worry about. Essentially make an archiveinvalidation on the PC version of what you want and change the tag at the end of the file names to .ddx instead of .dds it will load the external textures.

I've seen this (I think you also have to edit the ESP to point to .ddx instead of .dds?), but there are a lot of people (myself included) that haven't had any luck doing it that way. I'm HOPING that if we can get get to a point where we can make DDX files, then users won't have such sporadic and varied results.

I have issues with some textures showing black or purple , and then some not at all , those being built for the PC , the SDK documentation has info on the proper format DDS to use for the 360 and a nice little plugin for an ATI tool you can use to convert there format.

I just figured out where i recognized your name from. Mod Fallout 3 to ... nSins Forums - Page 3. That post was/is going into my section on further leads/research. That plugin you mentioned would be perfect, if we could get our hands on it.

I also just realized that most of the places that I've posted links to this post have your name somewhere in the thread/site. I promise I'm not stalking

[Wizardlörd♦]

bsa files can be loaded in oblivion as long they contain only meshes and are signed with the right key

[Tutorial] Custom Mod Meshes In Oblivion - PS3Hax Network - Playstation 3 Hacks and Mods

[dronesplitter]

Just checking back and, since I still haven't had a chance to try out your Python script, I was wondering, does it do the conversion correctly? That is, does it create two DDS files that are viewable in a program like Paint.NET and show correctly?

[mgreed]

maybe i'm just an idiot, but I'm trying to update my post and i'm not finding the 'Edit' button... Did the layout change that much, and I just didn't notice till now?

edit: now im more confused, as this post has an edit button, but my first post doesn't...

moar edit: I guess I can't edit posts that are more then $NUMBER$ minutes old now?

[Sean Stieber]

So now that you have come up with this method... is there any way of re-encrypting said 2 dds files (1 half resolution of original on both x and y size) with reverse endian and zlib to make authentic .ddx files so that we can make our own custom textures? I'm extremely interested in doing this for my skyrim mods on ps3. Currently able to make mods just fine, but new textures do not work properly as .dds format. Also is there any information known about the properties of the scaled down douplicate dds? can we just take a current dds and import it into adobe photoshop, then scale image to half it's size in x and y, save as another .dds for combing with original .dds and then use said compressed method to create a valid .ddx?