Improved rom0:FONTM and proper rom0:KROM font support for gsKit. Now with SHIFT-JIS (Japanese) character support

[SP193]

I found something slightly unpleasant about gsKit: It doesn't support SHIFT-JIS characters, and the font that the Japanese consoles use to display Japanese characters in their OSDs is rom0:FONTM.

So I embarked on a journey to rewrite the fontm font support of gsKit and to implement a totally new version of my KROM support for gsKit.

For those who don't know:
1. rom0:FONTM is the font file that your PS2 uses for everything (Usually the OSD).
2. rom0:KROM is a 1bpp, headerless FONTX2 font file. I don't know where it's used, but it's now accessible for those who want to use it.

It's nearly completed. However, I found that I cannot seem to use 16-bit palettes with gsKit for some reason.

I'm weak with mathematics for some reason (I easily get lost while working on equations), so I've been spending a lot of time being lost and confused with pointer arithmethic - something that I was and am still not thrilled with at all.

I think that support for the Russian script in the FONTM support module might be broken, as I rushed to fix the SHIFT-JIS to JIS X 0208 conversion table for it... since I was a little fustrated after spending the entire day on this thing.

(Confession: I wasted a couple of hours getting that table right via trial-and-error )

Right now, both font loaders use 32-bit palettes (From gsKit's fontm source files)... but I'm aiming to use 16-bit palettes to save space. Unfortunately, the fonts all turn out yellow and the alpha channels don't seem to work right.

Another thing that would be nice is if someone could answer the questions I have left in the files. I still cannot figure out why certain formulas I've been using to calculate pointer values work in one place but need some hackery to work elsewhere....

Providing a complete dump of all FONTM's characters in a bitmap or any other suitable format would be wonderful in helping towards adding support for the PS2-only characters.

If anyone could help me to complete this properly and to integrate this with gsKit, I'll appreciate it greately.

Of course, there might be (probably many) bugs, so please feel free to help me with and criticize my work.

If there is something that I didn't explain in the included files, please feel free to let me know, so that I can update the supporting documents accordingly.

Downloads/Links
gsKit font project: [120420]gsKit_FONT_project.7z

Thanks!

[kadorna2]

I found something slightly unpleasant about gsKit: It doesn't support SHIFT-JIS characters, and the font that the Japanese consoles use to display Japanese characters in their OSDs is rom0:FONTM.

So I embarked on a journey to rewrite the fontm font support of gsKit and to implement a totally new version of my KROM support for gsKit.

For those who don't know:
1. rom0:FONTM is the font file that your PS2 uses for everything (Usually the OSD).
2. rom0:KROM is a 1bpp, headerless FONTX2 font file. I don't know where it's used, but it's now accessible for those who want to use it.

It's nearly completed. However, I found that I cannot seem to use 16-bit palettes with gsKit for some reason.

I'm weak with mathematics for some reason (I easily get lost while working on equations), so I've been spending a lot of time being lost and confused with pointer arithmethic - something that I was and am still not thrilled with at all.

I think that support for the Russian script in the FONTM support module might be broken, as I rushed to fix the SHIFT-JIS to JIS X 0208 conversion table for it... since I was a little fustrated after spending the entire day on this thing.

(Confession: I wasted a couple of hours getting that table right via trial-and-error )

Right now, both font loaders use 32-bit palettes (From gsKit's fontm source files)... but I'm aiming to use 16-bit palettes to save space. Unfortunately, the fonts all turn out yellow and the alpha channels don't seem to work right.

Another thing that would be nice is if someone could answer the questions I have left in the files. I still cannot figure out why certain formulas I've been using to calculate pointer values work in one place but need some hackery to work elsewhere....

Providing a complete dump of all FONTM's characters in a bitmap or any other suitable format would be wonderful in helping towards adding support for the PS2-only characters.

If anyone could help me to complete this properly and to integrate this with gsKit, I'll appreciate it greately.

Of course, there might be (probably many) bugs, so please feel free to help me with and criticize my work.

If there is something that I didn't explain in the included files, please feel free to let me know, so that I can update the supporting documents accordingly.

Downloads/Links
gsKit font project: [120420]gsKit_FONT_project.7z

Thanks!

If i remember correctly, for alpha blending (aka transparencies) you need to use at least 24 bpp, at least on PC.

[SP193]

I've completed the FONTM support library, in terms of JIS X 0208 character support.

The only thing missing now is a rigorous bugtest and proper escape character support.

Summary of my findings from this project so far:

The Playstation 2 console has at least two JIS X 0208 fonts in it's ROM chip, and the two that are supported by the libraries within this release are rom0:FONTM and rom0:KROM.

Neither rom0:KROM nor rom0:FONTM contain all the characters in the JIS X 0208 character set, as both of them lack all level 2 Kanji (Tables 48 and above).

rom0:FONTM is the font file used by the OSD (rom0:OSDSYS) and contains some special characters that are unique to the Playstation 2 and that are not in the JIS X 0208 specification.

rom0:FONTM is supported by gsKit, but the implementation within gsKit was not very good. It was not only complicated in structure, but it does not support most of the characters in the JIS X 0208 character set as it mostly supported only ASCII characters.

I do not know what uses rom0:KROM, but it's a headerless FONTX2 font file that does not have any blank characters.
rom0:KROM is a 1-bit font, and hence is not very nice to look at. It's more difficult to use, considering that the Playstation 2's graphics hardware seems to only be capable of handling textures that are at least 4-bit in colour dept.

Since rom0:FONTM looks better and is better for usage with the Playstation 2, I did not put as much effort into completing the library for utilizing rom0:KROM. Maybe one day I'll rewrite the KROM library to have a more complete implementation like the FONTM library does, but for now, it's included for informative purposes.

Working on developing these libraries requires a rather solid understanding of how the JIS, SHIFT-JIS and ASCII encodings work, the design of the JIS X 0208 character set and a strong understanding that a character set is NOT an encoding.

A short write-up on the KROM library is included (Read KROM.txt) for documentation on this rarely-discussed font file and on its incomplete library within this package.

I hope that someone would help me to complete the FONTM library and add it to gsKit.

I've included the converted fonts (converted into bitmap files) in one of the downloads, so feel free to have a look at these unique font files that the Playstation 2's ROM chip has.

The fonts were converted from rom0:FONTM and rom0:KROM of my SCPH-10000. Although the files were generated by me... I don't know whether Sony would mind as they belonged to the copyrighted BIOS. Naturally, if someone has a problem with me uploading these files, I'll gladly remove them.

Notes for programmers
How can one get one of those Playstation 2 characters within FONTM to get displayed? Simple. You need to know it's character code (In SHIFT-JIS), and enter it into your C-formatted string using escape characters.

E.g. In main.c of the FONTM library, I did a demonstration using the 'pipes'. One of those characters had a code of 0x84AC, and I got it displayed by entering \x84\xAC in the string.

Downloads/Links:
FONTM and KROM font support libraries for gsKit: http://www.mediafire.com/?d4dp8tpbd1u1i1k
FONTM and KROM font bitmap converter utilities and the converted fonts: Playstation 2 ROM fonts.7z
FONTM and KROM font bitmap converter utilities (Source code): Playstation2 ROM font converters_src.7z
Project homepage: - Playstation 2 font project page -

As usual, please do not hotlink to the file as the link/filename will change with every release.

@kadorna2: Thanks, but the Sony GS chip documentation indicates that 16-bit colour supports alpha. I think that it's probably a bug in PCSX2.

[ffgriever]

@kadorna2: Thanks, but the Sony GS chip documentation indicates that 16-bit colour supports alpha. I think that it's probably a bug in PCSX2.

I've been using 16bit palettes with gsKit and had no problem. PCSX2 also was displaying the images just fine. Just note, that for 16bit palettes, if you scale anything that is bilinear filtered, the transparency can sometimes be a little bit odd (depending on parameters).

Anyway, just set gsGlobal->Test->AREF, gsGlobal->Test->ATST, gsGlobal->Test->AFAIL and then invoke gsKit_set_texa and gsKit_set_primalpha with the parameters you want. I've been using the values that were simulating PSX type palettes and transparency effects most acurately. You will most likely need something else

As for the clut memory. If the texture is 4 bit, just copy 32 bytes to the clut buffer. If the texture is 8 bit, and you have a clut consisting of 256 consecutive 16bit entries, then just copy them in following order:

for (i = 0; i < 8; i++)
{
    memcpy(&clbuffer[i*4*16 +0*16], &clsrc[i*4*16 +0*16], 16);
    memcpy(&clbuffer[i*4*16 +1*16], &clsrc[i*4*16 +2*16], 16);
    memcpy(&clbuffer[i*4*16 +2*16], &clsrc[i*4*16 +1*16], 16);
    memcpy(&clbuffer[i*4*16 +3*16], &clsrc[i*4*16 +3*16], 16);
}

... or simply exchange middle two 16bytes of every 64 bytes .

Then just send clubuffer to vram (GS_PSM_CT16: 16:16 for 8bit, 8:2 for 4bit).

PS. I'm writing it from memory, but it seems to be right .

Edit:
Don't all the glyphs have the exact same clut? If that's the case, then why bother whether the clut is 32 or 16 bit? In that case you're saving 512bytes at most... Not really a big deal (unless I'm wrong and each glyph uses own, different palette - I've never actually used fontm at all).

[SP193]

Welcome back again ffgriever!

I've been using 16bit palettes with gsKit and had no problem. PCSX2 also was displaying the images just fine. Just note, that for 16bit palettes, if you scale anything that is bilinear filtered, the transparency can sometimes be a little bit odd (depending on parameters).

Thanks for clarifying this. It's good to know that I've been working on a relatively sane platform (PCSX2).

So far, the only problem that I've come across is a occasional weird colour problem with PCSX2 - which gets solved if I change the render mode to software. I remember that the problem occurs if I used certain libraries like libgs and the libdebug scr_printf() function.

Anyway, just set gsGlobal->Test->AREF, gsGlobal->Test->ATST, gsGlobal->Test->AFAIL and then invoke gsKit_set_texa and gsKit_set_primalpha with the parameters you want. I've been using the values that were simulating PSX type palettes and transparency effects most acurately. You will most likely need something else

As for the clut memory. If the texture is 4 bit, just copy 32 bytes to the clut buffer. If the texture is 8 bit, and you have a clut consisting of 256 consecutive 16bit entries, then just copy them in following order:

for (i = 0; i < 8; i++)
{
    memcpy(&clbuffer[i*4*16 +0*16], &clsrc[i*4*16 +0*16], 16);
    memcpy(&clbuffer[i*4*16 +1*16], &clsrc[i*4*16 +2*16], 16);
    memcpy(&clbuffer[i*4*16 +2*16], &clsrc[i*4*16 +1*16], 16);
    memcpy(&clbuffer[i*4*16 +3*16], &clsrc[i*4*16 +3*16], 16);
}

... or simply exchange middle two 16bytes of every 64 bytes .

Then just send clubuffer to vram (GS_PSM_CT16: 16:16 for 8bit, 8:2 for 4bit).

PS. I'm writing it from memory, but it seems to be right .

Thanks for the example! It helps a lot since I realized that the gsKit toolkit functions are not fully bug-free.

I cannot work on my personal projects at the moment, so I'll have to test this sometime later.

The bitmap and PNG image toolkit functions have some bugs in them, mostly related to how they are not always converting the alpha level to the range supported by the PS2 (0 to 128). I think that the 4-bit bitmap tookit function seems to be causing graphics corruption in some tests too.... but I don't remember whether those tests were accurate or not.

Just in case I wasn't too clear on explaining the problem I had with 16-bit graphics earlier on (As in further up this thread), the 16-bit CLUT seems to be screwed up in some way that causes the glyphs to all look greenish in colour.

I hope that by following what you've suggested, I will have a good chance at fixing this odd issue.

Edit:
Don't all the glyphs have the exact same clut? If that's the case, then why bother whether the clut is 32 or 16 bit? In that case you're saving 512bytes at most... Not really a big deal (unless I'm wrong and each glyph uses own, different palette - I've never actually used fontm at all).

Yes, you're right. I am doing this to save VRAM.

Being the person I am, I have chose to write libraries that are as efficient as possible. So that the 'end-user' programmers can do whatever they want with the remaining resources without worrying about the libraries being resource hogs.

By the way, I think that it probably is a dumb question to ask since the Sony documentation doesn't seem to mention anything about this, but does the PS2 support 1bpp textures? rom0:KROM is a 1bpp font file (FONTX2 format)... but I really haven't come across any part of the PS2's ROM that ever loads and uses the characters in that file.

[ffgriever]

Just in case I wasn't too clear on explaining the problem I had with 16-bit graphics earlier on (As in further up this thread), the 16-bit CLUT seems to be screwed up in some way that causes the glyphs to all look greenish in colour.

Some RGBA-BGRA issues or 5:5:5:1 vs others?

Yes, you're right. I am doing this to save VRAM.

Is saving a total of 512 bytes of VRAM worth the hassle (well, it looks a little more professional if your libraries use as low resources as possible )? If all the glyphs use the same clut, then you can load the clut once and make all the glyph textures use it... Unless you're loading the clut with every texture again and again (which is completely unnecessary)?

Being the person I am, I have chose to write libraries that are as efficient as possible. So that the 'end-user' programmers can do whatever they want with the remaining resources without worrying about the libraries being resource hogs.

I understand, but I can't get how 512bytes of VRAM taken can make a library a resource hog .

By the way, I think that it probably is a dumb question to ask since the Sony documentation doesn't seem to mention anything about this, but does the PS2 support 1bpp textures? rom0:KROM is a 1bpp font file (FONTX2 format)... but I really haven't come across any part of the PS2's ROM that ever loads and uses the characters in that file.

No. At least I don't know about such feature.

[SP193]

Some RGBA-BGRA issues or 5:5:5:1 vs others?

That's most likely, although I can't seem to see the problem.

I have defined pure white as 0xFFFF. Shouldn't that give a pure white colour? D:
If I'm not mistaken, I think that it turned out reddish instead....

But setting that colour to 0x0000 displays black like it should.

EDIT: I remember that if the ATEST is enabled, the glyphs are not visible (For some reason...). But if a 32-bit palette is used, the glyphs are visible and they are coloured properly.

Is saving a total of 512 bytes of VRAM worth the hassle (well, it looks a little more professional if your libraries use as low resources as possible )? If all the glyphs use the same clut, then you can load the clut once and make all the glyph textures use it... Unless you're loading the clut with every texture again and again (which is completely unnecessary)?

It depends on how you look at it.

The CLUT is uploaded only once - when the library is initialized. Afterwards, only the pages (Groups of glyphs) in VRAM are replaced using a LRU algorithm.

I understand, but I can't get how 512bytes of VRAM taken can make a library a resource hog .

Well, it's 512 bytes of memory wasted.

But you're right. It's not a huge issue even if that cannot be fixed as 512 bytes of VRAM isn't a lot - even though the PS2 has only 4MB of VRAM.

No. At least I don't know about such feature.

Thanks.

[ffgriever]

Anyway, I'll try to put an example together and post both source and binary (just to make sure). My gsKit setup is heavily modified, though. Yet I don't recall changing anything regarding this functionality. AFAIR, it simply worked fine from the very beginning .

[SP193]

Anyway, I'll try to put an example together and post both source and binary (just to make sure). My gsKit setup is heavily modified, though. Yet I don't recall changing anything regarding this functionality. AFAIR, it simply worked fine from the very beginning .

Thanks for offering to write me some sample code.

Well, I won't have much time to work on so many projects at once, so please take your time with doing that.

I don't remember clearly, but I remember that I've gotten 256-colour images working before... but 16-colour images were always a pain to work with. Sometimes, the colours disappeared or were just wrong.

Maybe it's got something to do with the way I am using gsKit. Maybe I'll try again someday with my modified version of libgs.

[SP193]

Changelog

1. Fixed graphics corruption that was occurring when characters were being drawn after a page in VRAM has been replaced.
   The VRAM buffer type has been changed to GS_CLUT_NONE from GS_CLUT_TEXTURE, since anything other than GS_CLUT_TEXTURE will cause gsKit to initiate a write to the TEXFLUSH register.
2. Improved cleanup code, in the even that the KROM or FONTM font file cannot be loaded.
3. Made all writes to the pages stored in RAM go through the uncached segment, so that DMA transfers will work right without needing a D-cache writeback.
4. The alignment of all page buffers in RAM has been adjusted to 16 bytes, down from 64. 64-byte alignment is not required if the EE D-Cache is not written back.

In other words, the two libraries should be usable on a real Playstation 2 console.

I'm hoping that people can help me to test this, and I wish to see KROM replace the console font used by the EE debugging library.

Downloads/Links
Font project: [121230] Font project.7z