Generic CNF file parser

[dlanor]

Hi Everyone:
I'm back again after a long absence, during most of which I have done almost no PS2 coding. But I have been very active the last couple of days, trying to catch up on events here, and also starting to solve a very old problem of LaunchELF.

That problem is the difficulty of adding new features to LaunchELF, without causing problems with the inflexible CNF file format it uses. Many of the features discussed in recent months require a more flexible format, allowing arbitrary item order and optional inclusion of various items.

For this reason I have now designed a 'Generic CNF file parser', that can be used by LaunchELF, or any other application, to read configuration variables from a file using arbitrary/optional order and inclusion of any items, falling back on defaults for absent items (in ways up to to each application, as that is not a task for the parser).

Such a method obviously requires some way of identifying each item, and I have done that by implementing them as pseudo variable definitions. Thus each line in such a CNF file will either be a comment line (starting with a special character), or a variable definition of the generic form:

SomeName = Some value string

Whitespace is allowed and ignored in all parts of such a line, except in the value string itself, starting with the first non-whitespace character after the '=' character. Any whitespace characters used in the value string will be considered part of that string, which ends with the end of the line.

The C file in the appended ZIP contains three C functions:

'get_CNF_string' == The main parsing function. It contains nothing that is specific to any application that uses it, being a fully generic solution. This can be included exactly as-is in any application needing CNF parsing.

'Load_CNF' == The application specific function that calls 'get_CNF_string' to retrieve each variable's name and value string pair. It needs to be edited for each application, as it contains the basic response to each CNF variable found, though in most cases this simply consists of storing the value data in an internal program variable (often after an 'atoi' call for numbers). The version included shows some obvious examples of how to do this for the version of InfoNES I worked on.

'Save_CNF' == The application specific function used to save current settings to a CNF file. This too needs to be edited for each application, as it must write all of the CNF variables that you need in the CNF file. But remember that you can have even more variables defined, to be entered optionally by other means (including manual text editing). The version included shows how to handle the six simple variables used by my test program, but the methods used are very simple to extend to any degree of complexity.

For further details of the operation and design of this parser I refer you to the source code itself, which is heavily commented. Its current form is not suitable for direct inclusion with the LaunchELF sources, as I found it too awkward to implement this stuff from scratch with that rather extensive CNF. Instead I started with a simple CNF for a modified version of InfoNES, and used this to verify and polish off the parsing functionality.
NB: No InfoNES version posted yet has this stuff implemented, so don't waste time looking for it.

Note that I'm not going to implement this directly into LaunchELF, as I believe that we need more than one person's opinions on a change of this magnitude. And this obviously means that suggestions and comments on this stuff are invited, from anyone feeling he or she has anything to contribute.

Edit: I made a silly editing error in the appended source file, which made one comment line describing the parser a bit confusing. So I have now replaced that file with a corrected one.

Edit2: I've changed some declarations in the main parser function so that all character tests will be unsigned. This should help allowing CNF strings using characters with ascii codes above 0x7F (non-english character sets).

Edit3: I've modified one line of the parser to cope with cases where variables are assigned values consisting only of whitespace (=> empty string).

Best regards: dlanor

[zabolyx]

So now with this we can finally add menu names, rename commands to particular names, and set iop resets to individual apps?

[Experiment1106]

hmmm i dont really understand what i can do with this.
is it a tool for coder or have you easy words for me to understand
that, my english is not so perfect.

[dlanor]

So now with this we can finally add menu names, rename commands to particular names, and set iop resets to individual apps?
Yes. Once this has been merged into the LaunchELF sources (and that addition has been debugged), it should be significantly easier to add the kind of features you mention.

Even so, I have to add that it doesn't solve ALL problems of features such as IOP reset, since there are many technical issues involved. That particular change will definitely require some more tailor-made changes to PS2SDK. (But that's OK, really. After all, we've done that before.)

Best regards: dlanor

[dlanor]

hmmm i dont really understand what i can do with this.
is it a tool for coder or have you easy words for me to understand
that, my english is not so perfect.
Yes. The file I appended is a C language source code, which can be merged into an existing project (such as LaunchELF) to improve its ability to handle configuration files. So it can only be used by a 'coder', in modifying a program that way.

But all users of such a program can benefit from such a change, once it has been made, as the programmers may then find it easier to add and improve features of that program. More and better features should mean happier users all around... :)

Best regards: dlanor

[E P]

Wow dlanor, I see you have given this quite a lot of thought. The only thing I'm still wondering about is backward compatibility. Do we keep it or just drop it entirely? I wanted to keep backward compatibility early on but I think uLaunchELF can probably now stand on its own. Sure some users use both which would be a problem.

[dlanor]

Wow dlanor, I see you have given this quite a lot of thought.
Yes, we've talked about this need for a long while now, and I've been mulling it over in my mind since long before I started coding it.

The only thing I'm still wondering about is backward compatibility. Do we keep it or just drop it entirely?
I'd like to say BOTH, though that obviously won't add up. ;)
But perhaps we can find a compromise that comes close to it.

We do need to keep some kind of compatibility so that people can keep using their favourite configs without starting from scratch. But on the other hand we need to avoid bloat, and it would be ridiculous to keep the old and obsolete config code in the program for all future versions.

Perhaps the best solution would be to make a freestanding CNF converter:
When starting a new version of uLaunchELF with an old CNF set, the CNF parser will reject the entire CNF, because there are no variable assignments. But the program will still start up properly, using defaults for everything. One of our new defaults should then be to define one button ('X' or 'O'?) as leading to the FileBrowser, so that the user can browse to the converter program and use it to transform the old CNF into the new format, which will then be loaded at the next boot.

It is important that the FileBrowser can be reached through a button defined by default, as mentioned above, because if the user has to define it himself, then that will cause the old CNF to be overwritten at exit from the CONFIG screen.

I wanted to keep backward compatibility early on but I think uLaunchELF can probably now stand on its own.
I agree. And since we have always prioritized efficiency it would be inconsistent to keep old code in the program for ever, just to handle an obsolete CNF format.

Sure some users use both which would be a problem.
I don't see any problem with that. Sure, only one CNF set can be chosen as default by storing it in the SYS-CONF folder, but if the user wants to use an old ELF with the old CNF format he can still do so, by storing both that ELF and its CNF set in a separate MC folder. In much the same way I already use alternate CNF sets depending on my boot mode, using one colour scheme for Dev1 and another for Dev2, and a third for exploit, so the basic method is thoroughly tested.

Best regards: dlanor

[E P]

YWe do need to keep some kind of compatibility so that people can keep using their favourite configs without starting from scratch. But on the other hand we need to avoid bloat, and it would be ridiculous to keep the old and obsolete config code in the program for all future versions.
Yeah avoiding bloat was just what I was thinking about. I can't see adding in extra code for backward compatibility if the new config system is completely overhauled to be more organized.

Perhaps the best solution would be to make a freestanding CNF converter:
When starting a new version of uLaunchELF with an old CNF set, the CNF parser will reject the entire CNF, because there are no variable assignments. But the program will still start up properly, using defaults for everything. One of our new defaults should then be to define one button ('X' or 'O'?) as leading to the FileBrowser, so that the user can browse to the converter program and use it to transform the old CNF into the new format, which will then be loaded at the next boot.
Yeah that would make sense although it seems like a little much. Although it all hinges on what the final layout of the CNF file is in the end.

It is important that the FileBrowser can be reached through a button defined by default, as mentioned above, because if the user has to define it himself, then that will cause the old CNF to be overwritten at exit from the CONFIG screen.
I use R1 for my configs for filemanager but I understand what you mean.

I don't see any problem with that. Sure, only one CNF set can be chosen as default by storing it in the SYS-CONF folder, but if the user wants to use an old ELF with the old CNF format he can still do so, by storing both that ELF and its CNF set in a separate MC folder. In much the same way I already use alternate CNF sets depending on my boot mode, using one colour scheme for Dev1 and another for Dev2, and a third for exploit, so the basic method is thoroughly tested.

Best regards: dlanor
Yes, that's the way to do it. There are a few people that like to use the same config between different versions of launchELF although I'm not one of them. :)

[HypERSoniC]

this will solve the problem with the inflexible "line by line" settings file implementation in the current launchelf... nice work dlanor

[dlanor]

Yeah avoiding bloat was just what I was thinking about. I can't see adding in extra code for backward compatibility if the new config system is completely overhauled to be more organized.
Good, then we agree on this.

----- snip ----- re: separate CNF converter program
Yeah that would make sense although it seems like a little much.
It does, so whether we do this or not depends very much on user demand. So far, not ONE user has requested it, as far as I know...
If it remains that way, I see no reason for us to worry about it.

Although it all hinges on what the final layout of the CNF file is in the end.
Actually there won't be much 'layout' involved anymore, except that the very first line in the CNF should identify the CNF version with a string that could never exist there using the old CNF format. Other than that, all CNF items will be optional and may be freely mixed in any order at all...

Since each setting can be identified by its CNF variable name, 'layout' as such is not relevant anymore. But instead we have to define meaningful variable names for all the CNF entries.

----- snip ----- re: default button for filemanager (to start CNF converter)
I use R1 for my configs for filemanager but I understand what you mean.
In any case this is only a problem if/when we make that converter, so people can use it without erasing their old CNF. Another way to get around it is of course to implement new CNF file names for the new format. This eliminates the risk of overwriting completely, and also eliminates the need for that special string to identify the new CNF format. Then we have NO mandatory layout at all :). (Though we still have a mandatory syntax.)

----- snip ----- re: separating old/new versions simply in separate folders
Yes, that's the way to do it. There are a few people that like to use the same config between different versions of launchELF although I'm not one of them. :)
Good. We seem to agree on most of this stuff, so now we just have to decide on the details of implementation. Here's a brief summary of those decisions:

1: Choose CNF variable names for all existing settings
2: Decide to keep or change the CNF filenames (.INI, .CFG, ...?)
3: Decide to make or forget the CNF converter (subject to demand)

And then we naturally need to implement those decisions, but I think that this will be fairly simple. It will still be time-consuming, of course, but not in any way difficult.

Best regards: dlanor

[E P]

----- snip ----- re: separate CNF converter program
It does, so whether we do this or not depends very much on user demand. So far, not ONE user has requested it, as far as I know...
If it remains that way, I see no reason for us to worry about it.
Alright, yes I agree.

Actually there won't be much 'layout' involved anymore, except that the very first line in the CNF should identify the CNF version with a string that could never exist there using the old CNF format. Other than that, all CNF items will be optional and may be freely mixed in any order at all...

Since each setting can be identified by its CNF variable name, 'layout' as such is not relevant anymore. But instead we have to define meaningful variable names for all the CNF entries.
The only thing was whether or not the menus would be changed. I had looked into other peoples ideas to overhaul the LaunchELF list. However, I later pursued finishing all the FTP stuff instead after Slam-Tilt showed some interest in the menu based stuff.

I don't think it would be too much of a problem either way if each config item has a marker in the CNF file.

----- snip ----- re: default button for filemanager (to start CNF converter)
In any case this is only a problem if/when we make that converter, so people can use it without erasing their old CNF. Another way to get around it is of course to implement new CNF file names for the new format. This eliminates the risk of overwriting completely, and also eliminates the need for that special string to identify the new CNF format. Then we have NO mandatory layout at all :). (Though we still have a mandatory syntax.)
Yeah, I also thought about this before like going from LAUNCHELF.CNF to ULAUNCHELF.CNF or something to that effect.

----- snip ----- re: separating old/new versions simply in separate folders
Good. We seem to agree on most of this stuff, so now we just have to decide on the details of implementation. Here's a brief summary of those decisions:

1: Choose CNF variable names for all existing settings
2: Decide to keep or change the CNF filenames (.INI, .CFG, ...?)
3: Decide to make or forget the CNF converter (subject to demand)

And then we naturally need to implement those decisions, but I think that this will be fairly simple. It will still be time-consuming, of course, but not in any way difficult.

Best regards: dlanor
1. Hard for me to say. KeyLauncher uses [], |>, and O but I think that confuses some.
2. I'd stay with CNF as it appears to be the most standard extension name.
3. I'd say forget it for now. I mean it's not like people would have too many CNF's to convert.

[dlanor]

The only thing was whether or not the menus would be changed. I had looked into other peoples ideas to overhaul the LaunchELF list.
Ok, but such issues don't really affect our choices for the new CNF format. It's rather the other way around, as the new CNF format will open up lots of new possibilities for what we can do in the future.

I don't think it would be too much of a problem either way if each config item has a marker in the CNF file.
Each item will have a distinct unique name.

1: Choose CNF variable names for all existing settings
2: Decide to keep or change the CNF filenames (.INI, .CFG, ...?)
3: Decide to make or forget the CNF converter (subject to demand)
1. Hard for me to say. KeyLauncher uses [], |>, and O but I think that confuses some.
Agreed. I prefer clear descriptive names using only normal ascii text.

2. I'd stay with CNF as it appears to be the most standard extension name.
Agreed. For now I'll assume we're keeping the old filenames.

3. I'd say forget it for now. I mean it's not like people would have too many CNF's to convert.
Agreed, and the max number of settings is only 25, so it should be acceptable to do this over once.

I have written a fairly complete proposal for the new CNF format, including new defaults for some settings. The proposal has the form of a LAUNCHELF.CNF file in the new format, with lots of comment lines explaining everything. I'm attaching it to this post in a ZIP.

Please check it out and let me know what you think.

Best regards: dlanor

[E P]

Oh yeah, I looked over the LAUNCHELF.CNF 2.0 spec and I agree with pretty much everything. Those changes would allow for better compatibilty, flexability, and readability. I also like the idea of changing those defaults: IOP reset to ON and DISC CONTROL to OFF. :)

[dlanor]

Oh yeah, I looked over the LAUNCHELF.CNF 2.0 spec and I agree with pretty much everything. Those changes would allow for better compatibilty, flexability, and readability. I also like the idea of changing those defaults: IOP reset to ON and DISC CONTROL to OFF. :)
Good. Then I'm almost ready to start coding.

There's just one more thing I need to do first, and that is to update all parts of my PS2Dev setup to the same state as yours, by recompiling all the libs (esp. PS2SDK) using v3.2.2 of the IOP compiler. This is necessary to ensure identical results when you and I work on new releases.

Hopefully that update will work glitch-free, and then I'll start coding the new CNF handlers of LaunchELF.

Edit:
PS2SDK updated fine, but I ran into a small snag with libito v0.2.1

For some reason 'make' exits with an error. This happens _after_ producing 'libito.a', in processing the 'data' folder for the example programs. Hopefully it only means that the example setup is botched, though the lib itself may be fine. I'll assume as much for now anyway. My only worry is that I don't remember this error from earlier updates. (I probably just forgot.)

Best regards: dlanor

[dlanor]

I have now completed my update of libs and support modules using v3.2.2 of the IOP compiler. As you know I have tried this long ago, with miserable results. Nothing would compile right with optimizations, and without them the bloat was horrible. Fortunately results today are a lot better.

For example, here's a comparison of results for ps2ftpd:

ps2ftpd.irx in modules folder of v3.41s had a size of 30657 bytes
ps2ftpd.irx as compiled by v2.8.1 here has a size of 30093 bytes
ps2ftpd.irx as compiled by v3.2.2 here has a size of 30657 bytes

That's a bloat increase of less than 2%, which I find acceptable, but the most important result is that you and I again have consistent results. (I've checked this for other modules too, like cdvd.irx.) I suspect that our earlier differring results were due to differencies in how far we had adapted things to the new compiler, as well as to insufficient adaption in the older PS2SDK.

I am now going to start on the new CNF functions.

Edit:
I rejoiced to soon. I had forgotten to recompile usb_mass, and doing so I got a result that is miles away from what you had for v3.41s.

Your usb_mass.irx has a size of 54577 bytes
My usb_mass.irx is much smaller at 33925 bytes

I got the latest version from SVN and replaced three files in it with the ones from your latest uLaunchELF package, and then compiled it to that small size. I'm really confused now, as I never expected a size decrease with the new compiler... Could it possibly be due to changed optimization flags in the makefile ?

Please compile usb_mass again, to test if our results are in fact consistent or not. That fact is crucial for our ability to develop this stuff together.

Best regards: dlanor

[E P]

PS2SDK updated fine, but I ran into a small snag with libito v0.2.1

For some reason 'make' exits with an error. This happens _after_ producing 'libito.a', in processing the 'data' folder for the example programs. Hopefully it only means that the example setup is botched, though the lib itself may be fine. I'll assume as much for now anyway. My only worry is that I don't remember this error from earlier updates. (I probably just forgot.)
Yeah, I believe it has always done that. My libito.a is 70.3 KB (72,082 bytes).

I rejoiced to soon. I had forgotten to recompile usb_mass, and doing so I got a result that is miles away from what you had for v3.41s.

Your usb_mass.irx has a size of 54577 bytes
My usb_mass.irx is much smaller at 33925 bytes

I got the latest version from SVN and replaced three files in it with the ones from your latest uLaunchELF package, and then compiled it to that small size. I'm really confused now, as I never expected a size decrease with the new compiler... Could it possibly be due to changed optimization flags in the makefile ?

Please compile usb_mass again, to test if our results are in fact consistent or not. That fact is crucial for our ability to develop this stuff together.
I believe you forgot to change the makefile to allow write access for the mass driver. Should be like this:
#uncomment to enable write support
WRITE_SUPPORT=1

We really should have included that changed makefile a long time ago in the source package.

Edit:
Oh a word of warning, I just noticed that the new changes to usbd in the ps2sdk prevents usb_mass driver from compiling. It has to do with changes to usbd.h. Just stick the old version of that file in the ps2sdk/iop/include directory to get usb_mass to compile.

[dlanor]

----- snip ----- re: libito 'make' errors
Yeah, I believe it has always done that. My libito.a is 70.3 KB (72,082 bytes).
Great. That's exactly the size I got too, so score another point for consistency :)

I believe you forgot to change the makefile to allow write access for the mass driver. Should be like this:
#uncomment to enable write support
WRITE_SUPPORT=1
Yes, that was it. But I still don't get exactly the same size as you.
Yours was 54577 bytes, whereas I now get 54325 bytes.

We really should have included that changed makefile a long time ago in the source package.
Indeed, so I just did... :)

Btw: The uLaunchELF sources today resemble a kind of jigsaw puzzle. To some extent it has to be that way, as we use unrelated projects too, but some of the mess can be simplified.

The main simplification is just to start releasing the complete LaunchELF sources each time, instead of relying on the original v3.41 zip to provide some of the puzzle pieces. That will actually help two ways. First, when people receive these sources, they don't have to add in those old pieces. Then again, when people release modified sources, they don't have to rip out the old pieces, as we've done so far. I really think this simplification will improve our chances of getting others to join in the fun too.

Edit:
Oh a word of warning, I just noticed that the new changes to usbd in the ps2sdk prevents usb_mass driver from compiling. It has to do with changes to usbd.h. Just stick the old version of that file in the ps2sdk/iop/include directory to get usb_mass to compile.
Those must be very recent changes then, because I'm using the PS2SDK I got from SVN on the 16th of January, just over two weeks ago. I didn't udate those sources today, but only recompiled them with the new IOP compiler. This version works fine.

I think those ongoing changes in PS2SDK may also explain the size difference we get for usb_mass.

Do you think we need to synchronize our PS2SDK updates ?
Or should we just ignore such minor differencies ?

Btw: I've just finished the code for saving the new config file, and verified that it at least compiles correctly. Now I'll start on the code to load the new format CNF, and then I'll start testing it.

If all goes well a new release could be ready sometime tomorrow, but I'm not promising anything at this stage. Some debugging is usually needed...

Best regards: dlanor

[E P]

----- snip ----- re: libito 'make' errors
Great. That's exactly the size I got too, so score another point for consistency :)
:)

Yes, that was it. But I still don't get exactly the same size as you.
Yours was 54577 bytes, whereas I now get 54325 bytes.
Hum, that's odd because mine too is 53.0 KB (54,325 bytes). :chinscrat I even recompiled it and it matches the one in my modules folder.

Btw: The uLaunchELF sources today resemble a kind of jigsaw puzzle. To some extent it has to be that way, as we use unrelated projects too, but some of the mess can be simplified.
I considered changing this quite a while back. Maybe even getting rid of pre-compiled modules and only including the necessary source files like you did with ps2host. Unfortunately, we can't go that route for libcdvd since it relies on the old ps2lib which few people have.

The main simplification is just to start releasing the complete LaunchELF sources each time, instead of relying on the original v3.41 zip to provide some of the puzzle pieces. That will actually help two ways. First, when people receive these sources, they don't have to add in those old pieces. Then again, when people release modified sources, they don't have to rip out the old pieces, as we've done so far. I really think this simplification will improve our chances of getting others to join in the fun too.
I agree completely. Only other issue is that napalm's usbd.irx is not supposed to be distributed so I don't know if we want to start distributing it now.

Those must be very recent changes then, because I'm using the PS2SDK I got from SVN on the 16th of January, just over two weeks ago. I didn't udate those sources today, but only recompiled them with the new IOP compiler. This version works fine.
Yeah, the changes are less than a week or so old. I can't get too excited about the issue yet as I'm sure other changes are currently in the pipeline.

I think those ongoing changes in PS2SDK may also explain the size difference we get for usb_mass.
No, I checked and they're the same after changing that usbd.h file.

Do you think we need to synchronize our PS2SDK updates ?
Or should we just ignore such minor differencies ?
I don't think it would matter I recompiled the entire ps2sdk at the beginning of january. I update my ps2sdk files from svn regularly with a batch file. Not too much has changed since radad's big change to the partition size code in November.

Btw: I've just finished the code for saving the new config file, and verified that it at least compiles correctly. Now I'll start on the code to load the new format CNF, and then I'll start testing it.

If all goes well a new release could be ready sometime tomorrow, but I'm not promising anything at this stage. Some debugging is usually needed...

Best regards: dlanor
Alright sounds good. If you need anything, let me know.

BTW: I have put together a listing of all the flags for mcSetFileInfo so I'll reply to the main thread with all that info later on. So far I've found 3 flags although I'm sill not sure how to change them to a specific values:
1. sets the created date and time
2. sets the modified date and time
3. sets the attributes

[dlanor]

----- snip ----- re: differing sizes of usb_mass
Hum, that's odd because mine too is 53.0 KB (54,325 bytes). :chinscrat I even recompiled it and it matches the one in my modules folder.
I was referring to the usb_mass you enclosed with LaunchELF v3.41s. Apparently something has affected the size since then, but never mind. My only worry was that we couldn't produce the same code from identical sources, so I'm happy to learn that it wasn't so.

----- snip ----- re: Simplifying LaunchELF source releases
I considered changing this quite a while back. Maybe even getting rid of pre-compiled modules and only including the necessary source files like you did with ps2host. Unfortunately, we can't go that route for libcdvd since it relies on the old ps2lib which few people have.
Agreed. For such things the 'modules' folder is good.

I agree completely. Only other issue is that napalm's usbd.irx is not supposed to be distributed so I don't know if we want to start distributing it now.
Not if the author is against it. But perhaps we can get permission, or maybe even use a module from PS2SDK sometime in the future. For now we'll just leave it out of the release and tell people to add it themselves.

----- snip ---- re: possible need to synch our PS2SDK updates
Ok then. From what you said I assume we can keep going like we have been. I just got worried when I thought we got different binaries from identical source, but apparently this is no longer the case, so all is well.

BTW: I have put together a listing of all the flags for mcSetFileInfo so I'll reply to the main thread with all that info later on. So far I've found 3 flags although I'm sill not sure how to change them to a specific values:
1. sets the created date and time
2. sets the modified date and time
3. sets the attributes
Great. That will prove very useful for the mcPaste implementation.

Btw: I have now finished the implementation of the new CNF handler. Of course it can still be improved upon, but all the basic functionality is there.

During this work I also noticed a mistake I had made in the parser, for cases where variables are assigned values of empty strings. So I corrected that error and reposted the new source in the first post of this thread. This is mainly of interest to anyone who wants to implement similar parsing in other programs.

Best regards: dlanor