Memory Problems - File Joiner

[Kayot]

Memory Problems - File Joiner

I've been working on a little utility that will join files on the PS3 internal HD. I'm hitting a bit of a problem though. The program reads 5mb from one file and appends it to another file. It does this till all the files in the directory are merged.

It works fine on the PC due to the fact that my PC has 8gigs of ram, however the PS3 only has 256mb of ram. I bolded where I think the problem is.

Basically the PS3 runs out of ram and crashed to XMB. What I don't understand is why. The buffer is only 5megs and it should flush when the files are closed. The same thing happens on the desktop. It occupies a ton of space fast.

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <time.h>

typedef struct {
	char*	sGameName;
	char*	sLongGameID;
	char*	sShortGameID;
} game;

void merge_filepiece(char* sFilePart, char* sFile) {
	FILE * pFile;
	FILE * gFile; //I got g and p backwards, sorry. This is the put file
	long lSize;
	
	//size_t result;

	int whole_number;
	long remainder;
	long chunk_size_mb = 5;
	long chunk_size_total = chunk_size_mb * 1000000;


	//Open all relivant files
	pFile = fopen ( sFilePart , "rb" );
	gFile = fopen ( sFile , "ab" );

	// obtain file size:
	fseek (pFile , 0 , SEEK_END);
	lSize = ftell (pFile);
	rewind (pFile);

	//break file size down for chunk management
	whole_number = (lSize / chunk_size_total);
	remainder = lSize - (whole_number*chunk_size_total);
	

	fclose (pFile);
	fclose (gFile);
	

    int x;
	for ( x = 0; x <= whole_number; x++ ) {
		char * buffer;
		pFile = fopen ( sFilePart , "rb" );
		gFile = fopen ( sFile , "ab" );

		if (x<whole_number) {
			fseek ( pFile , (x*chunk_size_total) , SEEK_SET );
			buffer = (char*) malloc (sizeof(char)*chunk_size_total);
			fread (buffer,1,chunk_size_total,pFile);
			fwrite (buffer , 1 , chunk_size_total , gFile );
		} else {
			if (remainder>0) {
				fseek ( pFile , (x*chunk_size_total) , SEEK_SET );
				buffer = (char*) malloc (sizeof(char)*remainder);
				fread (buffer,1,remainder,pFile);
				fwrite (buffer , 1 , remainder , gFile );
			}
		}
		fclose (pFile);
		fclose (gFile);
		
		free (buffer); //disengages the buffer
		
	}
	
	//close all files
	
	//fclose (pFile);
	//fclose (gFile);
}


void file_merge(char* sFileName) {
	FILE *istream;
	int iLoopCount = 1;
	char strFileNameExt[1024];
	char strFileName[1024];

	int x = 0;

	while ( x == 0 ) {
	sprintf(strFileNameExt, "%.03d", iLoopCount);
	sprintf(strFileName, "%s.%s", sFileName,strFileNameExt);

		if ( (istream = fopen ( strFileName, "rb" ) ) == NULL ) {
			x=1;
		} else {
			//printf ( "file \"%s\" exists!\n",strFileName ); //Since the PS3 has no output this is pointless
			iLoopCount++;
			fclose ( istream );
			merge_filepiece(strFileName,sFileName);
			//remove(strFileName); //Uncomment this before compilling for ps3 pkg files
		}
	}

	//return 0;

}

int main () {
	game p = {	"Test",
			"HWTP00001",
			"T00001"};

	char sMergeMePath[1024];
	sprintf(sMergeMePath, "/dev_hdd0/game/%s/USRDIR/MergeMe/z008_us.ps3.wmp",p.sLongGameID);

	//Use the whole ps3 path, thats the only way.
	file_merge(sMergeMePath);


	return 0;
}

I tried to use sleep() but that crashes the system and the sniglet that I found below:

void sleep(unsigned int mseconds)
{
    clock_t goal = mseconds + clock();
    while (goal > clock());
}

Won't compile on the PSL1GHT SDK. I get the error:

/usr/local/ps3dev/host/ppu/lib/gcc/ppu/4.5.2/../../../../ppu/lib/libc.a(lib_a-clock.o): In function `clock':
/usr/src/ps3toolchain/build/newlib-1.19.0/build-ppu/ppu/newlib/libc/time/../../../../../newlib/libc/time/clock.c:62: undefined reference to `._times_r'
collect2: ld returned 1 exit status
make[1]: *** [/home/kayot/Desktop/hello/hello.elf] Error 1
make: *** [build] Error 2

So anyone have any idea's what's wrong or how I can fix the program to not kill the system ram?

[Ben Jeremy]

I can't give this a lot of attention right now, but I'd say you probably shouldn't be allocating anything inside of a loop.

Best to allocate your buffer at the start of the app and free it at the end of the app.

I worked for quite a few years in the automotive industry in embedded systems, and loops are just horrible sources of all sorts of problems. I always look there first when debugging problems.

[Kayot]

I went ahead and redesigned it so it doesn't set any variables in a loop. My problem isn't a buffer overrun or anything like that. The program only uses 8 megs while merging. The problem is the ram isn't clearing fast enough leading to a no memory available situation.

In windows, when the program is running I see a spike in memory usage yet the program only reports using 8 megs. I need a flow control of the data, like maybe a max read speed set at 20mb/s.

Then again, if someone gets NTFS support on a poke bin this wouldn't be necessary and I could close this part of the project. Meh, I can dream.

Current Source:

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <time.h> //Has issues on PS3


typedef struct {
	char*	sFileName;
	int		iMain_Parts;
	int		iLast_Part;
	long	lMain_Remainder;
	long	lLast_Remainder;
	int		file_parts;
} file_details;

void read_write_file(char * sFileName_Part,char * buffer,long chunk_size_total,long chunk_size_total2, char * sFileName, int iPlace) {
	FILE * f_File_Final;
	FILE * f_File_Piece;
	
	f_File_Piece = fopen ( sFileName_Part, "rb" );
	fseek ( f_File_Piece , (iPlace*chunk_size_total) , SEEK_SET );
	fread (buffer,1,chunk_size_total2,f_File_Piece);
	fclose (f_File_Piece);
	
	f_File_Final = fopen ( sFileName , "ab" );
	fwrite (buffer , 1 , chunk_size_total2 , f_File_Final );
	fclose (f_File_Final);
}

void file_merge(file_details p) {
	char sFileName_Part[1024];
	
	//long chunk_size_mb = 5;
	long chunk_size_total = 50000000;
	
	//Buffer Data
	char * buffer = (char*) malloc (sizeof(char)*chunk_size_total);
	char * buffer_part = (char*) malloc (sizeof(char)*p.lMain_Remainder);	
	char * buffer_last_part = (char*) malloc (sizeof(char)*p.lLast_Remainder);
	
	for ( int x = 0; x <= p.file_parts-1; x++ ){
		sprintf(sFileName_Part, "%s.%.03d", p.sFileName,x+1);
		if (x < p.file_parts-1) {
			for ( int y = 0; y <= p.iMain_Parts; y++ ) {
				if (y < p.iMain_Parts) {
					read_write_file(sFileName_Part,
									buffer,
									chunk_size_total,
									chunk_size_total,
									p.sFileName,y);
				} else {
					if (p.lMain_Remainder>0) {
						read_write_file(sFileName_Part,
										buffer_part,
										chunk_size_total,
										p.lMain_Remainder,
										p.sFileName,y);
					}
				}
			}
		}
		char sOutPut[1024];
		sprintf(sOutPut,"x = %i -- File parts = %i \n",x,p.file_parts);
		printf(sOutPut);
		if (x == p.file_parts-1) {
			for ( int y = 0; y <= p.iLast_Part; y++ ) {
				if (y < p.iLast_Part) {
					read_write_file(sFileName_Part,
									buffer,
									chunk_size_total,
									chunk_size_total,
									p.sFileName,y);
				} else {
					if (p.lLast_Remainder>0) {
						read_write_file(sFileName_Part,
										buffer_last_part,
										chunk_size_total,
										p.lLast_Remainder,
										p.sFileName,y);
					}
				}
			}
		}
	}
	free (buffer); //disengages the buffer - I should probably not use this
}

void file_part_details (char* sFileName) {
	//This will do all the mesurements for merging so they only need to be done once.
	//int iParts_total;
	long lMain_Part_Count; //How many 5meg parts per full file
	long lMain_Remainder; //Remainder on each full file
	long lLast_Part_Count; //How many 5meg parts per last file
	long lLast_Remainder; //Remainder on last file
	long lCurrent_Part_Count;
	long lCurrent_Remainder;
	char sFileNameExt[1024];
	char sFileName_Part[1024];
	int iLoopCount = 1;
	FILE * f_File_Piece;
	
	long chunk_size_total = 5000000;

	long lSize;

	int x = 0;
	int y = 0;

	while ( x == 0) {
	sprintf(sFileNameExt, "%.03d", iLoopCount);
	sprintf(sFileName_Part, "%s.%s", sFileName,sFileNameExt);
		if ( (f_File_Piece = fopen ( sFileName_Part, "rb" ) ) == NULL ) {
			x=1;
			lLast_Part_Count = lCurrent_Part_Count;
			lLast_Remainder = lCurrent_Remainder;
		} else {
			iLoopCount++;
			// obtain file size:
			fseek (f_File_Piece , 0 , SEEK_END);
			lSize = ftell (f_File_Piece);
			fclose (f_File_Piece);

			lCurrent_Part_Count = 0;
			lCurrent_Remainder = 0;
			lCurrent_Part_Count = (lSize / chunk_size_total);
			lCurrent_Remainder = lSize - (lCurrent_Part_Count*chunk_size_total);

			if (y == 0) {
				lMain_Part_Count = lCurrent_Part_Count;
				lMain_Remainder = lCurrent_Remainder;
				y++;
			}
		}
	}

	file_details p = {
	sFileName,
	lMain_Part_Count,
	lLast_Part_Count,
	lMain_Remainder,
	lLast_Remainder,
	iLoopCount-1};
	
	file_merge(p);
}

Edit:

I've found that if I can slow the process speed down it works fine, however the following code won't compile on the PSL1GHT SDK

void sleep(unsigned int mseconds)
{
    clock_t goal = mseconds + clock();
    while (goal > clock());
}

Can someone see if they can get that to compile?

[Ben Jeremy]

I see you have three mallocs, but only one free... make sure everything is matched up.

"Back in the day" we had several different libraries to check this sort of thing, like MemCheck. It would report where we allocated memory in the program that was never freed up. It's a shame the PS3 SDK doesn't have something like this.

It's also a good reason to code in C++ - you can use smart pointers that would clean themselves up at the end of a function (or scope) when declared locally.

At any rate, I think the missing free calls are what's left for you to fix.

[Kayot]

Added:

	free (buffer_part);
	free (buffer_last_part);

I think that's what you were talking about right?

I wish sleep() would work. When I put a .5 second wait between file closes the ram clears in time for the next loop. The PS3 is just plain unforgiving. That or my SDK is bad still.