as for the installer's folders, no, they are no longer needed on your usb stick after installation unless you want to install on more memory cards.
as for running programs from the USB drive, you would copy ELF files to it. Attached is a shot of the directory structure I keep on my USB drive for my ps2. I tend to keep it fairly organized so I can find anything I want on it without running any searches quickly.
Then you would use the FMCB Configurator to create entries for the FMCB menu as well as set the path to the usb stick for each entry. djflexmaster79 has a video tutorial on how to do that floating around.
If you need help, post a question in the
forums, not my PM inbox. Thank you.