Well if you were on a custom firmware MKV's play fine in Showtime, but I assume you are not so UMS (aka Universal Media Server) is going to be your best option IMO. But if media server was your main goal you could have just bought a media hub that plays all formats of video for much cheaper and supports nearly 4 times as many codecs as a stock PS3.
PS3 CECH-2501A [CFW] & CECH-3001 [OFW]/ PS2 SCPH-3901 & SCPH-7501/ PS1 SCPH-5501
Mod Cobra 4.46 v7.00 w/ E3 Flasher [CFW] & Cobra ODE 4.50 [OFW]/ Mod HDD-OSD [Phat] & Modbo 4.0 [Slim]/ Mod MM3
HDD 1tb [CFW] & 640gb [OFW]/ HDD 640gb & OPL v0.92 w/ 1.5tb SMB