I've made movies with the DivX codec and those, in my experience, make the best movies. They take a while to encode, and are large, but have the best output quality.
I used mencoder to make a movie. The best encoding command line was,
mencoder -mc 0 -noskip -skiplimit 0 -ovc lavc -lavcopts \ codec=mpeg4:vhq:trell:mbd=2:vmax_b_frames=1:v4mv:vb_strategy=0:vlelim=0:vcelim=0:cmp=6:subcmp=6:precmp=6:predia=3:dia=3:vme=4:vqscale=1 \ mf://*.png -mf type=png:fps=18 -o output.avi
This gave me an 18 frame-per-second divx/mpeg4 movie with superb quality. The movie is large and is at Arrestin Vs Rhodopsin (DivX). The input are all the files ending in ".png" in the current directory; the output is output.avi, the movie. The three lines are actually on long one; concatenate them at the black slashes \.
Recent attention to transparencies and QuickTime show that one typically wants to set
set ray_opaque_background, off
to tell PyMol to stop using Alpha-channel for transparencies and switch to blending.
If you're having transparency problems this could be a fix.