Video codec
I've spent most of this study focused on selecting good parameters for lavc. The results I expose here should give you entire satisfaction. However, it would be very naive to simply pretend that "those are the parameters you should use, they are the best conbination". It would also be very naive to believe me. That's why I present several alternatives in my conclusion, for example. But most importantly: I took the pain to describe in details the tests I did and left you free to exerce your critical eye.
I recommend to always use a 2-pass encode, with lavc. lavc can be pushed further than xvid at high bitrates.
With B frames
B frames save both time and bits, so I recommend using them first. The first pass is using the following set of parameters:
-ovc lavc -lavcopts
vbitrate=1200:mbd=2:v4mv:trell:cbp:mv0:last_pred=3:preme=2:precmp=3:cmp=2:subcmp=3:predia=3:dia=4:bidir_refine=2:vmax_b_frames=1:vb_strategy=1:qns=3:qprd:vqmin=1:vqmax=5:vpass=1
And the second pass...
-ovc lavc -lavcopts
vbitrate=1200:mbd=2:v4mv:trell:cbp:mv0:last_pred=3:preme=2:precmp=3:cmp=2:subcmp=3:predia=3:dia=4:bidir_refine=2:vmax_b_frames=1:qns=3:qprd:vqmin=1:vqmax=5:vpass=2:autoaspect
Note the use of qns=3:qprd:vqmax=5.
You should actually use either qns=1:qprd:vqmax=6
or qns=3:qprd:vqmax=4, depending on your
video and the selected bitrate. qns=3 will only waste time and quality
if your bitrate is not high enough to allow vqmax=4. The only way I
know to find out is to test a "long" sequence at vqmax=8 and count the
number of frames with a quantitizer larger than 4 in the psnr log.
Without B frames
In some cases (scenes with transparent of reflected layers, for example), B-frames produce bad results (that's the reason for using vb_strategy=1). An advantage of avoiding B-frames is also to be able to include the chroma component in motion estimation, whitch boosts quality (and slow you down a lot). The two passes use the same set of parameters:
-ovc
lavc
-lavcopts
vbitrate=1200:mbd=2:v4mv:trell:cbp:mv0:last_pred=3:preme=2:precmp=259:cmp=258:subcmp=262:predia=3:dia=4:vpass=½:autoaspect
Faster alternatives
With lavc
In genral you can save time at
little quality loss using last_pred=2:precmp=1:predia=3:dia=-1
instead of their respective values above. When you've done this, you
probably want to remove also any vqmin, vqmax, qprd or qns parameter
for a bigger speed boost.
With xvid
XVid is very fast (and very simple!). So, if you are in a hurry, use the following set of parameters in 2 passes:
-ovc xvid -xvidencopts
bitrate=1200:vhq=4:autoaspect:pass=½
Filters
hqdn3d is an excellent filter that surely improves the final result. Don't hesitate to de-interlace, there's just no other way! And, on MPEG2 sources, get a small improvement with pp=ac.
Putting it all together
While you can get excelent results from progressive material, lavc or xvid are really not good enough to archive interlaced sources such as DV. I will have to stick to DV for now and wait for H.264 to be more common and well supported by portable or set-top players. More specifially, x264 will support propper interlaced encoding and it is under active developpement. In the mean time you may consult the conclusion of the 4th test.