Encoding parameters - fourth test

Home
Parameters notes
Encoding parameters
Second test (old video)
Third test (TV play)
Fourth test (action DVD)

• Introduction
• Motion estimation, motion search range and B-frames
• Statistics to the rescue
• More options
• Fine tunning
• Comparisons
• Conclusion

Filtering
Conclusion

Introduction

For this fourth test I use a 4 minutes sequence of a recent action movie which has everything for a challenging test: dark scenes, bright scenes, slow motion, smoke, rain, fire, water surface, splashes and some particle effects. The raw source is progressive and rather clean from noise.

Good news: I got hold of a dual Xeon PC. Bad news: it's running XP. The MEncoder binary for Windows is getting old and its performances (both speed and quality) are falling behind. However, I expect the test conditions to be relevant for base parameters. This is an opportunity to explore a rather large number of combinations in a reasonable time (2 weeks!!!).

From previous experiences and other short tests I found that a 3 pass encode could significantly improve quality while not taking much longer time, when using "turbo" during the first pass. Surprisingly, using the turbo for just the first pass would in fact improve quality. I have also noticed that B-frames could prove rather destructive during scenes involving smoke or some kind of transparency.

As I wrote earlier, precmp should be tested afterwards. But in this test I will assume all parameters I usually adjusted afterwards so, for example, precmp will be set to 3. The following parameters remain constants during the tests:

• -mc 0 -noskip -skiplimit 0 -nosound
• -ovc lavc -lavcopts mbd=2:v4mv:trell:cbp:mv0:last_pred=3:preme=2:vqcomp=0.6:precmp=3:vpass=1/3/3:(turbo):vbitrate=900
• -vf crop=720:416:0:80,scale=-1:288

turbo is only active during the first pass. And since I had to run that test on XP, I also had to write a new test script (Warning: this links directly to a VBScript and the file has the "txt" extension instead of "vbs" to prevent mistakes of our Windows friends ;-).

Motion estimation, motion search range and B-frames

This time I will test the combinations of 5 parameters: max_bframes, cmp, subcmp, predia and dia. For cmp and subcmp, I will try the functions SAD, SSE, SATD and DCT numbered 1, 2, 3 and 4 respectively as before. For the diamonds sizes I will try -3, -2, -1, 1, 2, 3 and 4 respectively numbered from 1 to 7. B-frames will be either limited to 0 or to 1. There are 1568 combinations to test.

The following table shows the best results sorted by average PSNR. There is no "user time" because I don't know how to do that on XP.

Without B-frames, the combination cmp=2:subcmp=6 a clear winner (it's the first time DCT is giving good results for me) and dia=4 is popular. The choice of predia is more difficult but predia=-1 or 4  is well represented at the top.

With B-frames, higher average PSNR can be reached. But the standard deviation is larger and both min. and max. are lower. The combination cmp=2:subcmp=3 is popular as usual and dia=4:predia=4 is winning. As suspected after the third test, the quality increases with the diamond sizes. Surprisingly, cmp=1 is also well represented. 

no B-frames						max 1 B-frames
set	Log PSNR					set	Log PSNR
	Min.	Mean	Max.	SD			Min.	Mean	Max.	SD
2477	38.76	44.966	59.5	2.041		2377	38.07	44.983	58.21	2.11
2437	38.76	44.966	59.55	2.04		2277	38.05	44.981	58.19	2.111
2417	38.74	44.966	59.73	2.047		3377	38.1	44.971	58.07	2.11
2467	38.75	44.965	59.65	2.048		3277	38.07	44.97	58.03	2.109
2475	38.75	44.964	59.57	2.047		1337	38.15	44.969	58.23	2.098
2473	38.76	44.964	59.47	2.044		3373	38.08	44.968	58.03	2.111
2463	38.75	44.964	59.6	2.049		1376	38.15	44.968	58.19	2.099
2436	38.75	44.964	59.53	2.047		1367	38.15	44.968	58.39	2.1
2413	38.74	44.964	59.69	2.048		1356	38.06	44.968	58.35	2.099
3477	38.76	44.963	59.54	2.045		1336	38.06	44.968	58.32	2.099
2476	38.74	44.963	59.62	2.047		1327	38.15	44.968	58.37	2.098
2457	38.75	44.963	59.54	2.054		1317	38.15	44.968	58.35	2.099
2433	38.74	44.963	59.65	2.046		1316	38.07	44.968	58.28	2.1
2416	38.75	44.963	59.55	2.052		2317	38.09	44.967	58.54	2.117
3475	38.75	44.962	59.5	2.047		1377	38.15	44.967	58.16	2.097
3467	38.75	44.962	59.57	2.052		1366	38.07	44.967	58.25	2.099
3417	38.74	44.962	59.61	2.046		1365	38.05	44.967	58.31	2.1
2426	38.75	44.962	59.7	2.057		1357	38.15	44.967	58.35	2.099
3476	38.75	44.961	59.51	2.047		1347	38.15	44.967	58.11	2.098
3473	38.74	44.961	59.55	2.048		1326	38.15	44.967	58.29	2.101
3437	38.76	44.961	59.58	2.046		1247	38.12	44.967	58.02	2.093
3427	38.75	44.961	59.63	2.048		1217	38.12	44.967	58.36	2.099
2453	38.75	44.961	59.54	2.052		2337	38.08	44.966	58.13	2.113
2435	38.74	44.961	59.61	2.049		1355	38.04	44.966	58.36	2.102
2415	38.74	44.961	59.68	2.053		1353	38.15	44.966	58.36	2.098
3457	38.76	44.96	59.59	2.05		1346	38.15	44.966	58.25	2.099
3447	38.75	44.96	59.51	2.052		1345	38.14	44.966	58.31	2.096
3413	38.75	44.96	59.61	2.047		1313	38.14	44.966	58.27	2.095
2474	38.74	44.96	59.49	2.047		1277	38.13	44.966	58.14	2.098
2434	38.76	44.96	59.56	2.049		1267	38.13	44.966	58.26	2.099
2423	38.75	44.96	59.56	2.045		1256	38.12	44.966	58.25	2.1
3463	38.74	44.959	59.46	2.052		1246	38.12	44.966	58.29	2.1
3433	38.76	44.959	59.59	2.046		1237	38.13	44.966	58.12	2.097
2431	38.74	44.959	59.63	2.052		1236	38.12	44.966	58.09	2.092
2427	38.75	44.959	59.67	2.045		1227	38.12	44.966	58.3	2.097
3474	38.75	44.958	59.77	2.049		1226	38.12	44.966	58.3	2.099
3446	38.75	44.958	59.64	2.055		3273	38.06	44.965	57.99	2.111
3436	38.75	44.958	59.43	2.047		2357	38.08	44.965	58.37	2.117
3466	38.75	44.957	59.63	2.049		2217	38.07	44.965	58.44	2.116
3465	38.74	44.957	59.67	2.05		1375	38.04	44.965	58.25	2.099
3456	38.75	44.957	59.62	2.054		1335	38.13	44.965	58.32	2.1
3453	38.75	44.957	59.56	2.048		1325	38.13	44.965	58.22	2.095
3443	38.74	44.957	59.76	2.054		1315	38.04	44.965	58.28	2.099
3435	38.76	44.957	59.49	2.046		1276	38.11	44.965	58.19	2.097
3424	38.75	44.957	59.57	2.06		1266	38.12	44.965	58.3	2.093
3423	38.75	44.957	59.64	2.053		1265	38.09	44.965	58.3	2.101
3416	38.75	44.957	59.6	2.048		1257	38.13	44.965	58.24	2.098
2471	38.74	44.957	59.65	2.051		2367	38.08	44.964	58.36	2.116
3471	38.76	44.956	59.71	2.051		1371	38.06	44.964	58.31	2.106
3451	38.75	44.956	59.79	2.058		1363	38.05	44.964	58.35	2.101
3426	38.74	44.956	59.63	2.054		1343	38.14	44.964	58.16	2.094
3425	38.75	44.956	59.65	2.054		1333	38.05	44.964	58.24	2.094
3415	38.74	44.956	59.69	2.051		1323	38.06	44.964	58.33	2.101
3455	38.75	44.955	59.64	2.056		1321	38.06	44.964	58.33	2.106
3432	38.76	44.955	59.7	2.053		1275	38.09	44.964	58.21	2.1
3414	38.74	44.955	59.57	2.057		1255	38.08	44.964	58.32	2.1
3411	38.75	44.955	59.69	2.06		1215	38.09	44.964	58.2	2.099
3472	38.76	44.954	59.61	2.052		2327	38.09	44.963	58.4	2.12
3464	38.75	44.954	59.72	2.055		1361	38.06	44.963	58.33	2.105
3462	38.76	44.954	59.66	2.054		1311	38.06	44.963	58.33	2.105
3461	38.75	44.954	59.53	2.054		1261	38.03	44.963	58.26	2.104
3434	38.75	44.954	59.68	2.053		1245	38.09	44.963	58.3	2.101
3431	38.76	44.953	59.69	2.055		1243	38.11	44.963	58.07	2.093
1467	38.83	44.951	59.52	2.04		1235	38.08	44.963	58.06	2.097
2367	38.76	44.95	59.48	2.049		1225	38.09	44.963	58.24	2.1

Statistics to the rescue

The frequency table makes it easier to obtain an overview of the test results:

average PSNR partition	cmp				subcmp				predia							dia
	1	2	3	6	1	2	3	6	-3	-2	-1	1	2	3	4	-3	-2	-1	1	2	3	4
no B-frames
44.84 - 44.86	0	3	3	13	19	0	0	0	4	5	3	3	3	0	1	2	10	0	4	2	0	1
44.86 - 44.88	0	23	4	40	63	3	1	0	7	9	5	16	11	12	7	11	10	7	11	10	11	7
44.88 - 44.9	45	52	21	110	45	82	82	19	33	35	26	43	37	30	24	42	48	23	36	33	25	21
44.9 - 44.92	4	51	51	33	69	1	4	65	19	18	19	21	19	22	21	19	14	22	19	19	23	23
44.92 - 44.94	98	4	36	0	0	79	59	0	18	21	24	16	19	23	17	22	19	18	22	21	21	15
44.94 - 44.96	49	43	70	0	0	31	50	81	25	21	29	12	20	22	33	16	11	34	18	23	27	33
44.96 - 44.98	0	20	11	0	0	0	0	31	6	3	6	1	3	3	9	0	0	8	2	4	5	12
max 1 B-frame
44.82 - 44.83	0	0	0	11	6	0	0	5	2	2	1	2	2	0	2	0	0	0	11	0	0	0
44.83 - 44.84	0	0	0	14	7	0	0	7	3	3	1	2	3	2	0	0	8	0	3	3	0	0
44.84 - 44.85	0	0	0	28	14	0	0	14	3	3	6	4	3	4	5	11	6	0	0	11	0	0
44.85 - 44.86	0	0	0	32	15	2	2	13	4	4	4	6	4	5	5	3	0	11	4	0	14	0
44.86 - 44.87	0	0	0	25	7	5	4	9	4	6	2	2	4	3	4	0	3	3	6	0	0	13
44.87 - 44.88	0	0	0	19	0	10	8	1	2	2	4	3	3	4	1	0	8	0	4	6	0	1
44.88 - 44.89	0	1	0	22	1	9	13	0	4	2	4	5	3	2	3	10	3	0	1	8	1	0
44.89 - 44.9	1	4	7	27	12	14	13	0	7	6	5	5	7	4	5	4	3	12	9	0	11	0
44.9 - 44.91	0	16	23	18	27	9	9	12	8	9	7	9	7	10	7	8	15	2	13	2	3	14
44.91 - 44.92	19	27	26	0	45	1	1	25	7	12	10	10	10	11	12	19	17	3	12	15	6	0
44.92 - 44.93	49	32	49	0	53	13	11	53	21	15	21	19	19	19	16	12	13	22	27	19	22	15
44.93 - 44.94	29	44	34	0	7	26	24	50	16	19	14	14	16	15	13	20	21	16	8	10	13	19
44.94 - 44.95	1	29	29	0	1	30	23	5	8	8	9	9	8	7	10	11	2	6	0	21	14	5
44.95 - 44.96	6	27	24	0	1	27	28	1	8	7	8	8	9	10	7	0	5	17	1	3	14	17
44.96 - 44.97	91	14	3	0	0	49	58	1	15	14	16	14	14	16	19	14	8	20	13	14	14	25
44.97 - 44.98	0	0	1	0	0	0	1	0	0	0	0	0	0	0	1	0	0	0	0	0	0	1
44.98 - 44.99	0	2	0	0	0	1	1	0	0	0	0	0	0	0	2	0	0	0	0	0	0	2

• Don't use cmp=6, use cmp=2. With B-frames you can also try cmp=1, but without B-frames cmp=3 can be interesting.
• Don't use subcmp=1, use subcmp=3. subcmp=2 could also be used with B-frames. Without B-frames the best is subcmp=6.
• predia has little effect, it should be set equal to dia.
• dia=3 is good, but -1 can be better (and faster). If you have time, however, 4 is best.
• B-frames are good. But some scenes don't render well with them.

More options

Now let's experiment with the narrowed combinations described above, using the current Debian version of MEncoder: dev-CVS--4.0.3

No B-frames

Without B-frames, I would like to check cmp = 2 or 3, subcmp=6 and [pre]dia = -1, 3 or 4. 18 combinations...

set	user time	Log PSNR				ImageMagick MAE				Log quantitizer
		Min.	Mean	Max.	SD	Min.	Mean	Max.	SD	Min.	Mean	Max.	SD
32436	17m45s	36.82	44.98	59.72	2.082	31.635	519.17	1114.9	110.1	2	3.439	7	0.758
32467	19m40s	36.83	44.988	59.6	2.095	27.382	519.22	1116.8	111.3	2	3.433	7	0.7664
32466	17m51s	36.83	44.982	59.71	2.09	23.9	519.25	1118.4	110.4	2	3.439	7	0.7617
32476	19m31s	36.82	44.983	59.55	2.088	30.91	519.25	1115.6	110.6	2	3.436	7	0.7605
32437	19m21s	36.84	44.986	59.74	2.093	30.046	519.27	1113.9	111	2	3.433	7	0.7641
32463	17m24s	36.83	44.983	59.7	2.091	30.717	519.31	1116.5	110.9	2	3.433	7	0.7637
32473	18m56s	36.83	44.985	59.7	2.094	27.844	519.36	1116.4	111.2	2	3.433	7	0.7635
33467	30m06s	36.68	44.981	59.73	2.092	30.193	519.41	1126.6	111.1	2	3.442	7	0.766
32433	17m05s	36.83	44.983	59.47	2.094	29.592	519.46	1114.8	111.2	2	3.433	7	0.7648
32477	21m02s	36.84	44.984	59.72	2.093	27.898	519.51	1121.1	111.3	2	3.434	7	0.7631
33466	26m17s	36.83	44.974	59.6	2.089	28.855	519.62	1115.6	110.9	2	3.447	7	0.7616
33436	25m04s	36.84	44.979	59.58	2.092	29.761	519.67	1114.4	111.1	2	3.443	7	0.7667
33476	29m27s	36.69	44.977	59.82	2.095	31.267	519.69	1126.3	111.4	2	3.443	7	0.7665
33463	24m45s	36.82	44.977	59.53	2.092	27.133	519.7	1116.6	110.9	2	3.445	7	0.7642
33433	23m24s	36.84	44.976	59.68	2.094	27.214	519.72	1114.8	111.2	2	3.444	7	0.7663
33437	28m35s	36.85	44.976	59.47	2.094	28.33	519.8	1115	111.3	2	3.444	7	0.7643
33473	27m59s	36.69	44.977	59.55	2.094	29.122	519.81	1126.3	111.3	2	3.444	7	0.765
33477	32m49s	36.69	44.977	59.54	2.094	28.787	519.87	1126.3	111.3	2	3.442	7	0.7669

The results are sorted according to MAE and the best PSNR are marked. I would recommend the following:

• Extreme settings: cmp=2:subcmp=6:predia=3:dia=4
• Very good settings, not so slow: cmp=2:subcmp=6:predia=-1:dia=3

Max 1 B-frame

With up to 1 B-frame, I would like to check cmp = 1 or 2, subcmp= 2 or 3 and [pre]dia = -1, 3 or 4. 36 combinations...

set	user time	Log PSNR				ImageMagick MAE				Log quantitizer
		Min.	Mean	Max.	SD	Min.	Mean	Max.	SD	Min.	Mean	Max.	SD
32367	16m34s	36.49	45.016	58.41	2.119	23.091	508.47	1158.3	106.7	2	3.959	8	1.286
32337	16m26s	36.19	45.015	58	2.117	26.5	508.51	1200.3	106.9	2	3.96	8	1.286
32336	15m20s	36.46	45.01	57.94	2.117	25.826	508.58	1161.9	106.7	2	3.96	8	1.285
32363	14m51s	36.48	45.012	58.3	2.12	24.097	508.59	1160.3	106.8	2	3.959	8	1.287
32267	22m33s	36.49	45.014	58.39	2.122	23.206	508.64	1157.7	106.8	2	3.951	8	1.287
32377	16m50s	36.22	45.014	58.16	2.117	24.714	508.69	1197.1	106.7	2	3.961	8	1.286
32263	18m09s	36.48	45.012	58.35	2.121	23.734	508.69	1159.5	106.8	2	3.949	8	1.286
32237	22m19s	36.18	45.012	58	2.116	25.372	508.71	1202.9	106.8	2	3.952	8	1.286
32333	14m41s	36.48	45.012	57.95	2.117	26.838	508.72	1157.4	106.7	2	3.959	8	1.286
32236	19m11s	36.45	45.008	58.05	2.119	24.823	508.73	1162.4	106.8	2	3.952	8	1.285
32277	23m18s	36.46	45.011	58.01	2.117	25.198	508.76	1162.4	106.7	2	3.952	8	1.285
32233	17m52s	36.17	45.011	58.06	2.123	25.449	508.83	1199.7	107	2	3.95	8	1.288
32373	15m06s	36.47	45.009	57.97	2.118	25.757	508.85	1161.3	106.7	2	3.959	8	1.285
32273	18m55s	36.45	45.008	57.95	2.117	25.434	508.85	1164.4	106.7	2	3.951	8	1.285
32376	15m46s	36.62	44.997	58.04	2.111	24.962	509.1	1141.3	105.7	2	3.971	8	1.285
32276	20m14s	36.59	44.996	58.05	2.112	24.831	509.21	1147.2	105.9	2	3.96	8	1.284
31237	20m52s	36.06	44.997	58.02	2.111	24.98	509.29	1214.1	106.8	2	3.908	8	1.274
31367	15m27s	36.06	44.996	57.79	2.11	27.796	509.3	1213.2	106.8	2	3.915	8	1.271
32366	15m29s	37.2	44.991	58.45	2.126	22.862	509.32	1069.3	105.9	2	3.981	8	1.294
31267	21m03s	36.04	44.997	57.91	2.111	26.255	509.33	1216.2	106.8	2	3.907	8	1.274
31376	14m40s	36.05	44.995	58.02	2.113	25.141	509.34	1213.2	106.8	2	3.916	8	1.272
31336	14m16s	36.04	44.994	58.09	2.114	24.987	509.35	1215.4	106.9	2	3.917	8	1.273
31366	14m24s	36.06	44.995	57.94	2.112	25.241	509.35	1213.9	106.8	2	3.916	8	1.272
31377	15m43s	36.06	44.996	58.13	2.113	25.181	509.36	1212.6	106.9	2	3.916	8	1.273
32266	19m23s	37.2	44.989	58.48	2.125	22.875	509.38	1069.1	105.9	2	3.972	8	1.291
31236	17m50s	36.05	44.995	58.11	2.112	25.01	509.4	1214.2	106.9	2	3.907	8	1.273
31337	15m20s	36.04	44.995	58.02	2.112	25.176	509.41	1214.3	106.8	2	3.916	8	1.272
31276	18m46s	36.04	44.996	58.03	2.113	25.06	509.43	1215.2	106.8	2	3.908	8	1.274
31363	13m41s	36.12	44.993	57.87	2.114	25.504	509.44	1208.3	106.9	2	3.919	8	1.274
31266	17m56s	36.03	44.994	58.07	2.113	24.275	509.45	1216.5	106.9	2	3.909	8	1.274
31277	21m49s	36.05	44.997	57.87	2.111	26.126	509.45	1214.2	106.7	2	3.907	8	1.274
31333	13m35s	36.08	44.992	57.98	2.114	26.106	509.53	1212.1	106.8	2	3.921	8	1.273
31373	13m57s	36.11	44.991	57.96	2.113	25.393	509.53	1208.5	106.8	2	3.92	8	1.273
31233	16m26s	36.09	44.992	57.75	2.111	26.623	509.54	1213.9	106.8	2	3.911	8	1.272
31263	16m39s	36.09	44.992	58.05	2.112	25.104	509.59	1212.1	106.7	2	3.91	8	1.274
31273	17m25s	36.09	44.992	58.04	2.113	25.534	509.6	1213.9	106.7	2	3.911	8	1.274

The results are sorted according to MAE and the best PSNR are marked. I would recommend the following: cmp=2:subcmp=3:predia=3:dia=4. But predia=-1 would be good too.

Fine tunning

Let's test some variations based on the previous recommendations, just to check that the initial assumptions were correct.

No B-frames

set	user time	Log PSNR				ImageMagick MAE				Log quantitizer
		Min.	Mean	Max.	SD	Min.	Mean	Max.	SD	Min.	Mean	Max.	SD
reference	19m34s	36.83	44.988	59.6	2.095	27.382	519.22	1116.8	111.3	2	3.433	7	0.7664
last_pred1	16m37s	36.91	44.979	59.64	2.081	29.283	519.01	1116.6	109.9	2	3.441	7	0.7557
last_pred2	17m43s	36.82	44.986	59.79	2.085	27.172	518.85	1123.4	110.2	2	3.437	7	0.7577
last_pred4	21m58s	36.62	44.983	59.64	2.094	32.796	519.67	1137.3	111.5	2	3.431	7	0.764
precmp1	16m57s	36.63	44.981	59.62	2.094	27.387	519.71	1140.9	111.7	2	3.433	7	0.7639
precmp2	18m09s	36.81	44.984	59.57	2.092	30.614	519.5	1113	111.2	2	3.433	7	0.7633
preme	14m35s	36.87	44.983	59.57	2.085	27.465	519	1119.8	110.3	2	3.438	7	0.757
turbo	27m12s	36.86	44.974	59.49	2.068	28.727	519.38	1120.8	110	2	3.446	7	0.7221
vqcomp	19m38s	35.55	45.016	59.79	2.197	26.634	519.41	1252.2	116.1	2	3.403	9	0.8938
vqdiff	19m34s	36.81	44.986	59.56	2.089	31.141	519.26	1122.4	111	2	3.431	7	0.7626
vqscale	19m45s	36.48	44.972	59.58	2.099	30.458	520.09	1149.3	111.6	2	3.444	7	0.7838

Using last_pred=3:precmp=3:preme=2:(turbo) was good. I'm still amazed that using the turbo in n-pass mode can improve quality while saving time. Not using vqdiff=2 or vqscale=2 was also good, but leaving vqcomp to its default would have been better. The new reference in this case is:
mbd=2:v4mv:trell:cbp:mv0:last_pred=3:preme=2:precmp=3:cmp=2:subcmp=6:predia=3:dia=4:vpass=1/3/3:(turbo)

There is one new thing I would like to test, which only works without B-frames: using chroma on *cmp functions. This means using precmp=259:cmp=258:subcmp=262. The new vqcomp=0.5 selection is in use and I also check a 2-pass encode without turbo:

set	user time	Log PSNR				Log quantitizer
		Min.	Mean	Max.	SD	Min.	Mean	Max.	SD
xcmp	26m42s	35.6	45.045	60	2.214	2	3.397	9	0.8974
xcmp 2-pass, no turbo	25m40s	36.1	45.043	59.98	2.235	2	3.386	8	0.9764

Including chroma was a great idea: the PSNR is reaching even higher than with B-frames. The 3 passes didn't make much difference but they were not much expensive either, let's stick to 2-pass to keep it simple. The winner is:
mbd=2:v4mv:trell:cbp:mv0:last_pred=3:preme=2:precmp=259:cmp=258:subcmp=262:predia=3:dia=4:vpass=1/2

Max 1 B-frame

set	user time	Log PSNR				ImageMagick MAE				Log quantitizer
		Min.	Mean	Max.	SD	Min.	Mean	Max.	SD	Min.	Mean	Max.	SD
reference	22m31s	36.49	45.016	58.41	2.119	23.091	508.47	1158.3	106.7	2	3.959	8	1.286
last_pred1	19m59s	37.23	44.994	58.5	2.121	22.879	509.1	1066.3	105.7	2	3.984	8	1.293
last_pred2	20m55s	37.24	44.997	58.39	2.119	22.925	509.01	1066.6	105.7	2	3.985	8	1.293
last_pred4	24m37s	36.18	45.015	58.47	2.124	23.25	508.66	1201.3	107	2	3.961	8	1.287
precmp1	21m17s	36.2	45.012	58.17	2.121	24.636	508.72	1199.1	107.2	2	3.959	8	1.29
precmp2	21m52s	36.21	45.013	58.39	2.117	24.03	508.64	1196.9	106.8	2	3.96	8	1.287
preme	19m57s	36.51	45.018	58.53	2.121	23.16	508.31	1158.3	106.7	2	3.965	8	1.288
turbo	30m52s	36.64	45.003	58.14	2.101	24.422	508.85	1140.1	106	2	3.995	8	1.28
vqcomp	22m31s	36.66	45.028	58.42	2.17	24.049	508.63	1142.6	108.4	2	3.958	8	1.321
vqdiff	22m31s	36.12	45.015	58.41	2.118	23.091	508.48	1209.7	106.7	2	3.96	8	1.283
vqscale	22m49s	36.27	45.012	58.54	2.135	22.805	508.77	1192.2	106.6	2	3.939	8	1.288

Using last_pred=3:precmp=3:(turbo) was good. Not using vqdiff=2 or vqscale=2 was also good, but leaving vqcomp to its default would have been better. It looks like preme=1 would have been better too, but preme=2 is required to take advantage of predia. The new reference I would use is:
mbd=2:v4mv:trell:cbp:mv0:last_pred=3:preme=2:precmp=3:cmp=2:subcmp=3:predia=3:dia=4:vmax_b_frames=1:vpass=1/3/3:(turbo)

There are a few more tests to make: using vb_strategy=1 and higher values of bidir_refine, which are only relevant for B-frames. On top of that, vb_strategy gives more weight to the first pass so it's worth trying to remove the turbo and test only 2 pass...

set	user time	Log PSNR				Log quantitizer
		Min.	Mean	Max.	SD	Min.	Mean	Max.	SD
bidir_refine=1	23m45s	35.43	45.033	58.15	2.18	2	3.952	9	1.323
bidir_refine=2	26m10s	36.11	45.036	58.52	2.183	2	3.951	8	1.326
bidir_refine=3	29m30s	35.44	45.036	58.42	2.182	2	3.948	9	1.322
bidir_refine=4	31m9s	36.1	45.034	58.52	2.182	2	3.949	8	1.321
vb_strategy=1	20m21s	36.23	45.059	59.7	2.178	2	3.701	9	1.039
vb_strategy=1, no turbo	28m7s	36.35	45.054	59.57	2.152	2	3.712	9	1.018
vb_strategy=1, no turbo, 2-pass	18m51s	36.37	45.06	59.72	2.189	2	3.674	8	1.037
vb_strategy=2	25m54s	36.24	45	59.66	2.199	2	3.414	8	0.8989
vb_strategy=1, max 2 B-frames	20m50s	36.44	45.044	59.67	2.168	2	3.791	8	1.058
bidir_refine=2:vb_strategy=1	21m30s	36.24	45.06	59.7	2.178	2	3.698	9	1.043
bidir_refine=2:vb_strategy=1, no turbo, 2-pass	19m21s	36.3	45.069	59.66	2.196	2	3.668	8	1.04
vb_strategy=1, no turbo, 2-pass and chroma *cmp	25m45s	36.35	45.081	59.96	2.198	2	3.673	8	1.036
bidir_refine=2:vb_strategy=1, no turbo, 2-pass and chroma *cmp	26m18s	36.39	45.088	59.99	2.204	2	3.667	8	1.039

bidir_refine and vb_strategy sure did improve the PSNR. Removing the turbo was necessary to make full use of vb_strategy, so I ended up with only 2 passes instead of 3. You can also notice an attempt to use chroma on motion estimation even though the documentation mentions that it doesn't work correctly with B-frames: it worked and it paid off (but it's expensive!). The winner is:
mbd=2:v4mv:trell:cbp:mv0:last_pred=3:preme=2:precmp=3:cmp=2:subcmp=3:predia=3:dia=4:bidir_refine=2:vmax_b_frames=1:vb_strategy=1:vpass=1/2
And if you dare, you can try with precmp=259:cmp=258:subcmp=259.

Comparisons

Now I would like to show some curiosity for the expensive qns and qprd as usual, but also try things like dia=-10 and even XviD or x264. The parameters used to test XviD or x264 are taken from the best recommended options of the MPlayer manual.

set	user time	Log PSNR				ImageMagick MAE				Log quantitizer
		Min.	Mean	Max.	SD	Min.	Mean	Max.	SD	Min.	Mean	Max.	SD
ref0	25m17s	36.1	45.043	59.98	2.235	22.803	517.45	1210.4	114.3	2	3.386	8	0.9764
ref0dia	25m56s	35.36	45.043	59.98	2.254	22.384	517.71	1299.1	116.6	2	3.388	9	0.9912
ref0qns1	44m27s	35.5	45.053	59.91	2.25	22.663	515.24	1248	115.3	2	3.266	8	0.9491
ref0qns2	47m16s	35.42	45.048	60.02	2.264	22.671	515.63	1255.6	115.8	2	3.309	8	0.966
ref0qns3	70m27s	35.4	45.039	60.01	2.26	22.534	515.87	1256.1	115	2	3.331	8	0.9652
ref0qprd	27m13s	35.92	45.038	59.8	0.2782	24.315	517.27	1214.1	113.4	2	3.589	21.15	0.07696
ref1	19m31s	36.3	45.069	59.66	2.196	28.407	513.2	1176.1	113.7	2	3.668	8	1.04
ref1dia	19m58s	36.12	45.061	59.62	2.207	30.582	513.86	1201.6	115.3	2	3.677	8	1.051
ref1qns1	37m44s	35.77	45.054	59.51	2.191	29.826	511.52	1216.3	113.3	2	3.562	8	1.019
ref1qns2	39m50s	36	45.055	59.62	2.208	32.545	511.77	1186.2	114	2	3.604	8	1.03
ref1qns3	59m56s	36.16	45.053	59.66	2.21	29.708	511.91	1168	113.6	2	3.62	8	1.027
ref1qprd	21m31s	36.77	45.054	59.46	0.4619	28.693	513.06	1127.1	111	2	3.867	17.26	0.2635
x264	30m38s	-	45.641	-	-	44.214	484.03	1062.2	105.2	-	-	-	-
xvid	7m58s	37.61	44.502	55.69	2.099	38.414	521.77	1086.2	109.8	2	4.656	8	1.399

XviD was amazingly fast. It has the lowest quality of the test but it might be possible to push it up, given enough testing. x264 is excellent (and slow), but I had problems to play it with many media players and right now it doesn't look like a suitable alternative for archiving. dia=10 was not actually interesting.

To evaluate the effect of qns and qprd, I can't simply rely on psnr. I use (at last) my old script to build a mosaic of the frames and ease up a visual comparison. My conclusion is that qns=1 might have helped when not using B-frames, but in general it was best without qprd or qns for this type of movie. Another remark is that using B-frames gave better results than not using any and went faster, but I noticed during the visual comparison some color shifts: like a green or red stain around detailed areas. My guess is that it would have helped to use the chroma *cmp even with B-frames. An other thing I would like to check: is it OK to allow up to 2 B-frames when vb_strategy is set to 1 and limits it?...

set	user time	Log PSNR				Log quantitizer
		Min.	Mean	Max.	SD	Min.	Mean	Max.	SD
ref1	19m31s	36.3	45.069	59.66	2.196	2	3.668	8	1.04
ref1 with chroma *cmp	26m18s	36.39	45.088	59.99	2.204	2	3.667	8	1.039
ref1 with max 2 B-frames and chroma *cmp	25m40s	36.79	45.07	59.96	2.191	2	3.763	7	1.058

It seams to be best with max 1 consecutive B-frames. And using the chroma *cmp didn't help removing the colored stains.

Conclusion

The winner is:
mbd=2:v4mv:trell:cbp:mv0:last_pred=3:preme=2:precmp=3:cmp=2:subcmp=3:predia=3:dia=4:bidir_refine=2:vmax_b_frames=1:vb_strategy=1:vpass=1/2.

When rendering the full movie at 900 kbits, the short test sequence receives an higher average bitrate and the end result is near perfect on a CRT TV. Looking at the quantitizer histogram of the full movie, there is no problem setting vqmin=1 and vqmax=6. It is in those conditions that qprd and qns are able to make a positive difference. To obtain an encode that looks very good scaled at 200% on a LCD screen, I had to use up to 1100 kbits with qprd:qns=1:vqmin=1:vqmax=5. Also note that, if you have enough patience and enough bits, qns=3 will make most ringing disapear.

The next section will be about filtering. Encoding the original artifacts is probably not what we want and, regarding that point, home DV movies are very different than modern DVD productions. Testing on DVD sources will therefore end here, they have been useful as a high quality source that allowed me to defer filtering until now. But if you have already been burning your videos to DVD and didn't keep the raw DV sources, the next section will not help you get rid of mpeg2 artifacts. My suggestion is to at least filter out ringing and blocking before encoding, using a postprocessing filter: pp=hb/vb/dr. Or simply use pp=ac. Insert it after cropping, don't use it after the denoise filter.

In case you wondered, the complete command to encode the first pass is: mencoder stream.dump -oac copy -ovc lavc -lavcopts vbitrate=1100:mbd=2:v4mv:trell:cbp:mv0:last_pred=3:preme=2:precmp=3:cmp=2:subcmp=3:predia=3:dia=4:bidir_refine=2:vmax_b_frames=1:vb_strategy=1:qns=1:qprd:vqmin=1:vqmax=5:vpass=1 -vf crop=704:416:8:80,scale=-1:288,pp=hb/vb/dr,hqdn3d=0:0:4 -o /dev/null

And for the second pass: mencoder stream.dump -oac copy -ovc lavc -lavcopts vbitrate=1100:mbd=2:v4mv:trell:cbp:mv0:last_pred=3:preme=2:precmp=3:cmp=2:subcmp=3:predia=3:dia=4:bidir_refine=2:vmax_b_frames=1:qns=1:qprd:vqmin=1:vqmax=5:vpass=2:psnr:autoaspect -vf crop=704:416:8:80,scale=-1:288,pp=hb/vb/dr,hqdn3d=0:0:4 -o test.avi