View Issue Details

IDProjectCategoryView StatusLast Update
0000338Cinelerra-GG[All Projects] Bugpublic2019-11-08 02:56
ReporterAndrew-R Assigned ToPhyllisSmith  
PrioritynormalSeveritycrashReproducibilityalways
Status closedResolutionfixed 
Product Version2019-09 
Target VersionFixed in Version2019-10 
Summary0000338: mpeg encoder crashes in yuv422 mode
DescriptionIf you set render to mpeg video (not ffmpeg variant), and moreover select colorspace 422 - it will crash, at least form me

cin
Cinelerra Infinity - built: Nov 7 2019 04:11:42
git://git.cinelerra-gg.org/goodguy/cinelerra.git
(c) 2006-2019 Heroine Virtual Ltd. by Adam Williams
(c) 2007-2019 cin5 derivative by W.P. Morrow aka goodguy
Cinelerra is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. There is absolutely no warranty for Cinelerra.

Device type none is not supported.
Available device types: vdpau cuda vaapi opencl
Device type none is not supported.
Available device types: vdpau cuda vaapi opencl
Device type none is not supported.
Available device types: vdpau cuda vaapi opencl
FileMPEGVideo::run /usr/lib/cin//hveg2enc -a 1 -1 -422 -q 2 -n 45 -p /dev/shm/dog-xvid.m2v
Device type none is not supported.
Available device types: vdpau cuda vaapi opencl
Encoding: /dev/shm/dog-xvid.m2v frames 2147483647
   quantization 2
   45 frames between I frames 0 frames between P frames
   progressive
   MPEG-1
   YUV-422
   4 processors
   25.00 frames per second
   Denoise No
   Aspect ratio index 1
   Hires quantization No
SETTING EXTENDED MMX for MOTION!
SETTING EXTENDED MMX for PREDICTION!
SETTING EXTENDED MMX for QUANTIZER!
SETTING MMX for TRANSFORM!
** segv at 0xb8efc00 in pid 10049, tid 10135
writing debug data to /tmp/cinelerra_10049.dmp
lock_items: 22
lock_frees: 5
Ошибка сегментирования
TagsNo tags attached.

Activities

Andrew-R

Andrew-R

2019-11-08 02:46

reporter   ~0002439

compiled and tested - everything works, thanks, this one can be closed (I also only tried this encoder out of curiosity)
PhyllisSmith

PhyllisSmith

2019-11-08 01:40

manager   ~0002435

Thanks for patches! checked into GIT. Fixed "\n Done" too. Also, removed allowance to choose yuv422 for MPEG-1. But as far as "may be small input box "additional mpeg2enc flags" will fix this once and for all?" -- I do not know how relevant this code is any more, but gg does not want to work on it other than minor fixes since most people will just use ffmpeg anyway.
Andrew-R

Andrew-R

2019-11-07 23:09

reporter   ~0002433

from man (I think it applies to standalone mjpegtools's mpeg2enc, but applies to cin's use of same encoder, as we see from our experiments):

  --dualprime-mpeg2

       MPEG-2 supports a special motion estimation mode (DPME, Dual Prime Motion Estimation) for I/P-frame only streams that can somewhat improve compression. A number of players (both hardware and software) do not support this mode. Those players may or may not be MPEG-2 compliant depending if DPME is an option or not in the MPEG-2 specs. If you need to generate content for such players (e.g. Ogle or Apple's DVD player application) you should NOT turn on dualprime-mpeg2! Surprisingly at least one hardware/set-top player is known to be allergic to DPME being used.

so, probbaly not good as default flag.

may be small input box "additional mpeg2enc flags" will fix this once and for all?
PhyllisSmith

PhyllisSmith

2019-11-07 23:03

manager   ~0002432

Andrew: "--dualprime-mpeg2
Turn ON use of dual-prime motion compensation. Default is OFF unless this option is used"

Should we now turn this on too? and put in the patch? or not?

And this increase was added to the patch:
"And enlarged buffer size with -V flag up to 500 kb if generic mpeg2 used, allows bitrates up to 25Mbit/sec."
Andrew-R

Andrew-R

2019-11-07 22:49

reporter   ~0002431

also, purely cosmetic but "Done\n" string in putseq.c should also start from \n, for nicer output
Andrew-R

Andrew-R

2019-11-07 22:42

reporter   ~0002430

still andrew.patch2 miss -V buffer enlargement :}

But this is optional, I mean if you want > 10 000 Kbits/sec bitrate :}
Andrew-R

Andrew-R

2019-11-07 22:38

reporter   ~0002429

andrew.patch2 apparently works (I launched just compiled cin, not my system-installed copy)

root@slax:/dev/shm/tmp/cinelerra-goodguy-20191107/cinelerra-5.1# bin/cin
Cinelerra Infinity - built: Nov 7 2019 04:11:42
git://git.cinelerra-gg.org/goodguy/cinelerra.git
(c) 2006-2019 Heroine Virtual Ltd. by Adam Williams
(c) 2007-2019 cin5 derivative by W.P. Morrow aka goodguy
Cinelerra is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. There is absolutely no warranty for Cinelerra.

FileMPEGVideo::run /usr/lib/cin//hveg2enc -a 1 -422 -b 25000000 -n 2 -p -d -g /dev/shm/12.m2v
Encoding: /dev/shm/12.m2v frames 2147483647
   bitrate 25000000
   2 frames between I frames 0 frames between P frames
   progressive
   MPEG-2
   YUV-422
   4 processors
   25.00 frames per second
   Denoise Yes
   Aspect ratio index 1
   Hires quantization No
SETTING EXTENDED MMX for MOTION!
SETTING EXTENDED MMX for PREDICTION!
SETTING EXTENDED MMX for QUANTIZER!
SETTING MMX for TRANSFORM!
Done.ing frame 162. bitrate achieved: 26960560
Render::render_single: Session finished.
** rendered 161 frames in 14.206 secs, 11.333 fps

\o/
Andrew-R

Andrew-R

2019-11-07 22:31

reporter   ~0002428

root@slax:/dev/shm/tmp/cinelerra-goodguy-20191107/cinelerra-5.1/bin# gdb cin
GNU gdb (GDB) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "i586-slackware-linux".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from cin...Reading symbols from /dev/shm/tmp/cinelerra-goodguy-20191107/cinelerra-5.1/bin/cin.debuginfo...done.
done.
(gdb) run
Starting program: /dev/shm/tmp/cinelerra-goodguy-20191107/cinelerra-5.1/bin/cin
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/libthread_db.so.1".
Cinelerra Infinity - built: Nov 7 2019 04:11:42
git://git.cinelerra-gg.org/goodguy/cinelerra.git
(c) 2006-2019 Heroine Virtual Ltd. by Adam Williams
(c) 2007-2019 cin5 derivative by W.P. Morrow aka goodguy
Cinelerra is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. There is absolutely no warranty for Cinelerra.

[New Thread 0xf145fb40 (LWP 27722)]
[New Thread 0xf0c5fb40 (LWP 27723)]
[New Thread 0xf045fb40 (LWP 27724)]
[New Thread 0xefc5fb40 (LWP 27725)]
[New Thread 0xef332b40 (LWP 27727)]
[New Thread 0xee1c6b40 (LWP 27730)]
[New Thread 0xed9c6b40 (LWP 27731)]
[New Thread 0xed1c6b40 (LWP 27732)]
[New Thread 0xec9c6b40 (LWP 27733)]
[New Thread 0xec1c6b40 (LWP 27734)]
[New Thread 0xeb9c6b40 (LWP 27735)]
[New Thread 0xeafffb40 (LWP 27736)]
[New Thread 0xea5ffb40 (LWP 27737)]
[New Thread 0xe9dffb40 (LWP 27738)]
[New Thread 0xe95ffb40 (LWP 27739)]
[New Thread 0xe8dffb40 (LWP 27740)]
[New Thread 0xe85ffb40 (LWP 27741)]
[New Thread 0xe7dffb40 (LWP 27742)]
[New Thread 0xe75ffb40 (LWP 27743)]
[New Thread 0xe6dffb40 (LWP 27744)]
[New Thread 0xe65ffb40 (LWP 27745)]
[New Thread 0xe5dffb40 (LWP 27746)]
[New Thread 0xe55ffb40 (LWP 27747)]
[New Thread 0xe4bffb40 (LWP 27748)]
[New Thread 0xe43ffb40 (LWP 27749)]
[New Thread 0xe3bffb40 (LWP 27750)]
[New Thread 0xe33ffb40 (LWP 27751)]
[New Thread 0xe2bffb40 (LWP 27752)]
[New Thread 0xe23ffb40 (LWP 27753)]
[New Thread 0xe1bffb40 (LWP 27754)]
[New Thread 0xe13ffb40 (LWP 27755)]
[New Thread 0xe0bffb40 (LWP 27756)]
[New Thread 0xe03ffb40 (LWP 27757)]
[Thread 0xef332b40 (LWP 27727) exited]
[New Thread 0xef332b40 (LWP 27758)]
[New Thread 0xdf9ffb40 (LWP 27759)]
[New Thread 0xde9ffb40 (LWP 27760)]
[New Thread 0xddfffb40 (LWP 27761)]
[New Thread 0xdf1ffb40 (LWP 27762)]
[New Thread 0xdcfffb40 (LWP 27764)]
[New Thread 0xdd7ffb40 (LWP 27763)]
[New Thread 0xdc7ffb40 (LWP 27766)]
[New Thread 0xdbfffb40 (LWP 27765)]
[New Thread 0xdb7ffb40 (LWP 27767)]
[New Thread 0xda7ffb40 (LWP 27769)]
[New Thread 0xdafffb40 (LWP 27768)]
[New Thread 0xd9fffb40 (LWP 27770)]
[New Thread 0xd97ffb40 (LWP 27771)]
[New Thread 0xd8fffb40 (LWP 27772)]
[New Thread 0xd8210b40 (LWP 27776)]
[New Thread 0xd7a10b40 (LWP 27777)]
[New Thread 0xd7210b40 (LWP 27778)]
[New Thread 0xd5e01b40 (LWP 27779)]
[New Thread 0xd5601b40 (LWP 27780)]
[New Thread 0xd43ffb40 (LWP 27781)]
[New Thread 0xd3bffb40 (LWP 27782)]
[New Thread 0xd33ffb40 (LWP 27783)]
[Thread 0xd5601b40 (LWP 27780) exited]
[New Thread 0xd5601b40 (LWP 27784)]
[Thread 0xd5e01b40 (LWP 27779) exited]
[New Thread 0xd5e01b40 (LWP 27785)]
[New Thread 0xd0f20b40 (LWP 27786)]
[Thread 0xd0f20b40 (LWP 27786) exited]
[Thread 0xd5e01b40 (LWP 27785) exited]
[Thread 0xd3bffb40 (LWP 27782) exited]
[Thread 0xd43ffb40 (LWP 27781) exited]
[Thread 0xd33ffb40 (LWP 27783) exited]
[New Thread 0xd33ffb40 (LWP 27787)]
[New Thread 0xd3bffb40 (LWP 27788)]
[New Thread 0xd43ffb40 (LWP 27789)]
[New Thread 0xd5e01b40 (LWP 27790)]
[New Thread 0xd0f20b40 (LWP 27791)]
[Thread 0xd3bffb40 (LWP 27788) exited]
[Thread 0xd33ffb40 (LWP 27787) exited]
[New Thread 0xd33ffb40 (LWP 27792)]
[New Thread 0xd3bffb40 (LWP 27793)]
[Thread 0xd3bffb40 (LWP 27793) exited]
[Thread 0xd33ffb40 (LWP 27792) exited]
[New Thread 0xd33ffb40 (LWP 27795)]
[New Thread 0xd3bffb40 (LWP 27796)]
[Thread 0xd3bffb40 (LWP 27796) exited]
[Thread 0xd33ffb40 (LWP 27795) exited]
[New Thread 0xd33ffb40 (LWP 27797)]
[New Thread 0xd3bffb40 (LWP 27798)]
[Thread 0xd3bffb40 (LWP 27798) exited]
[Thread 0xd33ffb40 (LWP 27797) exited]
[New Thread 0xd3bffb40 (LWP 27799)]
[New Thread 0xce189b40 (LWP 27800)]
[New Thread 0xcd989b40 (LWP 27801)]
[New Thread 0xcd189b40 (LWP 27802)]
[New Thread 0xcc989b40 (LWP 27803)]
[New Thread 0xcc189b40 (LWP 27804)]
[New Thread 0xcb989b40 (LWP 27805)]
[New Thread 0xcb189b40 (LWP 27806)]
[Thread 0xcb989b40 (LWP 27805) exited]
[Thread 0xcb189b40 (LWP 27806) exited]
[Thread 0xcc189b40 (LWP 27804) exited]
[Thread 0xcc989b40 (LWP 27803) exited]
[Thread 0xcd989b40 (LWP 27801) exited]
[Thread 0xcd189b40 (LWP 27802) exited]
[Thread 0xce189b40 (LWP 27800) exited]
[New Thread 0xce189b40 (LWP 27808)]
[Thread 0xd3bffb40 (LWP 27799) exited]
[New Thread 0xd3bffb40 (LWP 27809)]
[New Thread 0xcc989b40 (LWP 27810)]
[New Thread 0xcd189b40 (LWP 27811)]
[Thread 0xcc989b40 (LWP 27810) exited]
[Thread 0xcd189b40 (LWP 27811) exited]
[Thread 0xd3bffb40 (LWP 27809) exited]
[New Thread 0xd3bffb40 (LWP 27812)]
[New Thread 0xcd189b40 (LWP 27813)]
FileMPEGVideo::run /usr/lib/cin//hveg2enc -a 1 -422 -b 25000000 -n 2 -p -d -g /dev/shm/12.m2v
[New Thread 0xcc989b40 (LWP 27814)]
[Thread 0xcc989b40 (LWP 27814) exited]
[New Thread 0xcc989b40 (LWP 27815)]
Encoding: /dev/shm/12.m2v frames 2147483647
   bitrate 25000000
   2 frames between I frames 0 frames between P frames
   progressive
   MPEG-2
   YUV-422
   4 processors
   25.00 frames per second
   Denoise Yes
   Aspect ratio index 1
   Hires quantization No
SETTING EXTENDED MMX for MOTION!
SETTING EXTENDED MMX for PREDICTION!
SETTING EXTENDED MMX for QUANTIZER!
SETTING MMX for TRANSFORM!
[New Thread 0xcd989b40 (LWP 27816)]
[New Thread 0xcc189b40 (LWP 27817)]
[New Thread 0xc7d61b40 (LWP 27818)]
[New Thread 0xc3cffb40 (LWP 27819)]
[New Thread 0xc34ffb40 (LWP 27820)]
[New Thread 0xc2cffb40 (LWP 27821)]
[New Thread 0xc24ffb40 (LWP 27822)]
[New Thread 0xc1cffb40 (LWP 27823)]
[New Thread 0xc14ffb40 (LWP 27824)]
[New Thread 0xc0cffb40 (LWP 27825)]
[New Thread 0xc04ffb40 (LWP 27826)]
[New Thread 0xbfcffb40 (LWP 27827)]
[New Thread 0xbf4ffb40 (LWP 27828)]
[New Thread 0xbecffb40 (LWP 27829)]
[New Thread 0xbe4ffb40 (LWP 27830)]
[New Thread 0xbdcffb40 (LWP 27831)]
[New Thread 0xbd4ffb40 (LWP 27832)]
[New Thread 0xbccffb40 (LWP 27833)]
[New Thread 0xbc4ffb40 (LWP 27834)]
Encoding frame 2. bitrate achieved: 138718600
Thread 91 "cin" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0xc3cffb40 (LWP 27819)]
0x0b8efc00 in dist2_mmx ()
(gdb) bt full
#0 0x0b8efc00 in dist2_mmx ()
No symbol table info available.
0000001 0x08a6a996 in fullsearch ()
No symbol table info available.
0000002 0x08a7466e in frame_ME.isra.4 ()
No symbol table info available.
0000003 0x08a74b50 in motion_engine_loop ()
No symbol table info available.
#4 0xf768e6f2 in start_thread () from /lib/libpthread.so.0
No symbol table info available.
0000005 0xf706a1ee in clone () from /lib/libc.so.6
No symbol table info available.
(gdb)


this was from gdb session .... strangely, dmp file often empty (even if run as root). will try to rebuild both cin and mpeg2enc dirs again (because despite printed patch to executable - cin apparently uses functions from .a file ?)
PhyllisSmith

PhyllisSmith

2019-11-07 22:26

manager   ~0002427

Just tested this combined patch on Debian 9i 32-bit. Tested several variations with no problems/no crashes. For sure tested MPEG-2 yuv420 and yuv422; MPEG-1 yuv420 and MPEG-1 yuv422 did not crash but is all green screen. Here is gg's patch with your patches in that "should" be all that is needed to make it work. You might want to start out clean to make sure the build is clean.

andrew.patch2 (5,973 bytes)
Andrew-R

Andrew-R

2019-11-07 22:23

reporter   ~0002426

hm, strange, my hack only works when I set I-frames interval to 1 :/ Obviously, this disables a lot of functions, and disabling nearly all asm doesn't fix this ..
PhyllisSmith

PhyllisSmith

2019-11-07 21:56

manager   ~0002425

"sadly andrew.patch1 still not resolves crash here :( (i applied on top of my patch and then commented out my newly added conditional in quantize.c)"

The last dump was more complete, but did not contain any gdb trace data (at the bottom). This is the most informative part of the dump capture. If possible, try to make it available and it will aid in the discovery considerably. The last dump did contain enough information to determine it is definitely the NX bug. The segv address (top lines) is in a .data section, and not a .text section. That is the NX stack security capability in the cpu flags (cpuinfo).
PhyllisSmith

PhyllisSmith

2019-11-07 21:48

manager   ~0002424

OK, it would be a lot easier if we both got the same types of crashes!! GG is looking at your latest dump now. Attached is an example of a complete dump while running nothing inside of CinGG. It is about 3,500 lines. If you look at the bottom, this is the good stuff to aid in analyzing:
        ...
        locale_path = 0x514aacb "/mnt0/build5/cinelerra-5.1/bin/locale"
        cin = 0x514bc88 "cin"
        loc = 0x5160e10 "en_US.UTF-8"
        load_backup = 0
        start_remote_control = 0
        scale = 0
        et = 139877204613808
        dt = 140724092563616
        ru = {ru_utime = {tv_sec = 0, tv_usec = 0}, ru_stime = {tv_sec = 0, tv_usec = 0}, {ru_maxrss = 0, __ru_maxrss_word = 0}, {ru_ixrss = 0, __ru_ixrss_word = 0}, {ru_idrss = 0, __ru_idrss_word = 0}, {ru_isrss = 139877218468031, __ru_isrss_word = 139877218468031}, {ru_minflt = 68719476740, __ru_minflt_word = 68719476740}, {ru_majflt = 139877218468031, __ru_majflt_word = 139877218468031}, {ru_nswap = 68719476740, __ru_nswap_word = 68719476740}, {ru_inblock = 24011439870050309, __ru_inblock_word = 24011439870050309}, {ru_oublock = 20401094658, __ru_oublock_word = 20401094658}, {ru_msgsnd = 3, __ru_msgsnd_word = 3}, {ru_msgrcv = 20401094657, __ru_msgrcv_word = 20401094657}, {ru_nsignals = 3, __ru_nsignals_word = 3}, {ru_nvcsw = 20401094656, __ru_nvcsw_word = 20401094656}, {ru_nivcsw = 139874199928835, __ru_nivcsw_word = 139874199928835}}
        usr_ms = 139877207072956
        us = 7658528
        ums = 32567
        sys_ms = 72117408
        ss = 85311600
        sms = 32567
A debugging session is active.

There is a section in the manual about getting dumps.

cinelerra_8052.dmp (211,738 bytes)
Andrew-R

Andrew-R

2019-11-07 21:39

reporter   ~0002423

Got more complete crash

cinelerra_23754.dmp (55,821 bytes)
Andrew-R

Andrew-R

2019-11-07 21:34

reporter   ~0002422

sadly andrew.patch1 still not resolves crash here :( (i applied on top of my patch and then commented out my newly added conditional in quantize.c)
PhyllisSmith

PhyllisSmith

2019-11-07 21:27

manager   ~0002421

Your patch 3 looks good for -H flag and -no-constraints so that will go in too.
Andrew-R

Andrew-R

2019-11-07 21:25

reporter   ~0002420

And enlarged buffer size with -V flag up to 500 kb if generic mpeg2 used, allows bitrates up to 25Mbit/sec.
man mpeg2enc used, where it says

-----
-V|--video-buffer num

       The maximum video buffer usage required to decode the stream in KBytes. The default is 46KB the (tiny) size specified for VCD. The size to use for SVCD is the (more reasonable) 230KB. If you are encoding for a half-decent software decoder it makes sense to push this up to 500K or more.

------

default was barely enough for 10Mbit/s, setting it higher resulted in

FileMPEG::open_file: Running /usr/lib/cin//mpeg2enc -v 0 -b 14000 -q 1 -a 1 -F 3 -H --no-constraints -I 0 -M 4 -f 3 -g 8 -G 8 -s -R 2 -o '/dev/shm/11.m2v'
**ERROR: [mpeg2enc] Rate control can't cope with a video buffer smaller 4 frame intervals

fileilempeg_patch_added_h_and_no-const_bigbuf__to_mpeg2_generic.diff (896 bytes)
diff --git a/cinelerra-5.1/cinelerra/filempeg.C b/cinelerra-5.1/cinelerra/filempeg.C
index 986ac92..62de887 100644
--- a/cinelerra-5.1/cinelerra/filempeg.C
+++ b/cinelerra-5.1/cinelerra/filempeg.C
@@ -622,10 +622,18 @@ int FileMPEG::open_file(int rd, int wr)
 			sprintf(string, " -F %d", frame_rate_code);
 			strncat(mjpeg_command, string, sizeof(mjpeg_command));
 
+			sprintf(string, " -H"); /* Maximise high-frequency resolution */
+			strncat(mjpeg_command, string, sizeof(mjpeg_command));
 
-
-
-
+			if(asset->vmpeg_preset == 3) /* no constrains for mpeg2 generic */
+			{
+			sprintf(string, " --no-constraints");
+			strncat(mjpeg_command, string, sizeof(mjpeg_command));
+			sprintf(string, " -V 500");
+			strncat(mjpeg_command, string, sizeof(mjpeg_command));
+			}
+			
+			
 			strncat(mjpeg_command,
 				asset->vmpeg_progressive ? " -I 0" : " -I 1",
 				sizeof(mjpeg_command));
PhyllisSmith

PhyllisSmith

2019-11-07 21:25

manager   ~0002419

Andrew, here is an NX patch that gg says fixes the 1 crash that he got. You apparently have NX on your computer as seen in the dump. He believes that your patch 1 and patch 2 work to get around the NX problem. Could you try this when you get a chance?

andrew.patch1 (2,934 bytes)
Andrew-R

Andrew-R

2019-11-07 20:56

reporter   ~0002418

Added patch to filempeg, it _unconditionally_ adds -H flag to mpeg2enc, and also adds --no-constraints for mpeg2 generic profile (so, for example 25 fps progressive 1920x1080 doesn't fail)

filempeg_patch_added_h_and_no-const_to_mpeg2_generic.diff (805 bytes)
diff --git a/cinelerra-5.1/cinelerra/filempeg.C b/cinelerra-5.1/cinelerra/filempeg.C
index 986ac92..504825a 100644
--- a/cinelerra-5.1/cinelerra/filempeg.C
+++ b/cinelerra-5.1/cinelerra/filempeg.C
@@ -622,10 +622,16 @@ int FileMPEG::open_file(int rd, int wr)
 			sprintf(string, " -F %d", frame_rate_code);
 			strncat(mjpeg_command, string, sizeof(mjpeg_command));
 
+			sprintf(string, " -H"); /* Maximise high-frequency resolution */
+			strncat(mjpeg_command, string, sizeof(mjpeg_command));
 
-
-
-
+			if(asset->vmpeg_preset == 3) /* no constrains for mpeg2 generic */
+			{
+			sprintf(string, " --no-constraints");
+			strncat(mjpeg_command, string, sizeof(mjpeg_command));
+			}
+			
+			
 			strncat(mjpeg_command,
 				asset->vmpeg_progressive ? " -I 0" : " -I 1",
 				sizeof(mjpeg_command));
PhyllisSmith

PhyllisSmith

2019-11-07 20:31

manager   ~0002417

Andrew: "basically it disables mmxext support in quantize.c (I hope)" -- appears to be true per gg. He is now testing on a 32-bit system and it did not crash with MPEG-1/yuv422 but all he got was a green screen. However, he DID get a crash with MPEG-2/yuv422 and even with the quanize_c_no_see.patch. He thinks that the SEGV is a result of NX / executing code not in code area but in data area.

Will download and send him your second patch.
Andrew-R

Andrew-R

2019-11-07 20:16

reporter   ~0002416

Like this ?

no_sse_for_422_fix.patch (1,922 bytes)
diff --git a/cinelerra-5.1/mpeg2enc/mpeg2enc.c b/cinelerra-5.1/mpeg2enc/mpeg2enc.c
index 5452fb4..a2e2215 100644
--- a/cinelerra-5.1/mpeg2enc/mpeg2enc.c
+++ b/cinelerra-5.1/mpeg2enc/mpeg2enc.c
@@ -222,8 +222,14 @@ static void init()
 	init_idct();
 	init_motion();
 	init_predict_hv();
-	init_quantizer_hv();
-	init_transform_hv();
+	
+	if (chroma_format==CHROMA420)
+	init_quantizer_hv(1);
+	else
+	if (chroma_format==CHROMA422)
+	init_quantizer_hv(0);
+	
+	init_transform_hv(0);
 
 /* round picture dimensions to nZearest multiple of 16 or 32 */
 	mb_width = (horizontal_size+15)/16;
diff --git a/cinelerra-5.1/mpeg2enc/quantize.c b/cinelerra-5.1/mpeg2enc/quantize.c
index f842843..83762b9 100644
--- a/cinelerra-5.1/mpeg2enc/quantize.c
+++ b/cinelerra-5.1/mpeg2enc/quantize.c
@@ -57,8 +57,9 @@ static void iquant_non_intra_m1(int16_t *src, int16_t *dst, uint16_t *quant_mat)
   Currently just setting up MMX routines if available...
  */
 
-void init_quantizer_hv()
+void init_quantizer_hv(int use_sse)
 {
+
 #ifdef X86_CPU
   int flags;
   flags = cpu_accel();
@@ -73,10 +74,10 @@ void init_quantizer_hv()
 /*
  * 		else if ( (flags & ACCEL_X86_MMXEXT) != 0 )
  * 		{
- * 			if(verbose) fprintf( stderr, "SSE and ");
+ *  			if(verbose) fprintf( stderr, "SSE and ");
  * 			pquant_non_intra = quant_non_intra_hv_sse;
- * 		}
- */
+ *		}
+*/
 		else 
 		{
 			pquant_non_intra = quant_non_intra_hv;
@@ -86,9 +87,11 @@ void init_quantizer_hv()
 		{
 			if(verbose) fprintf( stderr, "EXTENDED MMX");
 			pquant_weight_coeff_sum = quant_weight_coeff_sum_mmx;
+			if (use_sse == 1)
 			piquant_non_intra_m1 = iquant_non_intra_m1_sse;
 		}
 		else
+
 		{
 			if(verbose) fprintf( stderr, "MMX");
 			pquant_weight_coeff_sum = quant_weight_coeff_sum_mmx;
@@ -98,6 +101,7 @@ void init_quantizer_hv()
 	}
   else
 #endif
+
 	{
 	  pquant_non_intra = quant_non_intra_hv;	  
 	  pquant_weight_coeff_sum = quant_weight_coeff_sum;
no_sse_for_422_fix.patch (1,922 bytes)
Andrew-R

Andrew-R

2019-11-07 19:56

reporter   ~0002414

In general, it seems cinelerra-5.1/cinelerra/filempeg.C can be altered to use slightly more options from mpeg2enc
Namely,

--keep-hf|-H
    Maximise high-frequency resolution - useful for high quality sources
    and/or high bit-rates)

and

--no-constraints
    Deactivate constraints for maximum video resolution and sample rate.
    Could expose bugs in the software at very high resolutions!

and

--dualprime-mpeg2
    Turn ON use of dual-prime motion compensation. Default is OFF unless this option is used

But for this specific bug ...probably cinelerra-5.1/mpeg2enc/mpeg2enc.c should call init_quantizer_hv() with added parameter preventing sse usage if yuv422 colorspace is used ...
Andrew-R

Andrew-R

2019-11-07 19:17

reporter   ~0002413

from quant_mmx.s

;;; void iquant_non_intra_m1_{sse,mmx}(int16_t *src, int16_t *dst, uint16_t
;;; *quant_mat)
;;; mmx/sse Inverse mpeg-1 quantisation routine.
;;;.
;;; eax - block counter...
;;; edi - src
;;; esi - dst
;;; edx - quant_mat

so, this explains? I mean, this route shouldn't be called for yuv422 (mpeg2) case? But probably disabling it altogether will be simplest solution ?
Andrew-R

Andrew-R

2019-11-07 18:50

reporter   ~0002412

piquant_non_intra_m1 = iquant_non_intra_m1_sse; seems to be at fault (by looking at difference between what works and what not)

quanize_c_no_see.patch (974 bytes)
diff --git a/cinelerra-5.1/mpeg2enc/quantize.c b/cinelerra-5.1/mpeg2enc/quantize.c
index f842843..00993ea 100644
--- a/cinelerra-5.1/mpeg2enc/quantize.c
+++ b/cinelerra-5.1/mpeg2enc/quantize.c
@@ -59,6 +59,7 @@ static void iquant_non_intra_m1(int16_t *src, int16_t *dst, uint16_t *quant_mat)
 
 void init_quantizer_hv()
 {
+#if 1
 #ifdef X86_CPU
   int flags;
   flags = cpu_accel();
@@ -81,7 +82,7 @@ void init_quantizer_hv()
 		{
 			pquant_non_intra = quant_non_intra_hv;
 		}
-
+/*
 		if ( (flags & ACCEL_X86_MMXEXT) != 0 )
 		{
 			if(verbose) fprintf( stderr, "EXTENDED MMX");
@@ -89,6 +90,7 @@ void init_quantizer_hv()
 			piquant_non_intra_m1 = iquant_non_intra_m1_sse;
 		}
 		else
+*/
 		{
 			if(verbose) fprintf( stderr, "MMX");
 			pquant_weight_coeff_sum = quant_weight_coeff_sum_mmx;
@@ -97,6 +99,7 @@ void init_quantizer_hv()
 		if(verbose) fprintf( stderr, " for QUANTIZER!\n");
 	}
   else
+#endif
 #endif
 	{
 	  pquant_non_intra = quant_non_intra_hv;	  
Andrew-R

Andrew-R

2019-11-07 18:45

reporter   ~0002411

It works for mpeg1 (while I was forced to set project fps to 30, 10 was not enough):

FFMPEG::open_decoder: some stream times estimated: /home/guest/0005.avi
FFMPEG::open_decoder: some stream times estimated: /home/guest/0005.avi
FFMPEG::open_decoder: some stream times estimated: /home/guest/0005.avi
FFMPEG::open_decoder: some stream times estimated: /home/guest/0005.avi
FFMPEG::open_decoder: some stream times estimated: /home/guest/0005.avi
FFMPEG::open_decoder: some stream times estimated: /home/guest/0005.avi
FFMPEG::open_decoder: some stream times estimated: /home/guest/0005.avi
virtual int FileMPEG::open_file(int, int):
Неподдерживаемая частота кадров 9.615300
FileMPEG::open_file: Running /usr/lib/cin//mpeg2enc -v 0 -b 0 -q 2 -a 1 -F 4 -I 0 -M 4 -f 0 -g 1 -G 1 -s -R 2 -o '/dev/shm/4.m2v'
FFMPEG::open_decoder: some stream times estimated: /home/guest/0005.avi
Render::render_single: Session finished.
** rendered 271 frames in 1.901 secs, 142.557 fps
FFMPEG::open_decoder: some stream times estimated: /home/guest/0005.avi
virtual int FileMPEG::open_file(int, int):
Неподдерживаемая частота кадров 10.000000
FileMPEG::open_file: Running /usr/lib/cin//mpeg2enc -v 0 -b 0 -q 2 -a 1 -F 4 -I 0 -M 4 -f 0 -g 1 -G 1 -s -R 2 -o '/dev/shm/4.m2v'
FFMPEG::open_decoder: some stream times estimated: /home/guest/0005.avi
Render::render_single: Session finished.
** rendered 282 frames in 3.861 secs, 73.038 fps
FFMPEG::open_decoder: some stream times estimated: /home/guest/0005.avi
FFMPEG::open_decoder: some stream times estimated: /home/guest/0005.avi
FileMPEG::open_file: Running /usr/lib/cin//mpeg2enc -v 0 -b 0 -q 2 -a 1 -F 5 -I 0 -M 4 -f 0 -g 1 -G 1 -s -R 2 -o '/dev/shm/4.m2v'
FFMPEG::open_decoder: some stream times estimated: /home/guest/0005.avi
Render::render_single: Session finished.
** rendered 848 frames in 6.639 secs, 127.730 fps
Session time: 0:16:12
Cpu time: user: 0:11:09.148 sys: 0:00:17.427

for 320x240 (:) ) video ....
Andrew-R

Andrew-R

2019-11-07 18:35

reporter   ~0002410

this patch makes it work for me:

--- a/cinelerra-5.1/mpeg2enc/quantize.c
+++ b/cinelerra-5.1/mpeg2enc/quantize.c
@@ -59,6 +59,7 @@ static void iquant_non_intra_m1(int16_t *src, int16_t *dst, uint16_t *quant_mat)

 void init_quantizer_hv()
 {
+#if 1
 #ifdef X86_CPU
   int flags;
   flags = cpu_accel();
@@ -81,7 +82,7 @@ void init_quantizer_hv()
                {
                        pquant_non_intra = quant_non_intra_hv;
                }
-
+/*
                if ( (flags & ACCEL_X86_MMXEXT) != 0 )
                {
                        if(verbose) fprintf( stderr, "EXTENDED MMX");
@@ -89,6 +90,7 @@ void init_quantizer_hv()
                        piquant_non_intra_m1 = iquant_non_intra_m1_sse;
                }
                else
+*/
                {
                        if(verbose) fprintf( stderr, "MMX");
                        pquant_weight_coeff_sum = quant_weight_coeff_sum_mmx;
@@ -97,6 +99,7 @@ void init_quantizer_hv()
                if(verbose) fprintf( stderr, " for QUANTIZER!\n");
        }
   else
+#endif
 #endif
        {
          pquant_non_intra = quant_non_intra_hv;

basically it disables mmxext support in quantize.c (I hope)

after such hack it works for me:
 cin
Cinelerra Infinity - built: Nov 7 2019 04:11:42
git://git.cinelerra-gg.org/goodguy/cinelerra.git
(c) 2006-2019 Heroine Virtual Ltd. by Adam Williams
(c) 2007-2019 cin5 derivative by W.P. Morrow aka goodguy
Cinelerra is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. There is absolutely no warranty for Cinelerra.

FileMPEGVideo::run /usr/lib/cin//hveg2enc -a 1 -422 -q 2 -n 1 -p -d -g /dev/shm/h264--4.m2v
Encoding: /dev/shm/h264--4.m2v frames 2147483647
   quantization 2
   1 frames between I frames 0 frames between P frames
   progressive
   MPEG-2
   YUV-422
   4 processors
   25.00 frames per second
   Denoise Yes
   Aspect ratio index 1
   Hires quantization No
SETTING EXTENDED MMX for MOTION!
SETTING EXTENDED MMX for PREDICTION!
SETTING EXTENDED MMX for QUANTIZER!
SETTING MMX for TRANSFORM!
Encoding frame 3178. bitrate achieved: 114544128
Done. Be sure to visit heroinewarrior.com for updates.
Render::render_single: Session finished.
** rendered 3177 frames in 271.621 secs, 11.696 fps

mplayer h264--4.m2v
MPlayer SVN-r38152-5.5.0 (C) 2000-2019 MPlayer Team
224 audio & 466 video codecs

Playing h264--4.m2v.
libavformat version 58.33.100 (internal)
libavformat file format detected.
[mpeg1video @ 0x5769a7e0]interlaced frame in progressive sequence, ignoring
[mpegvideo @ 0x575429e0]Estimating duration from bitrate, this may be inaccurate
[lavf] stream 0: video (mpeg2video), -vid 0
VIDEO: [MPG2] 1920x1080 0bpp 25.000 fps 5000.0 kbps (610.4 kbyte/s)
==========================================================================
Opening video decoder: [ffmpeg] FFmpeg's libavcodec codec family
libavcodec version 58.59.102 (internal)
Selected video codec: [ffmpeg2] vfm: ffmpeg (FFmpeg MPEG-2)
==========================================================================
Load subtitles in ./
Audio: no sound
Starting playback...
[mpeg2video @ 0x5769a7e0]interlaced frame in progressive sequence, ignoring
Could not find matching colorspace - retrying with -vf scale...
Opening video filter: [scale]
Movie-Aspect is 1.78:1 - prescaling to correct movie aspect.
[swscaler @ 0x577d6140]bicubic scaler, from yuv422p to yuyv422 using MMXEXT
[swscaler @ 0x577d6140]using unscaled yuv422p -> yuyv422 special converter
VO: [vdpau] 1920x1080 => 1920x1080 Packed YUY2
[mpeg2video @ 0x5769a7e0]interlaced frame in progressive sequence, ignoring
Movie-Aspect is 1.78:1 - prescaling to correct movie aspect.
VO: [vdpau] 1920x1080 => 1920x1080 Packed YUY2
V: 0.0 0/ 0 ??% ??% ??,?% 0 0
[mpeg2video @ 0x5769a7e0]interlaced frame in progressive sequence, ignoring
V: 0.1 0/ 0 ??% ??% ??,?% 0 0
[mpeg2video @ 0x5769a7e0]interlaced frame in progressive sequence, ignoring
V: 0.1 0/ 0 ??% ??% ??,?% 0 0
[mpeg2video @ 0x5769a7e0]interlaced frame in progressive sequence, ignoring

ignore strange filename, I just used it for testing .....
PhyllisSmith

PhyllisSmith

2019-11-07 18:02

manager   ~0002409

It does not crash on a 64-bit Fedora system so GG is now looking at how to find a way to create a crash by using code that a 32-bit system would. However, we did find a problem using MPEG-1 with yuv422 ghosting colors and it thinks the output is yuv420. Now I find at:

   www.differencebetween.net/technology/difference-between-mpeg1-and-mpeg2/

QUOTE:
"Then again, the older MPEG1 has some weaknesses that were addressed by its successor, the MPEG2. These said weaknesses are:
-The audio compression is limited to two channels.
– There is no standardized support for interlaced video with poor compression when used for interlaced video
– It has a limited standardized profile — Constrained Parameters Bitstream — which was incompatible for video with higher resolutions.

MPEG1 might support 4k video but there was no practical way to encode video for higher resolutions. Identification of hardware capable of support is also limited.
– IT SUPPORTS ONLY ONE COLORSPACE — 4:2:0."

: ENDQUOTE

This renders slowly so, of course, using ffmpeg is much better. GG is going to remove the yuv422 option availability if choose MPEG-1. GG says that he has not looked at this code for 100 years -- pretty sure that is a bit of an exaggeration. I tested MPEG-1 with yuv420 and had no problems BUT if you have time (which I do not see how you could possibly do as much work as you do!) could you verify that that works on your 32-bit system? I also tested MPEG-2 with yuv422 and it worked just fine. I tried testing using yuv420 with Generic Mpeg-2 but get errors and since I do not know what the correct parameter values should be, I abandoned this effort.

More later when GG finishes analysis. Also, it would really, really help to get a complete dump. I will send an example.
Andrew-R

Andrew-R

2019-11-07 10:36

reporter   ~0002405

slightly better (but still incomplete) dump from root account

cinelerra_10256.dmp (9,425 bytes)
Andrew-R

Andrew-R

2019-11-07 10:31

reporter  

cinelerra_10049.dmp (4,096 bytes)

Issue History

Date Modified Username Field Change
2019-11-07 10:31 Andrew-R New Issue
2019-11-07 10:31 Andrew-R File Added: cinelerra_10049.dmp
2019-11-07 10:36 Andrew-R File Added: cinelerra_10256.dmp
2019-11-07 10:36 Andrew-R Note Added: 0002405
2019-11-07 18:02 PhyllisSmith Assigned To => PhyllisSmith
2019-11-07 18:02 PhyllisSmith Status new => acknowledged
2019-11-07 18:02 PhyllisSmith Note Added: 0002409
2019-11-07 18:35 Andrew-R Note Added: 0002410
2019-11-07 18:45 Andrew-R Note Added: 0002411
2019-11-07 18:50 Andrew-R File Added: quanize_c_no_see.patch
2019-11-07 18:50 Andrew-R Note Added: 0002412
2019-11-07 19:17 Andrew-R Note Added: 0002413
2019-11-07 19:56 Andrew-R Note Added: 0002414
2019-11-07 20:16 Andrew-R File Added: no_sse_for_422_fix.patch
2019-11-07 20:16 Andrew-R Note Added: 0002416
2019-11-07 20:31 PhyllisSmith Note Added: 0002417
2019-11-07 20:56 Andrew-R File Added: filempeg_patch_added_h_and_no-const_to_mpeg2_generic.diff
2019-11-07 20:56 Andrew-R Note Added: 0002418
2019-11-07 21:25 PhyllisSmith File Added: andrew.patch1
2019-11-07 21:25 PhyllisSmith Note Added: 0002419
2019-11-07 21:25 Andrew-R File Added: fileilempeg_patch_added_h_and_no-const_bigbuf__to_mpeg2_generic.diff
2019-11-07 21:25 Andrew-R Note Added: 0002420
2019-11-07 21:27 PhyllisSmith Note Added: 0002421
2019-11-07 21:34 Andrew-R Note Added: 0002422
2019-11-07 21:39 Andrew-R File Added: cinelerra_23754.dmp
2019-11-07 21:39 Andrew-R Note Added: 0002423
2019-11-07 21:48 PhyllisSmith File Added: cinelerra_8052.dmp
2019-11-07 21:48 PhyllisSmith Note Added: 0002424
2019-11-07 21:56 PhyllisSmith Note Added: 0002425
2019-11-07 22:23 Andrew-R Note Added: 0002426
2019-11-07 22:26 PhyllisSmith File Added: andrew.patch2
2019-11-07 22:26 PhyllisSmith Note Added: 0002427
2019-11-07 22:31 Andrew-R Note Added: 0002428
2019-11-07 22:38 Andrew-R Note Added: 0002429
2019-11-07 22:42 Andrew-R Note Added: 0002430
2019-11-07 22:49 Andrew-R Note Added: 0002431
2019-11-07 23:03 PhyllisSmith Note Added: 0002432
2019-11-07 23:09 Andrew-R Note Added: 0002433
2019-11-08 01:40 PhyllisSmith Note Added: 0002435
2019-11-08 02:46 Andrew-R Note Added: 0002439
2019-11-08 02:56 PhyllisSmith Status acknowledged => closed
2019-11-08 02:56 PhyllisSmith Resolution open => fixed
2019-11-08 02:56 PhyllisSmith Fixed in Version => 2019-10