View Issue Details

IDProjectCategoryView StatusLast Update
0000448Cinelerra-GG[All Projects] Featurepublic2020-07-27 20:52
ReporterKatacuts Assigned ToPhyllisSmith  
PriorityimmediateSeveritymajorReproducibilityunable to reproduce
Status acknowledgedResolutionopen 
Product Version2020-04 
Target VersionFixed in Version 
Summary0000448: Sync externally recorded audio via time code & audio waveform
DescriptionMy sole reason I cannot use this (or any other open source editing program) professionally is because it does not have a function to sync audio recorded on a separate audio device. This could be achieved via time code or audio waveform analysis, then make subclips to join the synced audio and video together in a new clip. I could not find these functions described in the manual or in the program. If I overlooked them forgive me. Otherwise I'm very impressed with how far this program has come and how production ready it is.
TagsNo tags attached.

Activities

Pierre

Pierre

2020-07-27 20:52

updater   ~0003851

@PhyllisSmith

I tried the new "Mix masters" option. It works; it creates a mixer for each track already present in the timeline. What seems strange to me is that it creates mixers for video tracks, but also for audio tracks. Is it useful to create a mixer screen for a sound track? At the moment I'm not sure how to use this.
ferdnyc

ferdnyc

2020-07-27 00:03

reporter   ~0003845

To the question @Andrea_Paz asked about other Linux software... I couldn't possibly give a definitive answer, but I'm not personally aware of any other free/libre NLEs that make use of embedded timecode metadata. (I have no idea what's out there in the commercial space, nor do I care.)

Shotcut would seem the most likely candidate, and there doesn't APPEAR to be any support, at least from a quick survey of the interface and available documentation. There's no mention at https://www.shotcut.org/features/ which seems like the place that info would be.

And in the UI, I can see where it's able to READ the timecode metadata out of the C0001.mp4 file, as an example:
1. Load C0001.mp4 as the active Source video
2. Open the Properties panel by clicking the main toolbar icon
3. Open that panel's hamburger menu and choose "More information"
4. One of the sections of the resulting data dump (which, as the first line notes, is just the results of `ffprobe`, the same software we've been using to examine the metadata) will show:
~~~~~~~~
[streams.stream.2.tags]
creation_time=2017-02-12T01:31:28.000000Z
language=und
handler_name=Timed Metadata Media Handler
timecode=83:01:01;02
~~~~~~~~

However (and these are the critical bits), the embedded timecode doesn't show up in the Metadata tab of the Properties panel itself, nor does 83:01:01;02 ever appear anywhere in the editing interface — unlike the Creation Data metadata, which is consumed by Shotcut, and can be modified in the UI.

So, if there is any support for metadata-based synchronization, it's terrifically well hidden. I suspect there just isn't. Cinelerra-GG may indeed be one of the first/only, at least in the Linux open-source world, which I agree makes it a differentiator worth promoting.

I'd only caution against jumping the gun on any — to borrow a turn of phrase from the posterior-covering branch of Madison Avenue — "forward-looking statements" regarding what features _may_ be present in the future. (No matter how near that future appears to be.) Holding off on "advertising" the timecode-sync tools until they're completed, tested, and known to be working/useful would seem prudent. �
PhyllisSmith

PhyllisSmith

2020-07-26 23:18

manager   ~0003844

The "Align Timecodes" was moved from under Window to under Tracks pulldown per Sam suggestion. Plus a minor fix was added to align_timecodes where a clip was overlapping moves. I am still working on understanding and fixing the document in respect to a lot of the questions/answers from Andrea/ferdnyc.

@Pierre
Thanks for more feedback. I did check the "Clowns" you had sent awhile back but they had no timecodes that I could see. There is a Mint 18 version with the latest changes at:
   https://cinelerra-gg.org/download/testing/cinelerra-5.1-mint18-x86_64-static.txz
In this version note that under the Window pulldown, the Mixers option, there is now a "Mix masters" option. This has not been documented yet, but it makes it easy to load up several tracks with video and then do a bunch of edits, and then you can use the option "Mix masters" to make mixer tracks out of the edited material. You will then have to add another Video Track on the top to create the desired video like as if you had "Open Mixers" on the assets in the Resources window. There are other ways to make use of "Mix masters" too but I have not figured them out yet.
ferdnyc

ferdnyc

2020-07-24 13:08

reporter   ~0003838

(And if the "Align Timecodes" option can do some or all of that automatically, then so much the better. But the concept is the same whether it's a manual process or an automatic one.)
ferdnyc

ferdnyc

2020-07-24 13:02

reporter   ~0003837

@Andrea_Paz

Here, a theoretical example may help. Imagine you have three videos, recordings of an event from three different angles, simultaneously shot on three cameras. The cameras all started and stopped recording at different times, but they were all synced to the same master clock, so each recording is timestamped with the synchronized "world time" of the recording — all three cameras using the same times relative to each other.

So, the camera files each have an embedded timestamp for the first frame of video:
Camera 1: 00:10:13;20
Camera 2: 00:18:03;05
Camera 3: 00:06:24;07

When you load in the three files to different tracks, they'll be placed on a timeline that starts from 00:00:00:00 as usual. But, by middle-clicking on any of the video tracks, you can view the timecode embedded at that point in that file.

To make it easier to align the videos onto the timeline, it makes sense to set the "origin point" (the leftmost time index) of the timeline ruler to the timecode of the earliest video, which would be camera 3.

So, by clicking the clapboard, you can set 00:06:24;07 as the start timecode for the timeline. Then, positioning the camera 3 video at the start of the timeline will line its frames up with the times on the ruler. Having done that, you can position the other two recordings so their timecodes line up, and all three tracks will be in sync.
Andrea_Paz

Andrea_Paz

2020-07-24 09:26

updater   ~0003835

I do not really understand how timecodes work; I would like to ask a few questions:

What's the point of using the timecode option in the timebar?
How do I choose the track with the minor timecode ("The first step is to locate the earliest timecode")?
How do I use RMB --> Timecode?
What are clock times value?
With the clapperboard button and Info --> timecode you can only see if the media has an embedded timecode or can you also edit an existing one or even create it if it is not there?
How do I scan the asset using MMB?

Generic question for those who know other NLEs for linux: Is CinGG the only one who has sync via timecode or are there others? (if it is the only one you should find a way to advertise it).
Pierre

Pierre

2020-07-23 04:00

updater   ~0003832

I understand Phyllis your problem in getting multi-camera video footage with timecode correctly adjusted for use in synchronization .

This requires that the timecode generators of all the cameras involved on a shoot (in multi-camera mode) have been connected by a suitable connector and precisely synchronized with each other before shooting. Their timecode generators must also have been configured identically, either in "drop frame" or "non-drop frame" mode, as well as in "free run" mode (where the timecode flows even when cameras are not shooting). Professional studios apply these procedures and have compatible cameras that allow this.

In less industrial productions, one often has to deal with more disparate cameras and equipment, not all of which offer a reference timecode and even less often the possibility of being precisely synchronized with another device.

Personally, all I have left is my old HDV camera (HVR-Z1U) which is now very outdated compared to the current cameras. From the "pro-user" camera range it allows to record timecode but does not include the necessary connections for synchronization with another device. All the video sequences I've sent you in the past probably have some timecode in the signal, but I doubt that this is enough to make really useful tests.
PhyllisSmith

PhyllisSmith

2020-07-23 02:49

manager   ~0003831

@Andrea_Paz
@Andrew-R
@ferdnyc

This is the code gg finally came up with and checked into GIT. As many statements have been made, what to use for the timecode and what it means is ill-defined. We never came up with any good "real" test for synchronizing with an auxiliary generated audio file so until we have that kind of material, this is all that can be done. It does work for multiple cameras that have video timecodes to line up.

Attached is a quick document that will be added eventually to the manual after converted to latex. As already mentioned in email, a demo is at: https://streamable.com/nwec0t

Align_Timecodes.pdf (388,992 bytes)
PhyllisSmith

PhyllisSmith

2020-07-12 00:34

manager   ~0003794

@Andrew-R
I have downloaded some of your suggested examples as I am sure that I will be doing a lot of testing.
@ferdnyc
Actually that last article is the kind GG likes and will make sense to him (not me!)
ferdnyc

ferdnyc

2020-07-11 01:41

reporter   ~0003793

@Andrew-R *nod* Same from an unmodified ffprobe 4.2.3, I guess that patch has long since landed upstream.

One interesting wrinkle here is that we've already seen, just in two files, two completely different timecode formats.

The C0001.MP4 file contained a first-frame timecode of '83:01:01;02'. The Sony sample file, it's '00:06:06:19'. Since that value is just a _string_ value, its interpretation is left up to the reading application, and there's a subtle but potentially important in how the value should be interpreted, when the last value is separated by a semicolon vs. a colon. A semicolon signifies _drop-frame_ timecode.

Drop-frame timecode has to do with the standard video frame rates for NTSC being 29.97 fps, and the slight timing difference that introduces compared to SMPTE's standard clock rates. (Of which there are apparently only four: 24, 25, and 30 frames/second, plus 30 fps drop-frame, which is really 29.97 fps.

I'll let Wikipedia take it from here...

====From: https://en.wikipedia.org/wiki/SMPTE_timecode#Drop-frame_timecode================

The altered frame rate meant that an "hour of timecode" at a nominal frame rate of 30 frame/s, when played back at 29.97 frame/s was longer than an hour of wall-clock time by 3.6 seconds, leading to an error of almost a minute and a half over a day.

To correct this, drop-frame SMPTE timecode was invented. In spite of what the name implies, no video frames are dropped or skipped when using drop-frame timecode. Rather, some of the timecodes are dropped. In order to make an hour of timecode match an hour on the clock, drop-frame timecode skips frame numbers 0 and 1 of the first second of every minute, except when the number of minutes is divisible by ten.[a] This causes timecode to skip 18 frames each ten minutes (18,000 frames @ 30 frame/s) and almost perfectly compensates for the difference in rate.

For example, the sequence when frame counts are dropped:
01:08:59:28
01:08:59:29
01:09:00:02
01:09:00:03

For each tenth minute:
01:09:59:28
01:09:59:29
01:10:00:00
01:10:00:01

While non-drop timecode is displayed with colons separating the digit pairs—"HH:MM:SS:FF"—drop-frame is usually represented with a semicolon (;) or period (.) as the divider between all the digit pairs—"HH;MM;SS;FF", "HH.MM.SS.FF"—or just between the seconds and frames—"HH:MM:SS;FF" or "HH:MM:SS.FF".
===================

An article used as a source by that Wikipedia article, extremely informative but it gets very deep in the weeds (there are timing graphs and everything): http://www.philrees.co.uk/articles/timecode.htm
Andrew-R

Andrew-R

2020-07-10 22:08

reporter   ~0003792

https://forum.shotcut.org/t/xavc-s-format/19211 has link to this file as ffmpeg ticket sample:

 https://samples.ffmpeg.org/ffmpeg-bugs/trac/ticket5645/sony_a7s_XAVC_S_1080_25p_10s.MP4

I patched ffmpeg 2.8.17 with slightly modified patch:

/home/guest/botva/src/ffmpeg/ffmpeg-2.8.11/ffmpeg_rtmd.patch 1589/1589 CP1251 100%
--- libavformat/mov.c.orig 2020-07-07 21:49:08.000000000 +0300
+++ libavformat/mov.c 2020-07-11 00:38:30.086124634 +0300
@@ -4312,6 +4312,31 @@
     return 0;
 }

+static int mov_read_rtmd_track(AVFormatContext *s, AVStream *st)
+{
+ MOVStreamContext *sc = st->priv_data;
+ char buf[AV_TIMECODE_STR_SIZE];
+ int64_t cur_pos = avio_tell(sc->pb);
+ int hh, mm, ss, ff, drop;
+
+ if (!st->nb_index_entries)
+ return -1;
+
+ avio_seek(sc->pb, st->index_entries->pos, SEEK_SET);
+ avio_skip(s->pb, 13);
+ hh = avio_r8(s->pb);
+ mm = avio_r8(s->pb);
+ ss = avio_r8(s->pb);
+ drop = avio_r8(s->pb);
+ ff = avio_r8(s->pb);
+ snprintf(buf, AV_TIMECODE_STR_SIZE, "%02d:%02d:%02d%c%02d",
+ hh, mm, ss, drop ? ';' : ':', ff);
+ av_dict_set(&st->metadata, "timecode", buf, 0);
+
+ avio_seek(sc->pb, cur_pos, SEEK_SET);
+ return 0;
+}
+
 static int mov_read_timecode_track(AVFormatContext *s, AVStream *st)
 {
     MOVStreamContext *sc = st->priv_data;
@@ -4578,8 +4603,11 @@
         if (mov->chapter_track > 0)
             mov_read_chapters(s);
         for (i = 0; i < s->nb_streams; i++)
- if (s->streams[i]->codec->codec_tag == AV_RL32("tmcd"))
+ if (s->streams[i]->codec->codec_tag == AV_RL32("tmcd")) {
                 mov_read_timecode_track(s, s->streams[i]);
+ } else if (s->streams[i]->codec->codec_tag == AV_RL32("rtmd")) {
+ mov_read_rtmd_track(s, s->streams[i]);
+ }
     }

     /* copy timecode metadata from tmcd tracks to the related video streams */

and then output was like this:

 ffprobe sony_a7s_XAVC_S_1080_25p_10s.MP4
ffprobe version 2.8.17 Copyright (c) 2007-2020 the FFmpeg developers
  built with gcc 5.5.0 (GCC)
  configuration: --arch=i486 --target-os=linux --prefix=/usr --libdir=/usr/lib --mandir=/usr/man --docdir=/usr/doc/ffmpeg-2.8.17 --enable-gpl --enable-version3 --disable-static --enable-shared --enable-runtime-cpudetect --enable-ffmpeg --enable-ffplay --enable-ffprobe --enable-ffserver --enable-doc --enable-avdevice --enable-avcodec --enable-avformat --enable-avutil --enable-swresample --enable-swscale --enable-postproc --enable-avfilter --enable-avresample --enable-pthreads --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnutls --enable-iconv --enable-ladspa --enable-libass --enable-libbluray --disable-libbs2b --disable-libcaca --disable-libcelt --enable-libcdio --disable-libdc1394 --disable-libflite --enable-libfreetype --enable-libfribidi --disable-libgme --enable-libgsm --enable-libiec61883 --disable-libilbc --disable-libkvazaar --disable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --disable-libpulse --disable-libquvi --enable-librtmp --enable-libschroedinger --enable-libsmbclient --enable-libsnappy --disable-libsoxr --enable-libspeex --disable-libssh --enable-libtheora --enable-libtwolame --enable-libv4l2 --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxcb --enable-libxvid --enable-libzvbi --enable-lzma --enable-openal --enable-opengl --enable-sdl --enable-x11grab --enable-zlib --disable-debug
  libavutil 54. 31.100 / 54. 31.100
  libavcodec 56. 60.100 / 56. 60.100
  libavformat 56. 40.101 / 56. 40.101
  libavdevice 56. 4.100 / 56. 4.100
  libavfilter 5. 40.101 / 5. 40.101
  libavresample 2. 1. 0 / 2. 1. 0
  libswscale 3. 1.101 / 3. 1.101
  libswresample 1. 2.101 / 1. 2.101
  libpostproc 53. 3.100 / 53. 3.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'sony_a7s_XAVC_S_1080_25p_10s.MP4':
  Metadata:
    major_brand : XAVC
    minor_version : 16785407
    compatible_brands: XAVCmp42iso2
    creation_time : 2016-03-05 14:17:25
  Duration: 00:00:10.08, start: 0.000000, bitrate: 49084 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv), 1920x1080 [SAR 1:1 DAR 16:9], 47339 kb/s, 25 fps, 25 tbr, 25k tbn, 50 tbc (default)
    Metadata:
      creation_time : 2016-03-05 14:17:25
      handler_name : Video Media Handler
      encoder : AVC Coding
    Stream #0:1(und): Audio: pcm_s16be (twos / 0x736F7774), 48000 Hz, 2 channels, s16, 1536 kb/s (default)
    Metadata:
      creation_time : 2016-03-05 14:17:25
      handler_name : Sound Media Handler
    Stream #0:2(und): Data: none (rtmd / 0x646D7472), 204 kb/s (default)
    Metadata:
      creation_time : 2016-03-05 14:17:25
      handler_name : Non-Real Time Metadata
      timecode : 00:06:06:19
Unsupported codec with id 0 for input stream 2
Andrew-R

Andrew-R

2020-07-10 20:53

reporter   ~0003791

@ferdnyc:

I think even my old ffmpeg 2.8.15 can write 'artificial' timecode into MOV container, like this:
  ffmpeg -i C0126.MP4 -timecode 02:02:03:03 -c:a copy output.mov
ffmpeg version 2.8.15 Copyright (c) 2000-2018 the FFmpeg developers
  built with gcc 5.5.0 (GCC)
  configuration: --arch=i486 --target-os=linux --prefix=/usr --libdir=/usr/lib --mandir=/usr/man --docdir=/usr/doc/ffmpeg-2.8.15 --enable-gpl --enable-version3 --disable-static --enable-shared --enable-runtime-cpudetect --enable-ffmpeg --enable-ffplay --enable-ffprobe --enable-ffserver --enable-doc --enable-avdevice --enable-avcodec --enable-avformat --enable-avutil --enable-swresample --enable-swscale --enable-postproc --enable-avfilter --enable-avresample --enable-pthreads --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnutls --enable-iconv --enable-ladspa --enable-libass --enable-libbluray --disable-libbs2b --disable-libcaca --disable-libcelt --enable-libcdio --disable-libdc1394 --disable-libflite --enable-libfreetype --enable-libfribidi --disable-libgme --enable-libgsm --enable-libiec61883 --disable-libilbc --disable-libkvazaar --disable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --disable-libpulse --disable-libquvi --enable-librtmp --enable-libschroedinger --enable-libsmbclient --enable-libsnappy --disable-libsoxr --enable-libspeex --disable-libssh --enable-libtheora --enable-libtwolame --enable-libv4l2 --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxcb --enable-libxvid --enable-libzvbi --enable-lzma --enable-openal --enable-opengl --enable-sdl --enable-x11grab --enable-zlib --disable-debug
  libavutil 54. 31.100 / 54. 31.100
  libavcodec 56. 60.100 / 56. 60.100
  libavformat 56. 40.101 / 56. 40.101
  libavdevice 56. 4.100 / 56. 4.100
  libavfilter 5. 40.101 / 5. 40.101
  libavresample 2. 1. 0 / 2. 1. 0
  libswscale 3. 1.101 / 3. 1.101
  libswresample 1. 2.101 / 1. 2.101
  libpostproc 53. 3.100 / 53. 3.100
Guessed Channel Layout for Input Stream #0.1 : stereo
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'C0126.MP4':
  Metadata:
    major_brand : XAVC
    minor_version : 16785407
    compatible_brands: XAVCmp42iso2
    creation_time : 2015-05-16 05:52:01
  Duration: 00:00:04.80, start: 0.000000, bitrate: 48822 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709/bt709/iec61966-2-4), 1920x1080 [SAR 1:1 DAR 16:9], 46870 kb/s, 50 fps, 50 tbr, 50k tbn, 100 tbc (default)
    Metadata:
      creation_time : 2015-05-16 05:52:01
      handler_name : Video Media Handler
      encoder : AVC Coding
    Stream #0:1(und): Audio: pcm_s16be (twos / 0x736F7774), 48000 Hz, 2 channels, s16, 1536 kb/s (default)
    Metadata:
      creation_time : 2015-05-16 05:52:01
      handler_name : Sound Media Handler
    Stream #0:2(und): Data: none (rtmd / 0x646D7472), 409 kb/s (default)
    Metadata:
      creation_time : 2015-05-16 05:52:01
      handler_name : Non-Real Time Metadata
[libx264 @ 0x8b4a8e0] using SAR=1/1
[libx264 @ 0x8b4a8e0] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX XOP FMA4 FMA3 LZCNT BMI1
[libx264 @ 0x8b4a8e0] profile High, level 4.2
[libx264 @ 0x8b4a8e0] 264 - core 148 - H.264/MPEG-4 AVC codec - Copyleft 2003-2016 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=6 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00
[mov @ 0x8b49ac0] Codec for stream 1 does not use global headers but container format requires global headers
Output #0, mov, to 'output.mov':
  Metadata:
    major_brand : XAVC
    minor_version : 16785407
    compatible_brands: XAVCmp42iso2
    timecode : 02:02:03:03
    encoder : Lavf56.40.101
    Stream #0:0(und): Video: h264 (libx264) (avc1 / 0x31637661), yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], q=-1--1, 50 fps, 12800 tbn, 50 tbc (default)
    Metadata:
      creation_time : 2015-05-16 05:52:01
      handler_name : Video Media Handler
      encoder : Lavc56.60.100 libx264
    Stream #0:1(und): Audio: pcm_s16be (twos / 0x736F7774), 48000 Hz, stereo, 1536 kb/s (default)
    Metadata:
      creation_time : 2015-05-16 05:52:01
      handler_name : Sound Media Handler
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> h264 (libx264))
  Stream #0:1 -> #0:1 (copy)
Press [q] to stop, [?] for help
frame= 240 fps= 12 q=-1.0 Lsize= 7864kB time=00:00:04.80 bitrate=13421.7kbits/s
video:6957kB audio:900kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.088144%
[libx264 @ 0x8b4a8e0] frame I:1 Avg QP:25.63 size:364332
[libx264 @ 0x8b4a8e0] frame P:60 Avg QP:25.98 size: 96772
[libx264 @ 0x8b4a8e0] frame B:179 Avg QP:32.04 size: 5324
[libx264 @ 0x8b4a8e0] consecutive B-frames: 0.4% 0.0% 1.2% 98.3%
[libx264 @ 0x8b4a8e0] mb I I16..4: 4.5% 18.4% 77.1%
[libx264 @ 0x8b4a8e0] mb P I16..4: 0.0% 0.0% 0.0% P16..4: 46.8% 28.3% 22.3% 0.0% 0.0% skip: 2.6%
[libx264 @ 0x8b4a8e0] mb B I16..4: 0.0% 0.0% 0.0% B16..8: 54.2% 1.1% 0.1% direct: 0.2% skip:44.4% L0:65.0% L1:28.3% BI: 6.7%
[libx264 @ 0x8b4a8e0] 8x8 transform intra:18.6% inter:49.1%
[libx264 @ 0x8b4a8e0] coded y,uvDC,uvAC intra: 99.7% 87.8% 44.3% inter: 19.6% 9.4% 0.0%
[libx264 @ 0x8b4a8e0] i16 v,h,dc,p: 3% 2% 72% 24%
[libx264 @ 0x8b4a8e0] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 13% 7% 14% 16% 7% 7% 5% 20% 12%
[libx264 @ 0x8b4a8e0] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 19% 10% 16% 11% 7% 7% 6% 15% 9%
[libx264 @ 0x8b4a8e0] i8c dc,h,v,p: 56% 20% 18% 7%
[libx264 @ 0x8b4a8e0] Weighted P-Frames: Y:0.0% UV:0.0%
[libx264 @ 0x8b4a8e0] ref P L0: 51.1% 15.8% 27.9% 5.2%
[libx264 @ 0x8b4a8e0] ref B L0: 94.1% 5.7% 0.2%
[libx264 @ 0x8b4a8e0] ref B L1: 98.5% 1.5%
[libx264 @ 0x8b4a8e0] kb/s:11872.72
[email protected]:/dev/shm$ ffprobe output.mov
ffprobe version 2.8.15 Copyright (c) 2007-2018 the FFmpeg developers
  built with gcc 5.5.0 (GCC)
  configuration: --arch=i486 --target-os=linux --prefix=/usr --libdir=/usr/lib --mandir=/usr/man --docdir=/usr/doc/ffmpeg-2.8.15 --enable-gpl --enable-version3 --disable-static --enable-shared --enable-runtime-cpudetect --enable-ffmpeg --enable-ffplay --enable-ffprobe --enable-ffserver --enable-doc --enable-avdevice --enable-avcodec --enable-avformat --enable-avutil --enable-swresample --enable-swscale --enable-postproc --enable-avfilter --enable-avresample --enable-pthreads --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnutls --enable-iconv --enable-ladspa --enable-libass --enable-libbluray --disable-libbs2b --disable-libcaca --disable-libcelt --enable-libcdio --disable-libdc1394 --disable-libflite --enable-libfreetype --enable-libfribidi --disable-libgme --enable-libgsm --enable-libiec61883 --disable-libilbc --disable-libkvazaar --disable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --disable-libpulse --disable-libquvi --enable-librtmp --enable-libschroedinger --enable-libsmbclient --enable-libsnappy --disable-libsoxr --enable-libspeex --disable-libssh --enable-libtheora --enable-libtwolame --enable-libv4l2 --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxcb --enable-libxvid --enable-libzvbi --enable-lzma --enable-openal --enable-opengl --enable-sdl --enable-x11grab --enable-zlib --disable-debug
  libavutil 54. 31.100 / 54. 31.100
  libavcodec 56. 60.100 / 56. 60.100
  libavformat 56. 40.101 / 56. 40.101
  libavdevice 56. 4.100 / 56. 4.100
  libavfilter 5. 40.101 / 5. 40.101
  libavresample 2. 1. 0 / 2. 1. 0
  libswscale 3. 1.101 / 3. 1.101
  libswresample 1. 2.101 / 1. 2.101
  libpostproc 53. 3.100 / 53. 3.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'output.mov':
  Metadata:
    major_brand : qt
    minor_version : 512
    compatible_brands: qt
    encoder : Lavf56.40.101
  Duration: 00:00:04.80, start: 0.000000, bitrate: 13421 kb/s
    Stream #0:0(eng): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], 11873 kb/s, 50 fps, 50 tbr, 12800 tbn, 100 tbc (default)
    Metadata:
      handler_name : DataHandler
      encoder : Lavc56.60.100 libx264
      timecode : 02:02:03:03
    Stream #0:1(eng): Audio: pcm_s16be (twos / 0x736F7774), 48000 Hz, stereo, s16, 1536 kb/s (default)
    Metadata:
      handler_name : DataHandler
    Stream #0:2(eng): Data: none (tmcd / 0x64636D74), 0 kb/s
    Metadata:
      handler_name : DataHandler
      timecode : 02:02:03:03
Unsupported codec with id 0 for input stream 2
[email protected]:/dev/shm$ mediainfo output.mov
General
Complete name : output.mov
Format : MPEG-4
Format profile : QuickTime
Codec ID : qt 0000.02 (qt )
File size : 7.68 MiB
Duration : 4 s 800 ms
Overall bit rate : 13.4 Mb/s
Writing application : Lavf56.40.101

Video
ID : 1
Format : AVC
Format/Info : Advanced Video Codec
Format profile : [email protected]
Format settings : CABAC / 4 Ref Frames
Format settings, CABAC : Yes
Format settings, Reference frames : 4 frames
Codec ID : avc1
Codec ID/Info : Advanced Video Coding
Duration : 4 s 800 ms
Bit rate : 11.9 Mb/s
Width : 1 920 pixels
Height : 1 080 pixels
Display aspect ratio : 16:9
Frame rate mode : Constant
Frame rate : 50.000 FPS
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Scan type : Progressive
Bits/(Pixel*Frame) : 0.115
Stream size : 6.79 MiB (88%)
Writing library : x264 core 148
Encoding settings : cabac=1 / ref=3 / deblock=1:0:0 / analyse=0x3:0x113 / me=hex / subme=7 / psy=1 / psy_rd=1.00:0.00 / mixed_ref=1 / me_range=16 / chroma_me=1 / trellis=1 / 8x8dct=1 / cqm=0 / deadzone=21,11 / fast_pskip=1 / chroma_qp_offset=-2 / threads=6 / lookahead_threads=1 / sliced_threads=0 / nr=0 / decimate=1 / interlaced=0 / bluray_compat=0 / constrained_intra=0 / bframes=3 / b_pyramid=2 / b_adapt=1 / b_bias=0 / direct=1 / weightb=1 / open_gop=0 / weightp=2 / keyint=250 / keyint_min=25 / scenecut=40 / intra_refresh=0 / rc_lookahead=40 / rc=crf / mbtree=1 / crf=23.0 / qcomp=0.60 / qpmin=0 / qpmax=69 / qpstep=4 / ip_ratio=1.40 / aq=1:1.00
Language : English
Codec configuration box : avcC

Audio
ID : 2
Format : PCM
Format settings : Big / Signed
Codec ID : twos
Duration : 4 s 800 ms
Bit rate mode : Constant
Bit rate : 1 536 kb/s
Channel(s) : 2 channels
Channel layout : L R
Sampling rate : 48.0 kHz
Bit depth : 16 bits
Stream size : 900 KiB (11%)
Language : English
Default : Yes
Alternate group : 1

Other
ID : 3
Type : Time code
Format : QuickTime TC
Duration : 4 s 800 ms
Time code of first frame : 02:02:03:03
Time code, striped : Yes
Language : English
Default : No

In theory, new-ish ffmpeg should support writing those in mp4, too:
https://git.ffmpeg.org/gitweb/ffmpeg.git/commit/b4dcd351ec50caaa484bc5c66b4a8d5557a0f1ea
committer Michael Niedermayer <[email protected]>
  Sun, 28 Feb 2016 05:47:23 +0300 (03:47 +0100)
  commit b4dcd351ec50caaa484bc5c66b4a8d5557a0f1ea
 movenc: Timecode in MP4 Although MP4 does not have a concrete specification to store timecode information, the following technical note from Apple describes a way to achieve this via timecode track, similar to how it is done for MOV files.
 
 https://developer.apple.com/library/mac/technotes/tn2174/_index.html
 
 - Enabled creation of timecode tracks for MP4 in the same way as MOV.
 - Used nmhd as media information header of timecode track of MP4 instead
   of gmhd used in MOV, thus avoiding tcmi also, as recommended above.
 - Bypassed adding source reference field for MP4, as suggested above.
 
 Issue: https://trac.ffmpeg.org/ticket/4704
----
but not tested by me personally, due to old system ffmpeg (may be I'll patch it)

Additionally, I found rtmd track timecode was supported in newer ffmpeg:
https://git.ffmpeg.org/gitweb/ffmpeg.git/commit/d693392886b8454c818e384c816b9ede53c570d8
Mon, 27 Jun 2016 11:27:25 +0300 (10:27 +0200)
  commit d693392886b8454c818e384c816b9ede53c570d8
   avformat/mov: parse rtmd track timecode
 
so, there is no need to mess with mp4v2 utilites .....

Hopefully this will provide some way to test changes in CinGG without hunting all those sample files, initially?
But I'll continue to look around for real ones ....
ferdnyc

ferdnyc

2020-07-10 19:11

reporter   ~0003790

@Andrew-R There's no question that GOPs are time_stamped_ when encoded, among other things it's how the decoder ensures that it's displaying the correct frames in the correct order. But like I said, normally those just start at 0 and work up from there — which is exactly what your mp4file dump shows.

To obtain timecode useful for synchronizing separate files, there would need to be some sort of field that contains, as that one user said in the discussion about which clock to use, "any field which is different from 0:0:0:0". (Or something monotonically increasing from 0 based on the playback rate, like all of the timecodes displayed in that dump.)

Ah! Here we go, yes. That C0001.MP4 file does contain useful timecode data, in a "Timed Metadata" RTMD (real-time metadata) stream. Though, the last line of output being an error message about the "unsupported codec" for the RTMD stream makes me nervous about ffmpeg's ability to usefully _decode_ that stream. Still, it can clearly at least extract the per-*stream* metadata, including the stored external-clock reference timecode for the start of recording:


$ ffprobe -i C0001.MP4
ffprobe version 4.2.3 Copyright (c) 2007-2020 the FFmpeg developers
  built with gcc 10 (GCC)
  configuration: --prefix=/usr --bindir=/usr/bin --datadir=/usr/share/ffmpeg --docdir=/usr/share/doc/ffmpeg --incdir=/usr/include/ffmpeg --libdir=/usr/lib64 --mandir=/usr/share/man --arch=x86_64 --optflags='-O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection' --extra-ldflags='-Wl,-z,relro -Wl,--as-needed -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld ' --extra-cflags=' ' --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libvo-amrwbenc --enable-version3 --enable-bzlib --disable-crystalhd --enable-fontconfig --enable-frei0r --enable-gcrypt --enable-gnutls --enable-ladspa --enable-libaom --enable-libdav1d --enable-libass --enable-libbluray --enable-libcdio --enable-libdrm --enable-libjack --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libmp3lame --enable-nvenc --enable-openal --enable-opencl --enable-opengl --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-librsvg --enable-libsrt --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libvorbis --enable-libv4l2 --enable-libvidstab --enable-libvmaf --enable-libvpx --enable-libx264 --enable-libx265 --enable-libxvid --enable-libzimg --enable-libzvbi --enable-avfilter --enable-avresample --enable-libmodplug --enable-postproc --enable-pthreads --disable-static --enable-shared --enable-gpl --disable-debug --disable-stripping --shlibdir=/usr/lib64 --enable-libmfx --enable-runtime-cpudetect
  libavutil 56. 31.100 / 56. 31.100
  libavcodec 58. 54.100 / 58. 54.100
  libavformat 58. 29.100 / 58. 29.100
  libavdevice 58. 8.100 / 58. 8.100
  libavfilter 7. 57.100 / 7. 57.100
  libavresample 4. 0. 0 / 4. 0. 0
  libswscale 5. 5.100 / 5. 5.100
  libswresample 3. 5.100 / 3. 5.100
  libpostproc 55. 5.100 / 55. 5.100
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x561ea444a440] st: 0 edit list: 1 Missing key frame while searching for timestamp: 1001
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x561ea444a440] st: 0 edit list 1 Cannot find an index entry before timestamp: 1001.
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'C0001.MP4':
  Metadata:
    major_brand : XAVC
    minor_version : 16785407
    compatible_brands: XAVCmp42iso2
    creation_time : 2017-02-12T01:31:28.000000Z
  Duration: 00:00:49.05, start: 0.000000, bitrate: 97074 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709/bt709/iec61966-2-4), 3840x2160 [SAR 1:1 DAR 16:9], 95289 kb/s, 29.97 fps, 29.97 tbr, 30k tbn, 59.94 tbc (default)
    Metadata:
      creation_time : 2017-02-12T01:31:28.000000Z
      handler_name : Video Media Handler
      encoder : AVC Coding
    Stream #0:1(und): Audio: pcm_s16be (twos / 0x736F7774), 48000 Hz, 2 channels, s16, 1536 kb/s (default)
    Metadata:
      creation_time : 2017-02-12T01:31:28.000000Z
      handler_name : Sound Media Handler
    Stream #0:2(und): Data: none (rtmd / 0x646D7472), 245 kb/s (default)
    Metadata:
      creation_time : 2017-02-12T01:31:28.000000Z
      handler_name : Timed Metadata Media Handler
      timecode : 83:01:01;02
Unsupported codec with id 0 for input stream 2
Andrew-R

Andrew-R

2020-07-10 15:27

reporter   ~0003789

Update on mp4v2 utils, I entered wrong command, now after installing new Slackware package (https://slackbuilds.org/repository/14.2/libraries/libmp4v2/ - this one lists https://github.com/sergiomb2/libmp4v2 as source) and typing

mp4trackdump -t 3 C0126.MP4

I see list of timecodes (?):
mp4trackdump -t 3 C0126.MP4
mp4trackdump version -r
mp4file C0126.MP4, track 3, samples 240, timescale 50000
sampleId 1, size 1024 duration 1000 time 0 00:00:00.000 S
sampleId 2, size 1024 duration 1000 time 1000 00:00:00.020 S
sampleId 3, size 1024 duration 1000 time 2000 00:00:00.040 S
sampleId 4, size 1024 duration 1000 time 3000 00:00:00.060 S
sampleId 5, size 1024 duration 1000 time 4000 00:00:00.080 S
sampleId 6, size 1024 duration 1000 time 5000 00:00:00.100 S
sampleId 7, size 1024 duration 1000 time 6000 00:00:00.120 S
sampleId 8, size 1024 duration 1000 time 7000 00:00:00.140 S
sampleId 9, size 1024 duration 1000 time 8000 00:00:00.160 S
sampleId 10, size 1024 duration 1000 time 9000 00:00:00.180 S
Andrew-R

Andrew-R

2020-07-10 15:12

reporter   ~0003788

More XAVC-S findings:
https://github.com/SK-Hardwired/xavc_rtmd2srt
XAVC S video real-time meta-data parser and GPS data extractor
Extract realtime meta-data (i.e. per-frame) from Sony XAVC video and render to srt subtitle file.

Written in python 3.7

More timecode woes from Lightswork user forum:
https://www.lwks.com/index.php?option=com_kunena&func=view&catid=12&id=131106&Itemid=81#ftop
----quote----
David Rasberry
 Pro User
 
OFFLINE
 Posts: 2658
  
 3 years, 2 months ago
   #145410
hugly wrote:
 
My personal favorite for amateur usage is still MPEG2 I-Frame codec. High transcoding speeds, reasonable file sizes and bitrates, good quality, fluently editable, support of almost any resolution and frame rate, no licensing problems, playable in any media player. What more do I (amateur) need?
 
 By the way, jpaulbiss footage (and other AVCHD/PCM.MOV camera recorded footage I tested) has a valid timecode track containing SMPTE compliant start T/C in Quicktime format. With ProRes/PCM in a MOV container, Lightworks will read the Quicktime timecode track properly, with AVCHD present it doesn't (T/C zero after import). I assume there are two different demuxers at work for MOV containers. From my point of view this is a major problem because AVCHD in MOV container is a commonly used recording format.
 All this isn’t new, but from my point of view this issue should be fixed soon.

 
 I looked into this sometime back comparing a Prores LT.mov compressed file with AVCHD i-frame compressed file in .mov format. Both .mov files have the same set of TC fields in the file header, which is where the various clock field values for different start frame clocks are registered. There is a specific TC field that is populated in the ProRes file that translates the relevant clock code to SMPTE format. That field is not populated in any H.264 AVC variant from most low end cameras. There are separate clock fields for shot start 00:00:00:00, time of day clock if present, and free run clock from camera startup. Any of these can be used for the SMPTE TC field depending on camera settings and whether the camera firmware sets the SMPTE field or not.
 Lightworks only reads the SMPTE reference field. QT timecode is not automatically SMPTE compliant, but can be. If you ask Lightworks to read something other than the SMPTE field, which of the 5 or so clock fields present is it to pick from?
  ---quote end--

On this user suggested:
>which of the 5 or so clock fields present is it to pick from?
 
 I'd say any field which is different from 0:0:0:0 would be better than none ,-)
 ---quote end----

So, may be CinGG can reuse same startegy - search for first non-all-zeros timecode, and display/use it,
at least initially ?
Andrew-R

Andrew-R

2020-07-10 13:20

reporter   ~0003787

some more files in XAVC-S format (?) found at forum:

https://forum.videohelp.com/threads/371906-XAVC-S-How-to-export-time-date-metadata-to-subtitles-(srt)

Not sure, my mp4extract fails to extract metadata track ...

mp4v2 utility apparently moved to
https://github.com/TechSmith/mp4v2 ?

format of this track posted as:

---quote---
This is the format of the RTMD stream for those that don't want to DL it
 
 

 mp4file C0126.MP4, track 3, samples 240, timescale 50000
 sampleId 1, size 1024 duration 1000 time 0 00:00:00.000 S
 sampleId 2, size 1024 duration 1000 time 1000 00:00:00.020 S
 sampleId 3, size 1024 duration 1000 time 2000 00:00:00.040 S
 sampleId 4, size 1024 duration 1000 time 3000 00:00:00.060 S
 sampleId 5, size 1024 duration 1000 time 4000 00:00:00.080 S
 sampleId 6, size 1024 duration 1000 time 5000 00:00:00.100 S
 sampleId 7, size 1024 duration 1000 time 6000 00:00:00.120 S
 sampleId 8, size 1024 duration 1000 time 7000 00:00:00.140 S
 sampleId 9, size 1024 duration 1000 time 8000 00:00:00.160 S
---quote end----




There is also ffmpeg ticket about wrong timecode extraction, currently downloading bigger file

https://trac.ffmpeg.org/ticket/6793
 wget http://50.19.4.114/bugs/ffmpeg/C0001.MP4

a bit slow ...
Andrew-R

Andrew-R

2020-07-10 12:30

reporter   ~0003786

More timecode discussion (in relation to proprietary editors):

https://www.vegascreativesoftware.info/us/forum/date-created-to-timecode-in-or-date-time-stamp--112646/

----quote----
 doubleJ wrote on 9/1/2018, 3:55 PM
So, on an even wilder front...
I downloaded a bunch of sample footage (R3D, XAVC, XDCAM, DNxHD, DV, ProRes, maybe others).
I imported 17 video files into Vegas Pro 16 and Premiere Pro CC 2018.
Of all the files, only 2 showed timecode in both Vegas and Premiere (and that was an old DV AVI).
There were 8 files with timecode in Vegas and 5 files with timecode in Premiere.
It's likely that some of the other files simply didn't have timecode, but that's a pretty narrow margin.
How is a person supposed to make "standards-compliant" files?
Hehehe...
Going back to my initial topic, though...
If I was going to transcode video into one of these formats that does work, would I be able to insert timecode during that process?
Maybe if I would use FFMPEG, TMPGEnc, or VirtualDubMod?
JJ
---
 doubleJ wrote on 9/2/2018, 6:43 PM
Ok, I'm making some head-way...
This just adds some metadata and it shows the timecode in both NLEs.
ffmpeg.exe -i originalfile.mp4 -metadata timecode="14:36:08:29" -metadata creation_time="2018-08-05 14:36:08" -c copy newfile.mp4
input file
timecode key and value in metadata
creation date/time metadata (this only seems to show in File Explorer properties details)
don’t do any encoding (just copy the video/audio)
output file
This does the same thing, but transcodes to the Avid DNxHD intermediate format (mxf didn’t open in Vegas, so I used mov).
ffmpeg.exe -i originalfile.mp4 -metadata timecode="14:36:08:29" -metadata creation_time="2018-08-05 14:36:08" -c:v dnxhd -b:v 145M -s 1920x1080 -r 30 -c:a pcm_s16le newfile.mov
I also tried ProRes, but Vegas Pro didn’t see the timecode.
I haven't been able to find ffmpeg information on encoding to CineForm.
So, now the work is to create a batch file that will read the Date Created file attribute and use it for timecode and creation_time.
I downloaded a batch script that reads a directory and echoes Date Created and Time Created, so I'm hopeful.
JJ
----quote end----

I don't know where exactly you can find mentioned source files (probably some camera sample file repo?)

pdf remotely related to this problem:
https://mediaarea.net/Events/2018-10-26_NoTimeToWait3/presentations/19. Ben Turkus, Kelly Haydon - If we could turn back timecode/nttw3_timecode_final.pdf (with spaces in URL)

Both files here should contain some 'timecode breaks' , or at least something Kino can recognize, so one file shows up as bunch of clips):

https://yadi.sk/d/I0jFMoL9Tg5V0w
Andrew-R

Andrew-R

2020-07-10 10:43

reporter   ~0003785

I think you can try some old DV samples - old Cin codebase apparently used libdv's interface to such data, but for ffmpeg's APIs testing/prototyping you _probably_ can reuse some of those files (AVI container apparently lack timecode track possibility, but codec-level timecode from camcoder still around).

Let me know if you need some HUGE (2Gb+) samples, via cloud.mail.ru or Yandex disk.

 I think some of my 'research' still lives at https://www.cinelerra-gg.org/bugtracker/view.php?id=332 and ..somewhere else (still searching for it)
PhyllisSmith

PhyllisSmith

2020-07-10 03:57

manager   ~0003784

@Andrea - your reference is helpful to me as I think this is what GG has been trying to explain to me for years. This article makes it a lot clearer than he does.

@ferdnyc
Thanks for the hints again. He is now talking about time since 1970, so that may be the current approach as you suggested in:
"That's defined as the "Start time of the stream in real world time, in microseconds since the Unix epoch (00:00 1st January 1970).", so if there were going to be an externally-referenced timecode value, that's likely where it would be stored (or it would be relative to that field)."

His plan is to write code to do the work with the idea that what he actually uses for the "timecode" synch may have to change if he ever gets example media.
Andrea_Paz

Andrea_Paz

2020-07-08 14:07

updater   ~0003764

I don't know if this will help:
http://dranger.com/ffmpeg/tutorial05.html
PhyllisSmith

PhyllisSmith

2020-07-08 14:00

manager   ~0003763

gg writes:

I have identified several sources of timecode data that can be considered for use.
There is still quite a lot of discovery needed before a good implementation may
be started.

The design can be broken into two program sets.
1) get_initial_timecode(edit)
2) align armed tracks on mwindow timeline

get_initial_timecode should access the edit asset media, determine the program set
that contains that edit's video layer or audio channel, determine the (video?) master
layer, and read the media for pts (presentation timestamp) and timecode data.
Once both (or equivalent) data are available, the apparent timecode of the initial
frame/sample should be returned. This should make it possible to add a new
edit popup menuitem to set the main window timebar (in timecode hh:mm:ss:ff),
by using the session frame rate as a time standard.

A new menubar menuitem, align armed tracks on mwindow timeline, can then
iterate over a set of selected edits, and similarly determine the edit timecode
offset, then adjust the edit startproject or startsource position to align the
edit with the main window timeline timecodes.

Each file codec can potentially provide timecode information. So far, I have
looked into fileffmpeg and filempeg, but I suspect that there is more to follow.
ffmpeg has at least 5 sources of timecode data that are present in various
formulation.
1) AVFormatContext metadata "timecode"
2) AVStream metadata "timecode"
3) AVFrame metadata "timecode"
4) AVFrameSideData type AV_FRAME_DATA_GOP_TIMECODE
5) AVFrameSideData type AV_FRAME_DATA_S12M_TIMECODE
filempeg has GOP (group of pictures) smtpe timecodes that are only present in
the video start_gop headers.

What to do when there are no usable timecodes is still an open question.
Audio tracks may not have any timecodes and may not belong to a program group.

All of this said, I definitely will need an example set of media data to do
testing. I do not have any such thing, and that is a show stopper.

Any suggestions at this point are welcome, and any example media would be
greatly appreciated.

gg
ferdnyc

ferdnyc

2020-07-08 04:43

reporter   ~0003760

The patches referenced by @Andrew-R will add internal timecode information to the metadata of various codecs. But for the most part, FFmpeg's internal timecode is relative to the start of the data stream. So it's not clear to me whether it would have a format that's real-world synchronizable.

The patchset's updated unit test for frame side data (something that already exists, even without those patches) uses 0-relative timecodes, which makes me suspect that it would not:
http://ffmpeg.org/pipermail/ffmpeg-devel/2020-June/265036.html

There's also the fact that timecode_frame_start in the codec context is a deprecated attribute: https://www.ffmpeg.org/doxygen/4.1/structAVCodecContext.html#a98fefd16c6f7eaaaca1aadf80b8a9453
That makes the whole thing sort of weird, TBH.

One *potential* for accessing external timecode data is the AVFormatContext attribute start_time_realtime:
https://www.ffmpeg.org/doxygen/4.1/structAVFormatContext.html#aa5ddb5cee1df28f21739133f2e37f1c5
That's defined as the "Start time of the stream in real world time, in microseconds since the Unix epoch (00:00 1st January 1970).", so if there were going to be an externally-referenced timecode value, that's likely where it would be stored (or it would be relative to that field).

But I'm not sure under what conditions, or if ever, start_time_realtime will be set for streams decoded from files. It sounds like it's primarily intended as an attribute for realtime streamed data (like internet livestreams, for example).

It's also entirely possible that timecode_frame_start WILL be set relative to an external timecode, if one is stored in the file's metadata. I agree that sample files would be extremely useful, since few of us ever have occasion to work with files that use an external clock reference.
PhyllisSmith

PhyllisSmith

2020-07-06 02:09

manager   ~0003741

@Katacuts
GG started an investigation on "timecodes" today. It is not at all clear which timecode in the file is the one to use -- there are 4 different things that are referred to as a timecode. Quite confusing.

In order to further investigate, it would be helpful if you could send a set of at least 3 files that belong to a set to be synched up so he can figure out which timecode is relevant. Any additional information for clarification would be highly appreciated as we are computer people and not A/V technicians. Unfortunately without more technical information and a set of valid test media, we are kind of stuck on whether or not this can even be implemented by us. If you can put the files in a dropbox or shared driver and email [email protected] we will keep them private.

@Andrew-R
THANKS for the ffmpeg patch hints as it provided a starting place for GG to look.
Andrew-R

Andrew-R

2020-07-04 07:20

reporter   ~0003733

More timecode patches about to hit (hopefully) ffmpeg 4.4:
http://ffmpeg.org/pipermail/ffmpeg-devel/2020-July/265560.html
[[FFmpeg-devel] [PATCH 1/4] avcodec/hevcdec: add timecode metadata]
and others ...
http://ffmpeg.org/pipermail/ffmpeg-devel/2020-July/265563.html
[FFmpeg-devel] [PATCH 3/4] avcodec/h264_slice: add timecode metadata

with commit message saying:
"Please test with below command:
./ffplay -vf drawtext="fontfile=/Library/Fonts/Arial.ttf:text=\\'%{metadata\\:timecode}\\'" \
                       ../fate-suite/h264/crew_cif_timecode-2.h264"

I guess developer was working from Mac ?

also as patch series:
https://patchwork.ffmpeg.org/project/ffmpeg/list/?series=1664
Andrew-R

Andrew-R

2020-06-15 18:36

reporter   ~0003620

Not sure if 'timecode' here and timecode in those patches talk about same thing, but I spotted those two patches on ffmpeg-devel:

http://ffmpeg.org/pipermail/ffmpeg-devel/2020-June/264622.html
[FFmpeg-devel] [PATCH 1/2] avcodec/hevc_sei: support HEVC timecode decode
[FFmpeg-devel] [PATCH 2/2] avcodec/hevcdec: create AVFrame side data from HEVC timecodes like H.264
PhyllisSmith

PhyllisSmith

2020-06-09 18:12

manager   ~0003576

@Andrea_Paz
Will you be adding your video to the YouTube channel? It is a good one!
PhyllisSmith

PhyllisSmith

2020-06-09 18:06

manager   ~0003575

@Katacuts Just some feedback.
First of all, there is nothing preventing the inclusion of this capability, EXCEPT development time which is in very short supply.
About "productions with hundreds of hours of footage" -- I am not certain that using Cinelerra for this much is really do-able.
About "workflow described here" -- thank you for that reference as it gives me a good idea of expectations.
About "if it is hard to program timecode synchronization? ...Has it been attempted before" -- a lot of work was done initially (like 10 years ago) to synchronize the video and audio, and I believe at the time trying to synchronize via timecode was fraught with varying amounts of missing timestamps. So synchronizing via waveform might be easier.

However, there are actually 2 things that prevent going forward with an effort to provide this feature. As already mentioned lack of development time pretty much stops the whole thing. The second thing is BT 0000433, part 2, for treating stereo (or multiple audio tracks) as a single entity. There needs to be a whole new container designed to fit and then implemented somehow or other, perhaps using program streams. We have been stuck on coming up with a method to do this for a couple of months now although we discuss it almost daily. Maybe if we can get past 0000433, it will be possible to work on this, but I do not see this happening any time soon.
Andrea_Paz

Andrea_Paz

2020-06-09 09:43

updater   ~0003574

Try watching this video too, which shows the multicam workflow from start to finish:

https://drive.google.com/file/d/1Gzf84dcjkFqGtoLbCsePyNakKdxK8lu8/view?usp=sharing

It is not all automated, some steps remain in manual. In the end, we have to manually synchronize the external track to the one created with OpenMixers and replace it with this one.
I understand that this is not suitable for a Pro environment.

However I agree with you: Timecode support, and being able to synchronize audio/video via timecode, would be a nice addition. Let's see if it's possible for Good Guy (maybe the SMTPE timecode which is STD).
Katacuts

Katacuts

2020-06-09 08:58

reporter   ~0003573

Unfortunately I don't see this method workable for productions with hundreds of hours of footage. Also it happens quite often that the camera doesn't record any audio. This looks like a decent way to sync multicam footage but manually marking every in point of the clapperboard or sync points is very time consuming. Time is always a factor so usually the first step is to use the timecode or waveforms to batch sync everything up roughly. Then you go by every clip and check if there was a drift in the timecode or the wavesync didn't work and then adjust manually if needed. But the first batch sync does 90% of the work. The workflow described here is quite close to what I do : https://blog.frame.io/2017/05/15/sync-clips-in-davinci-resolve/ . I was wondering if it is hard to program timecode synchronization? Has it been attempted before or is there some proprietary component needed? I can find no program in the open source video editing world who has it.
PhyllisSmith

PhyllisSmith

2020-06-09 02:38

manager   ~0003572

@Katacuts
As Andrea stated, maybe this will work. An old demo that was created at the time the "align audio" was added in March 2019 is at: https://streamable.com/ln2e5h
(Andrea can make a better one if needed since I am only a tester).
The "align audio" implementation as not as elegant as it could be due to being a late add on instead of designed in. Each mixer is like a program stream and mixers is the method that creates that container.

Let us know if this can work for you.
Andrea_Paz

Andrea_Paz

2020-06-08 15:03

updater   ~0003571

If the video footage has its own audio track then you can synchronize the audio taken externally. See manual 14.1 and 14.1.1 (if you need it I can provide an example video). If instead you only have video tracks without audio plus the external audio track, then the only possibility is to do it manually (better with a clapperboard), since CinGG does not support audio/video sync via timecode.

Issue History

Date Modified Username Field Change
2020-06-08 14:05 Katacuts New Issue
2020-06-08 15:03 Andrea_Paz Note Added: 0003571
2020-06-09 02:38 PhyllisSmith Note Added: 0003572
2020-06-09 02:38 PhyllisSmith Assigned To => PhyllisSmith
2020-06-09 02:38 PhyllisSmith Status new => acknowledged
2020-06-09 08:58 Katacuts Note Added: 0003573
2020-06-09 09:43 Andrea_Paz Note Added: 0003574
2020-06-09 18:06 PhyllisSmith Note Added: 0003575
2020-06-09 18:12 PhyllisSmith Note Added: 0003576
2020-06-15 18:36 Andrew-R Note Added: 0003620
2020-07-04 07:20 Andrew-R Note Added: 0003733
2020-07-06 02:09 PhyllisSmith Note Added: 0003741
2020-07-08 04:43 ferdnyc Note Added: 0003760
2020-07-08 14:00 PhyllisSmith Note Added: 0003763
2020-07-08 14:07 Andrea_Paz Note Added: 0003764
2020-07-10 03:57 PhyllisSmith Note Added: 0003784
2020-07-10 10:43 Andrew-R Note Added: 0003785
2020-07-10 12:30 Andrew-R Note Added: 0003786
2020-07-10 13:20 Andrew-R Note Added: 0003787
2020-07-10 15:12 Andrew-R Note Added: 0003788
2020-07-10 15:27 Andrew-R Note Added: 0003789
2020-07-10 19:11 ferdnyc Note Added: 0003790
2020-07-10 20:53 Andrew-R Note Added: 0003791
2020-07-10 22:08 Andrew-R Note Added: 0003792
2020-07-11 01:41 ferdnyc Note Added: 0003793
2020-07-12 00:34 PhyllisSmith Note Added: 0003794
2020-07-23 02:49 PhyllisSmith File Added: Align_Timecodes.pdf
2020-07-23 02:49 PhyllisSmith Note Added: 0003831
2020-07-23 04:00 Pierre Note Added: 0003832
2020-07-24 09:26 Andrea_Paz Note Added: 0003835
2020-07-24 13:02 ferdnyc Note Added: 0003837
2020-07-24 13:08 ferdnyc Note Added: 0003838
2020-07-26 23:18 PhyllisSmith Note Added: 0003844
2020-07-27 00:03 ferdnyc Note Added: 0003845
2020-07-27 20:52 Pierre Note Added: 0003851