View Issue Details
|ID||Project||Category||View Status||Date Submitted||Last Update|
|0000160||Cinelerra-GG||[All Projects] Feature||public||2019-03-10 21:25||2019-06-06 03:22|
|Priority||normal||Severity||minor||Reproducibility||have not tried|
|Target Version||Fixed in Version|
|Summary||0000160: Multi camera sync based on audio|
|Description||Feature request: Auto alignment for multicam clips based on in camera audio|
Synchronizing multiple camera videos based on their audio tracks and external audio tracks is task which has to be done every larger multicam project.
The DSPing doesn't look that hard, but everything else might be. Best thing I was
able to find is: https://github.com/bbc/audio-offset-finder which implements the algorithm quite nicely.
Nice features would be possibility hand hold the plugin giving hint where to start looking for syncing place and another low hanging fruit would be distance compensation (about 1 frame for every 12 meters when 25 fps. higher framerates would need it earlier.)
|Tags||No tags attached.|
Some status that does not seem to have been reported on:
1) "My search selection extended over end of one clip, but the hang could happen in 23% or in 80 % of processing and the end spot was more like 50 % so my first thought was deadlock or race condition."
I could not get a hang so we were hoping for a set of steps to reproduce the problem. Or a Ctrl-c dump as root with gdb.
2) "Another minor feature request: syncing to audio only tracks would be nice as it could come from external recorder."
Did not add this feature and I have yet to try to see if maybe there is a way to already do this.
3) "I would like to insert these mixer clips in the timeline at a later point in time and not always at the beginning."
An option was added in the Resource window called "Insert Mixers". I thought that that was reported, but maybe in email.
4) Somewhere Pierre stated "I will wait for better documentation on usage" as it did not line up well. I still have to work on that because I believe it can do a very good job, given the right starting point and that is not very clearly documented how to do this.
So I do not want to close this because I think the 3 unresolved items can be resolved in time.
|My experience is that even commercial solutions like DaVinci Resolve reach their limits when the audio quality is not optimal. In my last customer project (with five cameras), I also had to manually adjust everything in Resolve. The sound needs to be clear and distinct. If the audio recording is even minimally overdriven or set too low, it leads to these known problems. For these reasons I prefer to work with TimeCode sync.|
I'm not sure where it's at....
In the case of the project I was trying to synchronize automatically, it didn't work well. I had to decide to synchronize all the clips manually, often with difficulty, by comparing the movements in the images and the sounds common to the different sound tracks.
It was too extreme a case, I guess; four cameras not all running at the same time, few dialogues, repetitive and languorous background music... the system did not automatically find its way in.
Once synchronized manually, here is what the timeline gave:
|What's the status?|
|I just tried that new feature with 3 cams. Really powerful!|
|Just an update -- both Pierre and Sam's information is being worked on for incorporation. It is more involved than expected so will take some time to code.|
I have attached a picture showing what I mean by that.
Mixer-issue_01.png (107,617 bytes)
Mixer-issue_01.png (107,617 bytes)
|Pierre: got it. We will be looking at it tonight yet.|
My camera-4 (mixer-4) was not running continuously; I have to synchronize each of its clips individually with the continuous sound of camera-1.
On my current timeline everything that is present has already been synchronized (manually). If I try to automatically synchronize a new clip from camera-4 to its track, "Mixer Align by Audio" works... it correctly positions the new clip from camera-4, but at the same time it moves its entire track from the beginning of the timeline; desynchronizing all the previous clips from the mixer-4 timeline, clips that I had already synchronized.
Phyllis, I send by private email (the faces of the participants are recognizable) the image capture of my timeline so that you understand the problem.
Pierre: We found and fixed 1 SEGV, but I am not absolutely sure it is the only 1. GG tested a video with audio mixer that was over 1 hour long with no problems. We also tested some very long, past the end of the timeline, selections with no problems. However, our test media is not very good, very long, or very realistic with our inexpensive cameras that only record a few minutes.
There are newer test builds now available at:
Sam, about "For larger video projects... I would like to insert these mixer clips in the timeline ...not always at the beginning" -- mixers are for tracks, they are not edits. But I am not quite sure we really understand what you are trying to achieve. Do you mean that you already have clips/edits on several tracks in the beginning, you want to leave them there, and add mixer media at an insertion point on the same tracks after your clips? or do you mean you want to add new tracks for the mixers? ??
On my first try, I had it crash... I was hard on it...
4 cameras from a 1-hour show. I chose a short segment of my main sound track and applied the analysis to all the timeline I had already manually synchronized halfway through...
Cin-GG to after completing a good part of the calculation... has disappeared.
pierre@i7-3770k ~ $ /home/stone/Cinelerra-GG_5.1/cin
Cinelerra Infinity - built: Tue 23 2019 20:40:03
(c) 2006-2018 Heroine Virtual Ltd. by Adam Williams
(c) 2007-2018 cin5 derivative by W.P. Morrow aka goodguy
Cinelerra is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. There is absolutely no warranty for Cinelerra.
RenderFarmClient::main_loop: client started
master audio track length > 60 seconds
Error in match render.
** segv at 0xbb9f40 in pid 5098, tid 7716
writing debug data to /tmp/cinelerra_5098.dmp
** dump complete
Segmentation error (core dumped)
pierre@i7-3770k ~ $
cinelerra_5098.dmp (244,400 bytes)
|The mixer clips are always inserted at the beginning of the tracks. For larger video projects, however, I would like to insert these mixer clips in the timeline at a later point in time and not always at the beginning. It would be better to insert the clips at the playback position, or from the inpoint, so that I can decide for myself when I want to use the clips.|
Vuokko: thank you for testing. It is appreciated and very helpful.
You can "manually" mixer audio only now but GG will look into adding the automatic method. The manual method is like the recover method that I will be adding to the documentation when I get time but it is tedious until you try it a few times and get good at it.
The problem of the Progress Bar hanging at 23% or 80% is not reproducible here with our test cases. But GG would very much like to be able to see what the problem you encounter is. Like you said, it does sound like a lock problem, probably with threads -- the rendering progress is multi-threaded so it could be 1 thread gets done and the other one never finishes to give the lock back (possibly a bad spot in the media? possibly a coding mistake? ??).
In order to fix the error, we need all of the information you can provide if you can reproduce it. Probably the most helpful and least work for you would be to:
1 find a case that always hangs,
2 start cinelerra from a terminal window (not the application icon)
3 Ctrl-C once it hangs to get a dump and send us that and whatever messages it prints on the terminal window you started from
4 start cinelerra again, setup the same scenario as you did for that hanging case
5 BUT before you hit "Match" in the Align audio dialog box, use the File->Export Project menu to save all of the files in a single directory
6 tar that single directory up and upload somewhere and let us know where to find it so we can create the hang using your data
Yes, it is work for you but it will save you time in the future to not have something hang if we can find the error. There are other possibilities yet for debugging if this does not produce results.
Tested last night. When the alignment work, it's great. I haven't taken the latest fixes but sometimes the matching hangs. I'll test the new build later today.
My search selection extended over end of one clip, but the hang could happen in 23% or in 80 % of processing and the end spot was more like 50 % so my first thought was deadlock or race condition.
Another minor feature request: syncing to audio only tracks would be nice as it could come from external recorder.
We found a bug and are working on it. Not sure cause, but this case had 3 cameras and some problem with the last one. It might be because one of the 3 cameras only has 1 audio track and the other 2 had 2 audio tracks.
Bug was found and fixed and checked into GIT.
Re-built the following:
Unbelievable! I should know about GG's skills by now, but he always manages to surprise me. Thank you. Phyllis can you put the video on the forum? (and thanks to you too).
PS: my test is OK.
uokko: OK, Arch static tar is ready at:
BTW: we enjoyed testing with the military watch videos.
|Arch Linux package would be great for me.|
Pierre: we will make a Mint 18 for you to download. Give GG about 1 hour for him to build and upload.
It is now available.
Ohh...that I wish this new function had been ready two weeks ago....
Since then I frequently work on two shoots of long shows (one 3h30 and the other 1h00). I had shot the first one with three cameras and the second one with four cameras. Some of them were not running continuously.
I have already synchronized (to the precise frame) the beginning of these shoots but not everything. I will be happy to test the new feature to see how it manages to complete the remaining synchronizations and if it gets to the right frame each time.
Wow, over a thousand lines of new code! The first quick test was positive! It works fine for me. I'm still amazed how fast and effective GG can write such a new complex feature code. Also great demo video and description.
Thanks to both of you!
The latest GIT checkin now includes this feature. A demo is at:
A ubuntu16 build is being done and will be available in an hour or 2 dated today at:
Attached is a pdf file on how to use.
@vuokko - let me know if you need a static tar build for a specific operating system. We only build tars for testing.
Does anyone else need a tar for testing? Glad to create one, if you will test!
multi-cam_align_audio.pdf (326,517 bytes)
|Thank you Andrea and Vuokko! GG has been working on the design for the last couple of days so your input/samples is much appreciated.|
|And Andrea_Paz has right actions in screenshots. My idea of user interface would be something like wizard where would be tracks and those can be selected. Then one of those could be selected to be stationary and hitting syncing button would move the rest to sync with each other. This is something which would need to be done once when putting the clips to timeline.|
I took with my crappy phone and with DSLR a video of my watch and said something. So they both have same loysy audio. I think it is possible see the movement of the second hand to get visual proof of syncing.
17 and 54 MB files: from google drive: https://drive.google.com/drive/folders/1n2VL3XRhAZj_YIM-iSrREruCONNbtzDF?usp=sharing
I'll try to give my opinion.
Multicam footage of a single event can have various shots starting and ending at different times. So the footage is different, but the connected audio can be used to automatically synchronize the clips on the timeline. Another way is to use the timecode, but only if it was set during shooting. Instead the audiosync does it in post-production trying to find and align automatically the waveforms of the various shots.
https://www.youtube.com/watch?v=kmFmyXWaQHM (~ 1 min).
I did a photomontage to understand how multicam works in CinGG and how it works automatically in other video editors. Image01 = how CinGG works; Image02 = how Final Cut works.
It's probably too big a job to do, which involves merging the audio and video tracks in a single track and not always having the tracks separate.
Screenshot01.jpg (179,415 bytes)
Screenshot01.jpg (179,415 bytes)
Screenshot02.jpg (193,828 bytes)
Screenshot02.jpg (193,828 bytes)
|Not sure what this would entail. PLEASE send a specific set of small test files so GG can look at this to determine what the input would look like in the unsynchronized state. As a programmer instead of an Audio/Video expert, we need to get a better understanding.|
|I like the idea very much, especially with several cameras it makes sense and would be a clear relief. Maybe the Offset Finder can be used, then the integration might be easier. +1|
|2019-03-10 21:25||vuokko||New Issue|
|2019-03-10 21:33||Sam||Note Added: 0001166|
|2019-03-12 01:36||PhyllisSmith||Note Added: 0001183|
|2019-03-20 07:34||Andrea_Paz||File Added: Screenshot01.jpg|
|2019-03-20 07:34||Andrea_Paz||File Added: Screenshot02.jpg|
|2019-03-20 07:34||Andrea_Paz||Note Added: 0001235|
|2019-03-20 16:34||vuokko||Note Added: 0001236|
|2019-03-20 16:52||vuokko||Note Added: 0001237|
|2019-03-20 17:20||PhyllisSmith||Note Added: 0001238|
|2019-03-23 18:39||PhyllisSmith||File Added: multi-cam_align_audio.pdf|
|2019-03-23 18:39||PhyllisSmith||Note Added: 0001242|
|2019-03-23 19:10||Sam||Note Added: 0001243|
|2019-03-23 19:29||Pierre||Note Added: 0001244|
|2019-03-23 19:39||PhyllisSmith||Note Edited: 0001242||View Revisions|
|2019-03-23 19:40||PhyllisSmith||Note Added: 0001245|
|2019-03-23 20:15||vuokko||Note Added: 0001246|
|2019-03-23 20:29||PhyllisSmith||Note Edited: 0001245||View Revisions|
|2019-03-23 20:50||PhyllisSmith||Note Added: 0001247|
|2019-03-23 22:29||Andrea_Paz||Note Added: 0001248|
|2019-03-24 00:02||PhyllisSmith||Note Added: 0001249|
|2019-03-24 00:17||PhyllisSmith||Note Edited: 0001249||View Revisions|
|2019-03-24 03:11||PhyllisSmith||Note Edited: 0001249||View Revisions|
|2019-03-24 10:25||vuokko||Note Added: 0001250|
|2019-03-24 16:37||PhyllisSmith||Note Added: 0001251|
|2019-03-24 20:21||Sam||Note Added: 0001252|
|2019-03-24 21:01||Pierre||File Added: cinelerra_5098.dmp|
|2019-03-24 21:01||Pierre||Note Added: 0001253|
|2019-03-25 00:59||PhyllisSmith||Note Added: 0001254|
|2019-03-25 02:04||Pierre||Note Added: 0001255|
|2019-03-25 02:34||PhyllisSmith||Note Added: 0001256|
|2019-03-25 07:58||Sam||File Added: Mixer-issue_01.png|
|2019-03-25 07:58||Sam||Note Added: 0001257|
|2019-03-27 02:44||PhyllisSmith||Note Added: 0001259|
|2019-03-27 13:05||IgorBeg||Note Added: 0001260|
|2019-06-05 21:11||Olaf||Note Added: 0001665|
|2019-06-05 21:39||Pierre||Note Added: 0001667|
|2019-06-05 21:53||Sam||Note Added: 0001668|
|2019-06-06 03:21||PhyllisSmith||Note Added: 0001672|
|2019-06-06 03:22||PhyllisSmith||Assigned To||=> goodguy|
|2019-06-06 03:22||PhyllisSmith||Status||new => assigned|
|2019-06-06 03:22||PhyllisSmith||Status||assigned => acknowledged|