View Issue Details

IDProjectCategoryView StatusLast Update
0000160Cinelerra-GG[All Projects] Featurepublic2019-06-06 03:22
Reportervuokko Assigned Togoodguy  
PrioritynormalSeverityminorReproducibilityhave not tried
Status acknowledgedResolutionopen 
Product Version 
Target VersionFixed in Version 
Summary0000160: Multi camera sync based on audio
DescriptionFeature request: Auto alignment for multicam clips based on in camera audio

Synchronizing multiple camera videos based on their audio tracks and external audio tracks is task which has to be done every larger multicam project.

The DSPing doesn't look that hard, but everything else might be. Best thing I was
able to find is: https://github.com/bbc/audio-offset-finder which implements the algorithm quite nicely.

Nice features would be possibility hand hold the plugin giving hint where to start looking for syncing place and another low hanging fruit would be distance compensation (about 1 frame for every 12 meters when 25 fps. higher framerates would need it earlier.)
TagsNo tags attached.

Activities

PhyllisSmith

PhyllisSmith

2019-06-06 03:21

manager   ~0001672

Some status that does not seem to have been reported on:
1) "My search selection extended over end of one clip, but the hang could happen in 23% or in 80 % of processing and the end spot was more like 50 % so my first thought was deadlock or race condition."
I could not get a hang so we were hoping for a set of steps to reproduce the problem. Or a Ctrl-c dump as root with gdb.
2) "Another minor feature request: syncing to audio only tracks would be nice as it could come from external recorder."
Did not add this feature and I have yet to try to see if maybe there is a way to already do this.
3) "I would like to insert these mixer clips in the timeline at a later point in time and not always at the beginning."
An option was added in the Resource window called "Insert Mixers". I thought that that was reported, but maybe in email.
4) Somewhere Pierre stated "I will wait for better documentation on usage" as it did not line up well. I still have to work on that because I believe it can do a very good job, given the right starting point and that is not very clearly documented how to do this.

So I do not want to close this because I think the 3 unresolved items can be resolved in time.
Sam

Sam

2019-06-05 21:53

administrator   ~0001668

My experience is that even commercial solutions like DaVinci Resolve reach their limits when the audio quality is not optimal. In my last customer project (with five cameras), I also had to manually adjust everything in Resolve. The sound needs to be clear and distinct. If the audio recording is even minimally overdriven or set too low, it leads to these known problems. For these reasons I prefer to work with TimeCode sync.
Pierre

Pierre

2019-06-05 21:39

updater   ~0001667

I'm not sure where it's at....

In the case of the project I was trying to synchronize automatically, it didn't work well. I had to decide to synchronize all the clips manually, often with difficulty, by comparing the movements in the images and the sounds common to the different sound tracks.

It was too extreme a case, I guess; four cameras not all running at the same time, few dialogues, repetitive and languorous background music... the system did not automatically find its way in.

Once synchronized manually, here is what the timeline gave:
https://my.pcloud.com/publink/show?code=XZqOiA7ZaamIw1RQTX0IgMGhvtQIlfrVrK6y
Olaf

Olaf

2019-06-05 21:11

reporter   ~0001665

What's the status?
IgorBeg

IgorBeg

2019-03-27 13:05

reporter   ~0001260

I just tried that new feature with 3 cams. Really powerful!
PhyllisSmith

PhyllisSmith

2019-03-27 02:44

manager   ~0001259

Just an update -- both Pierre and Sam's information is being worked on for incorporation. It is more involved than expected so will take some time to code.
Sam

Sam

2019-03-25 07:58

administrator   ~0001257

I have attached a picture showing what I mean by that.

Mixer-issue_01.png (107,617 bytes)
Mixer-issue_01.png (107,617 bytes)
PhyllisSmith

PhyllisSmith

2019-03-25 02:34

manager   ~0001256

Pierre: got it. We will be looking at it tonight yet.
Pierre

Pierre

2019-03-25 02:04

updater   ~0001255

My camera-4 (mixer-4) was not running continuously; I have to synchronize each of its clips individually with the continuous sound of camera-1.

On my current timeline everything that is present has already been synchronized (manually). If I try to automatically synchronize a new clip from camera-4 to its track, "Mixer Align by Audio" works... it correctly positions the new clip from camera-4, but at the same time it moves its entire track from the beginning of the timeline; desynchronizing all the previous clips from the mixer-4 timeline, clips that I had already synchronized.

Phyllis, I send by private email (the faces of the participants are recognizable) the image capture of my timeline so that you understand the problem.
PhyllisSmith

PhyllisSmith

2019-03-25 00:59

manager   ~0001254

Pierre: We found and fixed 1 SEGV, but I am not absolutely sure it is the only 1. GG tested a video with audio mixer that was over 1 hour long with no problems. We also tested some very long, past the end of the timeline, selections with no problems. However, our test media is not very good, very long, or very realistic with our inexpensive cameras that only record a few minutes.

There are newer test builds now available at:
https://www.cinelerra-gg.org/download/testing/cinelerra-5.1-arch-x86_64-static.txz
https://www.cinelerra-gg.org/download/testing/cinelerra-5.1-mint18-x86_64-static.txz
https://www.cinelerra-gg.org/download/testing/cinelerra-5.1-ub16-x86_64-static.txz

Sam, about "For larger video projects... I would like to insert these mixer clips in the timeline ...not always at the beginning" -- mixers are for tracks, they are not edits. But I am not quite sure we really understand what you are trying to achieve. Do you mean that you already have clips/edits on several tracks in the beginning, you want to leave them there, and add mixer media at an insertion point on the same tracks after your clips? or do you mean you want to add new tracks for the mixers? ??
Pierre

Pierre

2019-03-24 21:01

updater   ~0001253

On my first try, I had it crash... I was hard on it...
4 cameras from a 1-hour show. I chose a short segment of my main sound track and applied the analysis to all the timeline I had already manually synchronized halfway through...

Cin-GG to after completing a good part of the calculation... has disappeared.


pierre@i7-3770k ~ $ /home/stone/Cinelerra-GG_5.1/cin
Cinelerra Infinity - built: Tue 23 2019 20:40:03
git://git.cinelerra-gg.org/goodguy/cinelerra.git
(c) 2006-2018 Heroine Virtual Ltd. by Adam Williams
(c) 2007-2018 cin5 derivative by W.P. Morrow aka goodguy
Cinelerra is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. There is absolutely no warranty for Cinelerra.

RenderFarmClient::main_loop: client started
void MixersAlign::load_master_audio(Track*):
master audio track length > 60 seconds
void MixersAlign::update_match():
Error in match render.
** segv at 0xbb9f40 in pid 5098, tid 7716
writing debug data to /tmp/cinelerra_5098.dmp
lock_items: 48
lock_frees: 11
** dump complete
Segmentation error (core dumped)
pierre@i7-3770k ~ $

cinelerra_5098.dmp (244,400 bytes)
Sam

Sam

2019-03-24 20:21

administrator   ~0001252

The mixer clips are always inserted at the beginning of the tracks. For larger video projects, however, I would like to insert these mixer clips in the timeline at a later point in time and not always at the beginning. It would be better to insert the clips at the playback position, or from the inpoint, so that I can decide for myself when I want to use the clips.
PhyllisSmith

PhyllisSmith

2019-03-24 16:37

manager   ~0001251

Vuokko: thank you for testing. It is appreciated and very helpful.
You can "manually" mixer audio only now but GG will look into adding the automatic method. The manual method is like the recover method that I will be adding to the documentation when I get time but it is tedious until you try it a few times and get good at it.

The problem of the Progress Bar hanging at 23% or 80% is not reproducible here with our test cases. But GG would very much like to be able to see what the problem you encounter is. Like you said, it does sound like a lock problem, probably with threads -- the rendering progress is multi-threaded so it could be 1 thread gets done and the other one never finishes to give the lock back (possibly a bad spot in the media? possibly a coding mistake? ??).

In order to fix the error, we need all of the information you can provide if you can reproduce it. Probably the most helpful and least work for you would be to:
1 find a case that always hangs,
2 start cinelerra from a terminal window (not the application icon)
3 Ctrl-C once it hangs to get a dump and send us that and whatever messages it prints on the terminal window you started from
4 start cinelerra again, setup the same scenario as you did for that hanging case
5 BUT before you hit "Match" in the Align audio dialog box, use the File->Export Project menu to save all of the files in a single directory
6 tar that single directory up and upload somewhere and let us know where to find it so we can create the hang using your data

Yes, it is work for you but it will save you time in the future to not have something hang if we can find the error. There are other possibilities yet for debugging if this does not produce results.
vuokko

vuokko

2019-03-24 10:25

reporter   ~0001250

Tested last night. When the alignment work, it's great. I haven't taken the latest fixes but sometimes the matching hangs. I'll test the new build later today.

My search selection extended over end of one clip, but the hang could happen in 23% or in 80 % of processing and the end spot was more like 50 % so my first thought was deadlock or race condition.

Another minor feature request: syncing to audio only tracks would be nice as it could come from external recorder.
PhyllisSmith

PhyllisSmith

2019-03-24 00:02

manager   ~0001249

Last edited: 2019-03-24 03:11

View 3 revisions

We found a bug and are working on it. Not sure cause, but this case had 3 cameras and some problem with the last one. It might be because one of the 3 cameras only has 1 audio track and the other 2 had 2 audio tracks.

Bug was found and fixed and checked into GIT.
Re-built the following:

https://www.cinelerra-gg.org/download/testing/cinelerra-5.1-arch-x86_64-static.txz
https://www.cinelerra-gg.org/download/testing/cinelerra-5.1-mint18-x86_64-static.txz
https://www.cinelerra-gg.org/download/testing/cinelerra-5.1-ub16-x86_64-static.txz

Andrea_Paz

Andrea_Paz

2019-03-23 22:29

manager   ~0001248

Unbelievable! I should know about GG's skills by now, but he always manages to surprise me. Thank you. Phyllis can you put the video on the forum? (and thanks to you too).
PS: my test is OK.
PhyllisSmith

PhyllisSmith

2019-03-23 20:50

manager   ~0001247

uokko: OK, Arch static tar is ready at:
https://www.cinelerra-gg.org/download/testing/cinelerra-5.1-arch-x86_64-static.txz

BTW: we enjoyed testing with the military watch videos.
vuokko

vuokko

2019-03-23 20:15

reporter   ~0001246

Arch Linux package would be great for me.
PhyllisSmith

PhyllisSmith

2019-03-23 19:40

manager   ~0001245

Last edited: 2019-03-23 20:29

View 2 revisions

Pierre: we will make a Mint 18 for you to download. Give GG about 1 hour for him to build and upload.

It is now available.
https://www.cinelerra-gg.org/download/testing/cinelerra-5.1-mint18-x86_64-static.txz

Pierre

Pierre

2019-03-23 19:29

updater   ~0001244

Ohh...that I wish this new function had been ready two weeks ago....

Since then I frequently work on two shoots of long shows (one 3h30 and the other 1h00). I had shot the first one with three cameras and the second one with four cameras. Some of them were not running continuously.

I have already synchronized (to the precise frame) the beginning of these shoots but not everything. I will be happy to test the new feature to see how it manages to complete the remaining synchronizations and if it gets to the right frame each time.
Sam

Sam

2019-03-23 19:10

administrator   ~0001243

Wow, over a thousand lines of new code! The first quick test was positive! It works fine for me. I'm still amazed how fast and effective GG can write such a new complex feature code. Also great demo video and description.
Thanks to both of you!
PhyllisSmith

PhyllisSmith

2019-03-23 18:39

manager   ~0001242

Last edited: 2019-03-23 19:39

View 2 revisions

The latest GIT checkin now includes this feature. A demo is at:
    https://streamable.com/l61qt
A ubuntu16 build is being done and will be available in an hour or 2 dated today at:
     https://www.cinelerra-gg.org/download/testing/cinelerra-5.1-ub16-x86_64-static.txz

Attached is a pdf file on how to use.

@vuokko - let me know if you need a static tar build for a specific operating system. We only build tars for testing.

Does anyone else need a tar for testing? Glad to create one, if you will test!



multi-cam_align_audio.pdf (326,517 bytes)
PhyllisSmith

PhyllisSmith

2019-03-20 17:20

manager   ~0001238

Thank you Andrea and Vuokko! GG has been working on the design for the last couple of days so your input/samples is much appreciated.
vuokko

vuokko

2019-03-20 16:52

reporter   ~0001237

And Andrea_Paz has right actions in screenshots. My idea of user interface would be something like wizard where would be tracks and those can be selected. Then one of those could be selected to be stationary and hitting syncing button would move the rest to sync with each other. This is something which would need to be done once when putting the clips to timeline.
vuokko

vuokko

2019-03-20 16:34

reporter   ~0001236

I took with my crappy phone and with DSLR a video of my watch and said something. So they both have same loysy audio. I think it is possible see the movement of the second hand to get visual proof of syncing.

17 and 54 MB files: from google drive: https://drive.google.com/drive/folders/1n2VL3XRhAZj_YIM-iSrREruCONNbtzDF?usp=sharing
Andrea_Paz

Andrea_Paz

2019-03-20 07:34

manager   ~0001235

I'll try to give my opinion.
Multicam footage of a single event can have various shots starting and ending at different times. So the footage is different, but the connected audio can be used to automatically synchronize the clips on the timeline. Another way is to use the timecode, but only if it was set during shooting. Instead the audiosync does it in post-production trying to find and align automatically the waveforms of the various shots.

https://www.youtube.com/watch?v=kmFmyXWaQHM (~ 1 min).

I did a photomontage to understand how multicam works in CinGG and how it works automatically in other video editors. Image01 = how CinGG works; Image02 = how Final Cut works.

It's probably too big a job to do, which involves merging the audio and video tracks in a single track and not always having the tracks separate.

Screenshot01.jpg (179,415 bytes)
Screenshot01.jpg (179,415 bytes)
Screenshot02.jpg (193,828 bytes)
Screenshot02.jpg (193,828 bytes)
PhyllisSmith

PhyllisSmith

2019-03-12 01:36

manager   ~0001183

Not sure what this would entail. PLEASE send a specific set of small test files so GG can look at this to determine what the input would look like in the unsynchronized state. As a programmer instead of an Audio/Video expert, we need to get a better understanding.
Sam

Sam

2019-03-10 21:33

administrator   ~0001166

I like the idea very much, especially with several cameras it makes sense and would be a clear relief. Maybe the Offset Finder can be used, then the integration might be easier. +1

Issue History

Date Modified Username Field Change
2019-03-10 21:25 vuokko New Issue
2019-03-10 21:33 Sam Note Added: 0001166
2019-03-12 01:36 PhyllisSmith Note Added: 0001183
2019-03-20 07:34 Andrea_Paz File Added: Screenshot01.jpg
2019-03-20 07:34 Andrea_Paz File Added: Screenshot02.jpg
2019-03-20 07:34 Andrea_Paz Note Added: 0001235
2019-03-20 16:34 vuokko Note Added: 0001236
2019-03-20 16:52 vuokko Note Added: 0001237
2019-03-20 17:20 PhyllisSmith Note Added: 0001238
2019-03-23 18:39 PhyllisSmith File Added: multi-cam_align_audio.pdf
2019-03-23 18:39 PhyllisSmith Note Added: 0001242
2019-03-23 19:10 Sam Note Added: 0001243
2019-03-23 19:29 Pierre Note Added: 0001244
2019-03-23 19:39 PhyllisSmith Note Edited: 0001242 View Revisions
2019-03-23 19:40 PhyllisSmith Note Added: 0001245
2019-03-23 20:15 vuokko Note Added: 0001246
2019-03-23 20:29 PhyllisSmith Note Edited: 0001245 View Revisions
2019-03-23 20:50 PhyllisSmith Note Added: 0001247
2019-03-23 22:29 Andrea_Paz Note Added: 0001248
2019-03-24 00:02 PhyllisSmith Note Added: 0001249
2019-03-24 00:17 PhyllisSmith Note Edited: 0001249 View Revisions
2019-03-24 03:11 PhyllisSmith Note Edited: 0001249 View Revisions
2019-03-24 10:25 vuokko Note Added: 0001250
2019-03-24 16:37 PhyllisSmith Note Added: 0001251
2019-03-24 20:21 Sam Note Added: 0001252
2019-03-24 21:01 Pierre File Added: cinelerra_5098.dmp
2019-03-24 21:01 Pierre Note Added: 0001253
2019-03-25 00:59 PhyllisSmith Note Added: 0001254
2019-03-25 02:04 Pierre Note Added: 0001255
2019-03-25 02:34 PhyllisSmith Note Added: 0001256
2019-03-25 07:58 Sam File Added: Mixer-issue_01.png
2019-03-25 07:58 Sam Note Added: 0001257
2019-03-27 02:44 PhyllisSmith Note Added: 0001259
2019-03-27 13:05 IgorBeg Note Added: 0001260
2019-06-05 21:11 Olaf Note Added: 0001665
2019-06-05 21:39 Pierre Note Added: 0001667
2019-06-05 21:53 Sam Note Added: 0001668
2019-06-06 03:21 PhyllisSmith Note Added: 0001672
2019-06-06 03:22 PhyllisSmith Assigned To => goodguy
2019-06-06 03:22 PhyllisSmith Status new => assigned
2019-06-06 03:22 PhyllisSmith Status assigned => acknowledged