Dynamic Animations (Lip Sync)

I'm looking to add proper Lip Synching character animations using the 8 basic mouth positions (Phonemes).
Therefore, I'd like to build up an animation dynamically at runtime using multiple frames from the Phonemes animation. The resulting animation will be the lip movements.

Example,

Proper lip syncing is not that easy to do, especially if you are planning on creating your game in multiple languages. I think the Hanna Barbera mouth animation approach is the simplest approach as it uses about 7 different mouth shapes, as opposed to one for each different kind of pronounced sound.

http://sunewatts.dk/lipsync/lipsync/article_02.php

http://www.angryanimator.com/word/2010/11/26/tutorial-3-dialog/

In the game I'm working on we created faux lip syncing, which involved preventing the characters mouth from moving during parts in the associated audio file where the character wasn't actually talking. Sometimes I forced it to play a specific mouth shape for words that began with certain sounds, but that wasn't too often.

My method involved opening up the speech files in audacity & then splitting up the recording into chunks based on spoken parts & silence. I then highlighted each of these parts to get their duration in milliseconds which I made a note of, before finally getting the duration of the entire audio file. I used both of these values to control the pauses & length of the dialog. I had to create the display texts as background display texts so that I could update the animation frames at the same time. As a result though that killed left click skipping of the texts, so I ended up wrapping each display text in cutscene action parts so that they could be skipped via the ESC key.

Sorry for the long post. Anyway... the best approach to what you want to do is to create loads of Lua tables. One per display text for each language. After that you will need a loop of some kind to iterate through the pauses & whatever else is needed based on the active text & language.

I'm not really sure how to tell you how to go about it. Daedalic used a similar method to what I just mentioned, but it's pretty complicated. They also (apparently) have a tool for calculating the mouth shapes based on the audio file. SimonS gave me a little tool for doing the same thing a little while back, but I found that it was often very inaccurate due to recording quality & the accent / tone of the person doing the talking.

Thanks for the reply.
So, i'll need a loop that chooses which animations to play.

1) Lua cannot wait for the animation to finish (Can it?), so I will need to watch the ActiveAnimations right?
2) Do character animations appear within ActiveAnimations?

1. yes there's 2 event flags in the registerEventHandler() function for listening out for animation started & animation stopped. I think Daedalic utilized that in some way or another.

2. also yes.

To control the character animation frames though you will need to display text as background display text or background narration text. The latter if you plan on multiple animations instead of forcing the animation frames. Everything else is all down to timing.

For selecting the correct animations from a Lua table you could use the registerEventHandler() function with the textstarted & textstopped event flags to check for when a text is started or ends & also to return the id associated with said text. If you figure out what the texts id value is then you could set that as each texts relevant index value inside of the Lua tables. So it would obtain the id value then start the loop & iterate through the table entry associated with that id value.

P.S: this is mostly theoretical. I've not needed to create actual lip sync for anything yet.

OK, thanks.
Why the character/background text though? Is character text not possible during animations? I thought i'd seen characters talking and walking before?

The background text is so that you can still control your characters &/or add other action parts afterwards (in the same action block) without having to wait until the text has finished.

I suppose if you were calling some called by other actions directly via Lua script then the background text wouldn't matter so much. The problem then though would be that if the player left clicked to skip the text, the actions / loop that was being used to control the characters mouth movements / facial expressions (whatever) would still carry on going, well unless you used the textstopped event flag we mentioned earlier - which you could use to quit the looping action block that is iterating through the table values associated with the text that just finished.

Like I said, it's complicated & I'm sure there are probably multiple approaches that could be taken to sort out lip syncing.

This has given me plenty to get on with, thanks for the help.

I've made a start, but i'm finding that you cannot call startAnimation from within the lua function.
Is this a Bug?

Note: The startAnimation(anim) code works fine called from elsewhere.



"Anim Started" is output to log

Why are you calling it inside of the animation stopped event flag & why have you wrapped it inside of an if query?

If queries when you call something with actually adding the query operators only really check if something exists which will return true or false.

Sorry, I'm not able to wrap my head around what the function is supposed to be doing.

I assume cObj should be returning the animation link that was just ended, like the text event flags return the links of the texts that just started or ended. Other than that I've no idea what you are trying to do with it. Then again I do have a bit of a migraine, so that's not helping matters much for me.

The onAnimFinished function gets called every time an animation stops, the if statement is therefore checking for the animation that i'm waiting for (This will eventually be a global variable).
So what i want to do is watch for my animation to finish and then kick off the next animation in my required sequence of mouth movements.

Example sequence:
Play Anim1 (Open mouth)
Play Anim2 (Closed mouth)
Play Anim3 (A different mouth shape)
etc...

But, the startAnimation command does not actually start the next animation.

Try creating a new function that contains the startAnimation() function.

By the way, did you know that you can update the global pause value between frames during runtime with Lua script for each animation? It would essentially give you better control on how long an animation should play for.

My suggestion though would be to force the frames because I'm not sure the talk animations unless you create them as manually started character animations will ever actually finish, as they are set to infinite loop while ever a character is talking. Forcing the frames & global pause value for the talk animation is easier.

... & if you wanted to save time then write a work-flow function or use the one I added to the wiki ages ago.