Question about animation and sound..

  • #1, by XerilonFriday, 03. February 2017, 19:07 3 years ago
    I have trouble mathcing speech sound to animated character portraits, not sure if it's me or the computer messing things up. This is what I do:

    I make the animation in Blender at 540x480px by mathicng it to a sound file. I render out png that is converted to WebP, and I know that the animation must play at 67ms speed to match up good. The problem is that longer animations play slower that short animations and the sound gets out of sync even if I preload the animation well before playing. 
    I can work around this by adjusting the speed of the sequence, but I feel that it would just push the problem further down the road. 

    Is this some sort of RAM issue which indicates it would be worse on slower computers?

    Is there any way at all to use video files instead of frames?

    (throwing in a picture of a frame to show what I'm working on)

    Newbie

    37 Posts


  • #2, by afrlmeFriday, 03. February 2017, 19:21 3 years ago
    I think one of new features Simon mentioned is that you can now overlay videos onto scenes, but I think it's only currently possible with Lua script & I don't currently have a clue how it works. Simon said he will upload some scripts/examples when he has the time to sort some out.

    As for your question about timing. Yes I believe it's possible that the time can be affected by the machine running the game as the system loop time is inconsistent. It doesn't loop every x (absolute) ms, but every random x ms instead, so it's possible each new frame isn't being shown bang on 67ms.

    Imperator

    6842 Posts

  • #3, by XerilonFriday, 03. February 2017, 23:02 3 years ago
    Burf, that's sad. 

    But hopefully a feature that can be implemented as soos as Simon has a script sample. I do not have a clue on coding razz

    I know most people just loop some simple mouth movements in these cases, but I want a bit better lipsync and that's why it has to match up.

    Thanks for the reply, I will keep my eyes open for the new feature smile

    Newbie

    37 Posts

  • #4, by afrlmeFriday, 03. February 2017, 23:44 3 years ago
    Burf, that's sad. 

    But hopefully a feature that can be implemented as soos as Simon has a script sample. I do not have a clue on coding razz

    I know most people just loop some simple mouth movements in these cases, but I want a bit better lipsync and that's why it has to match up.

    Thanks for the reply, I will keep my eyes open for the new feature smile
    Based on your first post, I'm guessing what you are currently doing is creating a unique animation for each dialog? Sounds like a lot of work to me.

    You could opt for simpler Hanna Barbera style talk animation. It won't look as nice as using millions of frames to generate all the different kinds of mouth movement, but for 2D I don't think it really matters that much. Basically you create about 6 or 7 or 10 (I forget the actual amount) of mouth shapes & then you use timings to try & pick the closest mouth shape to the word being spoken. You also force idle animation during pauses between words/paragraphs or Oh/Ahh mouth shapes if one of those kind of words were spoken & the time until the next word is not very long.

    http://www.angryanimator.com/word/2010/11/26/tutorial-3-dialog/

    There's another method too, which I call faux lip-syncing. It's kind of like Hanna Barbera method except you have it play the talk animation (set to random frame) then you force idle or oh/ah animation during silent parts of the currently playing speech audio file. It wasn't perfect, but it looked nicer than having character talk animation playing the entire duration of the speech audio file.

    Anyway you go about it though will require a lot of hard work & patience.

    P.S: Daedalic lip sync their games, but they use some kind of application to generate phoneme data then they create Lua tables & use a script/loop or something & probably text ID's to force which talk animation frames should be displayed & when - probably not perfect either due to the time thing we both mentioned, but I just figured I would mention it.

    Imperator

    6842 Posts

  • #5, by XerilonSaturday, 04. February 2017, 12:48 3 years ago
    I know smile I have tried the Hanna Barbera style and it would have worked fine on more stylized or pixelized graphics, but looked very paper doll-ish on 3d characters.

    Scripted phoneme data is cool, but I do actually animate eyebrows, eye direction and eyelids too. It gives off more "emotion" as David Cage would say :p

    It takes up a few frames, but it isn't that much work really. On the 3d characters I have premade poses for various phonetic sounds and facial expressions, and I use a video file of an actor and a sound file from a voice over actor to make the faces on every necessary frame for the animation, rendering out around 15fps. For a couple of speech sentences it doesn't take more than 30-60 minutes to animate and I think that is fast enough.

    This means that I don't put a pose on every frame, and I don't need a frame for each letter spoken, just the key poses. "Are you" needs only "A Y U" to be animated.

    Of course the quality isn't exaclty stunning on the final side of the uncanny valley, but I think it serves it's purpose smile

    Newbie

    37 Posts

  • #6, by afrlmeSaturday, 04. February 2017, 13:36 3 years ago
    I wonder if some lip sync feature could be added to support lip syncing of 3D character models, then again, that wouldn't cover facial expressions. I like the subtle details you are talking about & I think they work well for close-up talking scenes, but general talk animations of the regular sized character in scenes I don't think matters all that much.

    Maybe you could get away with the video thing when Simon posts a script/example for it... though I only imagine that would be useful for when character is in a set position when they are meant to talk otherwise it would be hard to line things up correctly.

    Imperator

    6842 Posts

  • #7, by XerilonSaturday, 04. February 2017, 15:50 3 years ago
    I agree. It would be perfect for these "dialogue portraits" though, and maybe Tvs and such in the background.

    Newbie

    37 Posts

Write post