Mouth animation using Rhubarb Lip Sync

  • #20, by afrlmeWednesday, 07. November 2018, 23:51 4 years ago
    Are you sure you are actually even using the tsv file & not just playing through the animation normally? The name of the tsv file has to be in the same folder as the audio file & needs to contain the exact same filename + extension with .tsv on the end of it, like this...

    example.ogg would example.ogg.tsv
    example.wav would be example.wav.tsv
    foo bar.ogg would be foo bar.ogg.tsv

    & so on.

    Also the example you have shared has a 1 second delay between showing H @ 0.00 to B .


    7211 Posts

  • #21, by chronologicThursday, 08. November 2018, 08:48 4 years ago
    Yes it is using the tsv file.
    The delay is deliberate so that you can easily see the frame that is being added.

    Here is a link where you can download an example mp4 file.
    (It is clearer if you download it from the top right corner rather than stream in the player)

    So here you can see that as the dialogue text starts, there is a random frame being inserted. The tsv has almost 2 seconds of idle, so that the character should only speaks his name.

    0.00 H
    1.51 B

    With the animation set to random, in this case, mouth shape 2 has been inserted.


    25 Posts

  • #22, by afrlmeThursday, 08. November 2018, 11:22 4 years ago
    strange. I've not had any issues like that. I've had some issues with the mouth shapes that Rhubarb has generated based on the audio recordings, but that's to be expected as audio analysis is not perfect software & it probably finds it hard to understand every recording due to how people sound, accents/dialect, background noise, etc.

    Could you send me a ved containing that recording, tsv file & a character to use it with?


    7211 Posts

  • #23, by chronologicThursday, 08. November 2018, 14:29 4 years ago
    Sure, if you message me where to send it, then I will take a look later.

    As I mentioned, you can easily solve this by setting the animation to play backwards, so the additional frame uses the idle one.

    My setting was originally random as I was using random mouth shapes before recording the voices, which is why I saw it.

    Might this also be happening for you, but you don't notice it?

    If the default setting is play forwards and the first mouth shape is an M (similar to idle), you may not be noticing it as much?


    25 Posts

  • #24, by afrlmeThursday, 08. November 2018, 15:08 4 years ago
    No, it doesn't occur for me. I have 9 mouth shapes in the same order as Rhubarb Lip Sync mouth shapes. A-H, X.

    A = 1, B = 2, C = 3, D = 4, E = 5, F = 6, G = 7, H = 8, X = 9, where X is the idle mouth shape. There's no need to create a mouth shape for x as you can just insert the first frame of your idle character animation for the same direction.

    The animation should be played forwards. Should not have the set pause for each frame option ticked. I don't think the pause value matters. The first & last value in the tsv file seems to get ignored. My recommendation is to not mess with the tsv files & just insert an extra frame as frame 8 where the H mouth shape should be so that it correctly uses frame 9 which is the X mouth shape.

    I had a mess around earlier & it seems that if you swap out mouth shapes for frame numbers in the tsv file that it starts from 0 instead of 1, so frame 1 is 0 instead of 1 & frame 9 would be 8 instead of 9 - I don't know why.

    Anyway, feel free to check out the tutorial & lip sync ved demo I created with the help of Sebastian, Ke4, & Joemid.


    7211 Posts

  • #25, by SimonSFriday, 09. November 2018, 00:54 4 years ago
    Small bug, fixed in the next version.

    Thread Captain

    1553 Posts

  • #26, by chronologicFriday, 09. November 2018, 07:37 4 years ago
    Thanks smile


    25 Posts

  • #27, by jonas-common-colorsWednesday, 26. December 2018, 13:37 4 years ago
    Hey guys!
    I've got another problem concerning rhubarb in Visionaire. Within the engine player, it works perfectly, but as soon as I create a game (Windows), it just displays the standard talking animations without any lipsync.

    Can anybody help me?


    29 Posts

  • #28, by afrlmeWednesday, 26. December 2018, 14:55 4 years ago
    Sorry, Simon hasn't yet implemented support for exported games yet. I believe he needs to include some method for the engine to include the tsv files inside of the vis file when the game is exported.

    You might actually be able to include them yourself manually, but I'm not sure that would work as the path would likely be different than the audio file itself as they seem to get included in the exported characters files unless you include characters in the main vis file.


    You can include files in the main vis files, but it doesn't seem to work currently, even when I dump everything inside of the main vis file despite the paths of both the audio file & tsv file matching.


    7211 Posts

  • #29, by daniel-wolfThursday, 07. February 2019, 22:11 4 years ago
    @AFRLme: Has there been any progress? I imagine it must be a bit of a letdown to have a lip-sync'ed game and not be able to publish it.

    On a different note, I just released Rhubarb Lip Sync 1.9. This is the first version that supports non-English recordings. So if your game is in a language other than English, be sure to check it out!

    I also created a video showcasing the new multi-language support:


    10 Posts

  • #30, by stothewThursday, 07. February 2019, 22:26 4 years ago
    Awesome! Thank you so much!

    Forum Fan

    127 Posts