Timing audio to animation inside VS - how would you approach it?

  • #1, by F_KalSunday, 09. April 2017, 00:22 3 years ago
    For a change, a non-lua question (@AFRLme can rest for once) :-P

    It's more of a "what has worked for you" kind of a question - I finished drawing a few frames for an animated sequence (realtime, not rendered), and thought I should dump them inside VS and reconstruct the sequence. 

    But then I realized that timing the frames to the music is a daunting task that I'm not sure how to approach efficiently. I've done it with video in the past, but in VS you don't have a timeline where you can preview the result . How would you people tackle the whole audio to character animation-timing thing? 

    Here's an example
    you have a 10-30sec sequence that you wish to create. The music will be rather theatrical like in the loney tunes/tom&jerry era, reflecting and following the pace/beats of the action on the screen. For the fun of it assume the following one: 
    «The main character is standing in a dark alley. carefully looks around for hostiles. Slowly opens his trench coat, searches for a while and violently pulls out a smelly shark. He starts saking it around in an uncontrollable fashion. With all the noise, Furious cats jump out of the garbage cans and disappear in all directions. Now it's quiet again, silent. The main character is alone in the scene. He turns towards the camera and says "run pigeons, run: it's Robert Frost
    I suppose one could either create the music after drawing the sequence but more often than not, the music is already made and you have to time the animation to the music. Don't you agree? Would you precompose the animation/audio in some program like premiere/aftereffects where you can stretch the video and redraw new frames where needed? Until you have the final results that is beat-perfect in your hands before moving to VS to have it replicated? Or would you jump directly inside VS? It seems that you cannot properly preview the animation speed inside the editor, so you have to run the game for that. Would you run the game again and again tweeking frame pauses until you get it right? 

    So far I've thought of breaking down my sequence into smaller segments, that are linked by a few repeatable frames. For instance the segments of looking around, searching inside his trench coat, shaking the shark and standing silently, could each be looped n-times; as many times needed to have the visuals synchronize to the audio.

    But is that all there is?
    Any good ideas/tips for syncing the music?

    Forum Fan

    107 Posts


  • #2, by afrlmeSunday, 09. April 2017, 02:28 3 years ago
    You could split the music into loopable segments so that you can loop them or player animations until one or the other gets to a certain point then move onto the next as needed.

    I add action parts / scripts inside of animation frames themselves to be able to update conditions, values, start or stop other animations, etc.

    Video is a viable option, but only if you don't have complex animations / environmental animations going on in the background as it's unlikely your video will start playing from the same frames the animations are at, which might look a little jarring, though most people probably wouldn't notice until someone points it out to them.

    Imperator

    6846 Posts

  • #3, by F_KalSunday, 09. April 2017, 14:20 3 years ago
    @AFRLme, that's a very good idea that I certainly hadn't thought about! Breaking the music into loopable segments! Hmm, time loopable music segments to loopable animations and you have a dynamic music system... hmm, interesting!
    I add action parts / scripts inside of animation frames themselves to be able to update conditions, values, start or stop other animations, etc.
    yes, that is what I've tried doing; It works, but admittedly it's a bit tedious - Especially the trial and error aspect of it for finding the perfect timing to the music progression.

    Do you preplan your animation sequences down to the frame (inside some other software perhaps?) before entering inside VS Editor, or do you throw the frames inside VS Editor, and then tweak it until you're happy with the result?

    Forum Fan

    107 Posts

  • #4, by loro-gamesSunday, 09. April 2017, 16:02 3 years ago
    I think if animation and music are very dependent and conncected to each other it would definetely make sense to plan the exact needed lenghts and the timing beforehand. For the cutscenes of our game I am still not sure how we are going to do them exactly but I have a tendency towards producing them as videos in after effects etc. just to be as precise as possible.

    For smaller stuff I will definitely use Visionaire but for long and complex scenes I think I will use After Effects. 

    Newbie

    64 Posts

  • #5, by afrlmeSunday, 09. April 2017, 17:19 3 years ago
    I don't create the animations or sort out the planning for them. Marian creates all the art & animations in cinema4d. He exports them & tells me the global delay value, but if I think it looks too fast or slow then I will adjust the values myself or even on occasion add custom delay values to specific individual frames until it looks right to me.

    Trying to time it to music is kind of hard whereas having it update certain conditions, values, trigger other animations, etc is a lot easier & more precise when done inside of animation frames that guessing the time with a pause action part.

    Broken Sword used animations & certain events to trigger music cues - which is a lot easier to handle. I believe another game (not a point & click game) called mark of the ninja chopped up their music into different segments & split the different sounds into separate tracks so that they could dynamically have different tracks & loop segments come in & out depending on what the player was currently doing - fighting, sneaking, hiding, being chased, etc.

    Imperator

    6846 Posts

  • #6, by F_KalSunday, 09. April 2017, 21:43 3 years ago
    thank you guys for the feedback! It's much appreciated; the whole audio aspect of game making is totally alien to me...

    Now that @AFRLme mentions Mark of the Ninja and mixing tracks on the fly, I recall a puzzle in Daedalic's adventure game Fire where you produce a tune by enabling different audio tracks(link here)! Of course this  demands that you have access to the master files of the audio or have the audio producer design some complementary pieces/variations for you. Not to mention that a very important aspect of all that is being able to start all these audio tracks at the same time so that they stay in sync... I wonder how good VS is in that respect? 

    Then again we have LucasArts' iMuse (here,here,here,here and iMuse in action here,here and here) where the add cues to the music, very elaborate stuff! There is so much depth to the whole topic that it gets scary!

    But I suppose that since I don't have access to the master files (I'm mostly using music from the public domain/creative commons) all these multi-track approaches are merely a philosophical debate.  
    For now it seems I'll have to keep animation-to-music co-dependability to a minimum.

    In any, case feel free to share your thoughts and tips, they are much appreciated!
    Thanks!

    Forum Fan

    107 Posts

  • #7, by afrlmeSunday, 09. April 2017, 23:34 3 years ago
    thank you guys for the feedback! It's much appreciated; the whole audio aspect of game making is totally alien to me...

    Now that @AFRLme mentions Mark of the Ninja and mixing tracks on the fly, I recall a puzzle in Daedalic's adventure game Fire where you produce a tune by enabling different audio tracks(link here)! Of course this  demands that you have access to the master files of the audio or have the audio producer design some complementary pieces/variations for you. Not to mention that a very important aspect of all that is being able to start all these audio tracks at the same time so that they stay in sync... I wonder how good VS is in that respect? 

    Then again we have LucasArts' iMuse (here,here,here,here and iMuse in action here,here and here) where the add cues to the music, very elaborate stuff! There is so much depth to the whole topic that it gets scary!

    But I suppose that since I don't have access to the master files (I'm mostly using music from the public domain/creative commons) all these multi-track approaches are merely a philosophical debate.  
    For now it seems I'll have to keep animation-to-music co-dependability to a minimum.

    In any, case feel free to share your thoughts and tips, they are much appreciated!
    Thanks!
    You could use Audacity to chop up the public domain tracks into loopable segments. I had to do that with a background music track for ALLD demo because I didn't like the full track. wink

    As for controlling when music parts play you could use the openAL Lua sound functions, but I must warn you that the volume for them is linked to the eSoundVolume channel & not the eMusicVolume channel.

    Imperator

    6846 Posts

  • #8, by F_KalMonday, 10. April 2017, 21:59 3 years ago
    thanks @AFRLme, I've already downloaded Audacity and am trying my hands on it! 

    Forum Fan

    107 Posts

  • #9, by afrlmeMonday, 10. April 2017, 22:37 3 years ago
    thanks @AFRLme, I've already downloaded Audacity and am trying my hands on it! 
    Here's a couple of quick tips...

    1. ctrl/cmd + f to fit waveform to window width
    2. ctrl/cmd + shift + f to fit waveform to window height

    It makes it a lot easier to work out where you need to cut although I recommend zooming in further for more precision. It doesn't take long to learn where to cut & how to create loops by looking at the waveform before you even highlight a section to listen to.

    Imperator

    6846 Posts

  • #10, by F_KalMonday, 10. April 2017, 23:22 3 years ago
    ah yes! These are some very handy shortcuts!!! Thanks!

    Forum Fan

    107 Posts

Write post