Sync game events to music beats?

How can I go about syncing game events to beats in a song?

Ideally I could set a marker in FMOD at every beat I want something to happen (Say color on a unity object shifts from black to white on every other beat). Then create a script in Unity that listens for the marker?

Maybe this could also be done with automating a parameter? Could you could read what the parameter is set at for each frame?

I’d like to easily tailor where these shifts happen within FMOD.

If this isn’t possible, what’s a sound way to go about syncing game events to a beat?
Any help would be appreciated. Thanks!

If you set a tempo marker on the timeline in Studio then markers are automatically quantized to the beat (hold ctrl to dragging a marker to disable quantization).

If you want to get every beat, simply use the beat callback.

public class MarkerDemo : MonoBehaviour 
{
    FMOD.Studio.EventInstance instance;
    FMOD.Studio.EVENT_CALLBACK cb;

    void Start()
    {
        instance = FMOD_StudioSystem.instance.GetEvent("event:/Music/Complex/Situation_Oriental");
    

        cb = new FMOD.Studio.EVENT_CALLBACK(StudioEventCallback);
        instance.setCallback(cb, FMOD.Studio.EVENT_CALLBACK_TYPE.TIMELINE_MARKER | FMOD.Studio.EVENT_CALLBACK_TYPE.TIMELINE_BEAT);
        instance.start();
    }

    public FMOD.RESULT StudioEventCallback(FMOD.Studio.EVENT_CALLBACK_TYPE type, IntPtr eventInstance, IntPtr parameters)
    {
        if (type == FMOD.Studio.EVENT_CALLBACK_TYPE.TIMELINE_MARKER)
        {
            FMOD.Studio.TIMELINE_MARKER_PROPERTIES marker = (FMOD.Studio.TIMELINE_MARKER_PROPERTIES)Marshal.PtrToStructure(parameters, typeof(FMOD.Studio.TIMELINE_MARKER_PROPERTIES));
            IntPtr namePtr = marker.name;
            int nameLen = 0;
            while (Marshal.ReadByte(namePtr, nameLen) != 0) ++nameLen;
            byte[] buffer = new byte[nameLen];
            Marshal.Copy(namePtr, buffer, 0, buffer.Length);
            string name = Encoding.UTF8.GetString(buffer, 0, nameLen);
            if (name == "HIGH")
            {
                UnityEngine.Debug.Log("Reached high intensity marker");
            }
        }
        if (type == FMOD.Studio.EVENT_CALLBACK_TYPE.TIMELINE_BEAT)
        {
            FMOD.Studio.TIMELINE_BEAT_PROPERTIES beat = (FMOD.Studio.TIMELINE_BEAT_PROPERTIES)Marshal.PtrToStructure(parameters, typeof(FMOD.Studio.TIMELINE_BEAT_PROPERTIES));
        }
        return FMOD.RESULT.OK;
    }
}
1 Like

Thanks for sharing this snippet, it helped me understand how to implement some beat-driven features in my own Unity project.

If I may ask a follow-up question, what kind of latency should we expect when receiving these callbacks, particularly the TIMELINE_BEAT callbacks? When using MonoBehaviour.Update to call FMOD.Studio.System.Update, I measure TIMELINE_BEAT callbacks occurring up to ±0.02s sooner/later than expected, and ±0.01s on average. When using MonoBehaviour.FixedUpdate instead, I still measure callbacks occurring up to ±0.02s sooner/later than expected, but ±0.00s on average. I am using UnityEngine.Time.realtimeSinceStartup to record timestamps.

These slight variations in period between callbacks are generally negligible for non-gameplay-critical features like beat-driven effects, but seem to be significant enough to cause trouble when trying to implement beat-driven input–sometimes input will be rejected because a callback occurred a little sooner or a little later than usual, for example.

The timeline processing occurs on the Studio thread once per Studio update (every 20ms). The beat callback is deferred for the main Unity update which will add some floating delay which is probably what you have seen.

It would be a nice feature for the callback structure to include some timing information that includes that delay, which would allow very accurate timing, but we don’t have any plans to implement that feature right now.

Hi, i’m rather new to unity and fmod integration, me being lazy was hoping nicholas wilcox’s snippet of code would build , however i am having a few problems setting up the code, initally i changed the line FMOD_StudioSystem.instance.GetEvent/ to FMODUnity.RuntimeManager.CreateInstance("event—
which solved an error, however IntPtr namePtr = marker.name; is throwing up an error, "cannot implicitly convert type FMOD.StringWrapper to system.IntPtr. I’m unsure as to the issue and how it could be resolved? any advice would be greatly appreciated

There’s a sample in the docs for the new integration regarding timeline callbacks: http://www.fmod.org/documentation/#content/generated/engine_new_unity/script_example_timeline.html

Thank you so much!! Works perfectly.

Note:
I had to add a few namespaces:

using UnityEngine;
using System.Collections;
using System;
using System.Runtime.InteropServices;
using System.Text;