In this tutorial, we’ll enhance the AI system from our previous “Unity AI Development: A Finite-state Machine Tutorial” by creating a graphical user interface (GUI). This GUI will enable us to develop the core finite-state machine (FSM) components more efficiently and with a smoother developer experience.

Revisiting Our FSM

In the previous tutorial, our FSM was built using C# scripts as architectural blocks. We created custom ScriptableObject actions and decisions as classes. Using ScriptableObjects, we achieved an easily maintainable and customizable FSM. This time, we’ll replace the drag-and-drop ScriptableObjects with a visual graph representation.

Tip: If you wish to make the game easier, you can swap the existing player detection script with this updated script, which reduces the enemy’s field of view.

Introducing xNode

To build our graphical editor, we’ll utilize xNode, a framework designed for visually representing node-based behavior trees. While Unity’s GraphView could be used, its API is still experimental and lacks comprehensive documentation. xNode offers a superior developer experience, making FSM prototyping and expansion more efficient.

Add xNode to our project as a Git dependency through the Unity Package Manager:

1. Open the Package Manager window: Window > Package Manager.
2. Click the + button in the top-left corner and choose Add package from git URL.
3. Enter or paste https://github.com/siccity/xNode.git in the text field and click Add.

Now we’re ready to delve into the key elements of xNode:

Understanding the xNode Visual Editor

xNode uses graphs where each State and Transition is represented as a node. These nodes connect using input and output connections, forming relationships within our graph.

Imagine a node that takes three inputs: two of any type and one boolean. This node outputs one of the arbitrary-type inputs based on the boolean input’s value.

To transition our FSM into a graph, we’ll modify the State and Transition classes. Instead of inheriting from ScriptableObject, they will now inherit from the Node class. Additionally, we’ll create a graph object of type NodeGraph to hold all our State and Transition objects.

Modifying BaseStateMachine for Inheritance

To build our GUI, we’ll add two new virtual methods to our existing BaseStateMachine class:

Declaring these methods as virtual enables us to override them in classes inheriting from BaseStateMachine. This allows us to define custom initialization and execution behaviors:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49

using System;
using System.Collections.Generic;
using UnityEngine;

namespace Demo.FSM
{
    public class BaseStateMachine : MonoBehaviour
    {
        [SerializeField] private BaseState _initialState;
        private Dictionary<Type, Component> _cachedComponents;
        private void Awake()
        {
            Init();
            _cachedComponents = new Dictionary<Type, Component>();
        }

        public BaseState CurrentState { get; set; }

        private void Update()
        {
            Execute();
        }

        public virtual void Init()
        {
            CurrentState = _initialState;
        }

        public virtual void Execute()
        {
            CurrentState.Execute(this);
        }

       // Allows us to execute consecutive calls of GetComponent in O(1) time
        public new T GetComponent<T>() where T : Component
        {
            if(_cachedComponents.ContainsKey(typeof(T)))
                return _cachedComponents[typeof(T)] as T;

            var component = base.GetComponent<T>();
            if(component != null)
            {
                _cachedComponents.Add(typeof(T), component);
            }
            return component;
        }

    }
}

Now, within our FSM folder, let’s create these two files:

For now, BaseStateMachineGraph will only inherit from the BaseStateMachine class:

1
2
3
4
5
6
7
8

using UnityEngine;

namespace Demo.FSM.Graph
{
    public class BaseStateMachineGraph : BaseStateMachine
    {
    }
}

We can’t add functionality to BaseStateMachineGraph until we create our base node type, which we’ll do next.

Implementing NodeGraph and Base Node Type

Let’s create the following two files under our new FSMGraph folder:

Currently, FSMGraph will solely inherit from the NodeGraph class without any added functionality:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

using UnityEngine;
using XNode;

namespace Demo.FSM.Graph
{
    [CreateAssetMenu(menuName = "FSM/FSM Graph")]
    public class FSMGraph : NodeGraph
    {
    }
}

Before creating classes for our nodes, let’s add a new C# Script:

This FSMNodeBase class will have an input named Entry of type FSMNodeBase. This allows us to connect nodes together.

We’ll also include two helper functions:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

using System.Collections.Generic;
using XNode;

namespace Demo.FSM.Graph
{
    public abstract class FSMNodeBase : Node
    {
        [Input(backingValue = ShowBackingValue.Never)] public FSMNodeBase Entry;

        protected IEnumerable<T> GetAllOnPort<T>(string fieldName) where T : FSMNodeBase
        {
            NodePort port = GetOutputPort(fieldName);
            for (var portIndex = 0; portIndex < port.ConnectionCount; portIndex++)
            {
                yield return port.GetConnection(portIndex).node as T;
            }
        }

        protected T GetFirst<T>(string fieldName) where T : FSMNodeBase
        {
            NodePort port = GetOutputPort(fieldName);
            if (port.ConnectionCount > 0)
                return port.GetConnection(0).node as T;
            return null;
        }
    }
} 

Eventually, we’ll have two types of state nodes. Let’s add a class to accommodate these:

1
2
3
4
5
6

namespace Demo.FSM.Graph
{
    public abstract class BaseStateNode : FSMNodeBase
    {
    }
} 

Next, modify the BaseStateMachineGraph class:

1
2
3
4
5
6
7
8

using UnityEngine;
namespace Demo.FSM.Graph
{
    public class BaseStateMachineGraph : BaseStateMachine
    {
        public new BaseStateNode CurrentState { get; set; }
    }
}

We’ve hidden the inherited CurrentState property and changed its type from BaseState to BaseStateNode.

Building Blocks of Our FSM Graph

Let’s create three new classes within our FSMGraph folder to represent the core elements of our FSM:

In our previous FSM, the State class iterated through the list of transitions. Here, StateNode serves a similar purpose, iterating through nodes retrieved via the GetAllOnPort helper method.

Add an [Output] attribute to outgoing connections (transition nodes) to signify their inclusion in the GUI. By design, xNode sources the attribute’s value from the source node (the one containing the field marked with [Output]).

Since [Output] and [Input] attributes describe relationships and connections managed by the xNode GUI, we can’t use them conventionally. Note the difference in how we handle Actions versus Transitions:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

using System.Collections.Generic;
namespace Demo.FSM.Graph
{
    [CreateNodeMenu("State")]
    public sealed class StateNode : BaseStateNode 
    {
        public List<FSMAction> Actions;
        [Output] public List<TransitionNode> Transitions;
        public void Execute(BaseStateMachineGraph baseStateMachine)
        {
            foreach (var action in Actions)
                action.Execute(baseStateMachine);
            foreach (var transition in GetAllOnPort<TransitionNode>(nameof(Transitions)))
                transition.Execute(baseStateMachine);
        }
    }
}

Transitions output can have multiple connected nodes. We use the GetAllOnPort helper method to obtain a list of those [Output] connections.

RemainInStateNode is straightforward. It doesn’t execute any logic and simply instructs the agent (the enemy) to stay in its current state:

1
2
3
4
5
6
7

namespace Demo.FSM.Graph
{
    [CreateNodeMenu("Remain In State")]
    public sealed class RemainInStateNode : BaseStateNode
    {
    }
}

The TransitionNode class is currently incomplete and won’t compile. These errors will resolve as we update the class.

To build TransitionNode, we need to work around xNode’s output value origin rule, similar to what we did with StateNode. Unlike StateNode, TransitionsNode’s output can only connect to a single node. We use GetFirst to fetch the one node attached to each port (one for the true state and one for the false state):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

namespace Demo.FSM.Graph
{
    [CreateNodeMenu("Transition")]
    public sealed class TransitionNode : FSMNodeBase
    {
        public Decision Decision;
        [Output] public BaseStateNode TrueState;
        [Output] public BaseStateNode FalseState;
        public void Execute(BaseStateMachineGraph stateMachine)
        {
            var trueState = GetFirst<BaseStateNode>(nameof(TrueState));
            var falseState = GetFirst<BaseStateNode>(nameof(FalseState));
            var decision = Decision.Decide(stateMachine);
            if (decision && !(trueState is RemainInStateNode))
            {
                stateMachine.CurrentState = trueState;
            }
            else if(!decision && !(falseState is RemainInStateNode))
                stateMachine.CurrentState = falseState;
        }
    }
}

Let’s visualize the outcome of our code.

Creating the Visual Graph

With our FSM classes set up, we can create our FSM Graph for the enemy agent.

1. In the Unity Project window, right-click the EnemyAI folder and select: Create > FSM > FSM Graph.
2. Rename the graph to EnemyGraph for clarity.
3. In the xNode Graph editor window (double-click EnemyGraph if it’s not visible), right-click to access the drop-down menu with State, Transition, and RemainInState options.
4. Create the Chase and Patrol states:
   1. Right-click and choose State, then name the node Chase.
   2. Repeat for the Patrol state.
   3. Drag and drop the existing Chase and Patrol actions onto their corresponding state nodes.
5. Create the transition:
   1. Right-click and choose Transition.
   2. Assign the LineOfSightDecision object to the transition’s Decision field.
6. Create the RemainInState node:
   1. Right-click and choose RemainInState.
7. Connect the graph:
   1. Connect Patrol’s Transitions output to the Transition node’s Entry input.
   2. Connect Transition’s True State output to Chase’s Entry input.
   3. Connect Transition’s False State output to Remain In State’s Entry input.

The graph should resemble this:

Currently, nothing indicates which state (Patrol or Chase) is our initial state. The BaseStateMachineGraph class doesn’t know which state to begin with.

To address this, let’s create:

The InitialNode output will designate the starting state. Now, inside FSMInitialNode, add:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

using XNode;
namespace Demo.FSM.Graph
{
    [CreateNodeMenu("Initial Node"), NodeTint("#00ff52")]
    public class FSMInitialNode : Node
    {
        [Output] public StateNode InitialNode;
        public StateNode NextNode
        {
            get
            {
                var port = GetOutputPort("InitialNode");
                if (port == null || port.ConnectionCount == 0)
                    return null;
                return port.GetConnection(0).node as StateNode;
            }
        }
    }
}

With the FSMInitialNode class created, we can connect it to the initial state’s Entry input and return that state via the NextNode property.

Back in our graph, add the initial node:

1. Right-click in the xNode editor window and select Initial Node.
2. Connect the FSM Node’s output to the Patrol node’s Entry input.

The graph should now look like this:

For convenience, let’s add the following to FSMGraph:

When we first access the InitialState property, the getter will search for the FSMInitialNode within our graph. Once found, it utilizes the NextNode property to determine our initial state node:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

using System.Linq;
using UnityEngine;
using XNode;
namespace Demo.FSM.Graph
{
    [CreateAssetMenu(menuName = "FSM/FSM Graph")]
    public sealed class FSMGraph : NodeGraph
    {
        private StateNode _initialState;
        public StateNode InitialState
        {
            get
            {
                if (_initialState == null)
                    _initialState = FindInitialStateNode();
                return _initialState;
            }
        }
        private StateNode FindInitialStateNode()
        {
            var initialNode = nodes.FirstOrDefault(x => x is FSMInitialNode);
            if (initialNode != null)
            {
                return (initialNode as FSMInitialNode).NextNode;
            }
            return null;
        }
    }
}

In our BaseStateMachineGraph, we’ll reference FSMGraph and override the Init and Execute methods inherited from BaseStateMachine.

• Overriding Init sets the CurrentState to the graph’s initial state.
• Overriding Execute calls the Execute method on the CurrentState:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

using UnityEngine;
namespace Demo.FSM.Graph
{
    public class BaseStateMachineGraph : BaseStateMachine
    {
        [SerializeField] private FSMGraph _graph;
        public new BaseStateNode CurrentState { get; set; }
        public override void Init()
        {
            CurrentState = _graph.InitialState;
        }
        public override void Execute()
        {
            ((StateNode)CurrentState).Execute(this);
        }
    }
}

Finally, let’s apply our graph to our Enemy object and observe it in action.

Testing the FSM Graph

In the Unity Editor’s Project window:

1. Open the SampleScene asset.
2. Locate the Enemy game object in the Hierarchy window.
3. Replace the BaseStateMachine component with the BaseStateMachineGraph component:
   1. Click Add Component and select the BaseStateMachineGraph script.
   2. Assign our FSM graph (EnemyGraph) to the Graph field.
   3. Remove the BaseStateMachine component (right-click and select Remove Component).

The Enemy game object should now appear as follows:

That’s it! We have successfully implemented a modular FSM with a graphic editor. Pressing the Play button will demonstrate that our graphical enemy AI functions identically to the previous ScriptableObject-based version.

Looking Ahead: FSM Optimization

The advantages of a graphical editor are clear, but a word of caution: As your game’s AI becomes more complex with numerous states and transitions, the FSM can become visually cluttered and difficult to interpret. The graphical editor might turn into a web of lines, making debugging a challenge.

As in the previous tutorial, we encourage you to experiment with the code and optimize the stealth game further. Consider color-coding state nodes to indicate active/inactive states or resizing the RemainInState and Initial nodes to manage screen space.

Such improvements aren’t merely aesthetic. Visual cues like color and size can greatly aid debugging. A visually organized graph is easier to understand and analyze. With the graphical editor foundation established, the possibilities for enhancing the developer experience are endless.

The editorial team of the Toptal Engineering Blog thanks Goran Lalić and Maddie Douglas for reviewing the code samples and technical content presented in this article.