Enhancing Game Immersion with On-Device AI & NLP in iOS Swift

There has been a lot of discussion around ChatGPT and similar tools that assist developers in writing code. While these tools are incredibly useful, I want to share how you can leverage AI to enhance immersion in your apps and games.

I also want to demonstrate that this can be done entirely on-device. While third-party services like OpenAI provide some of the most advanced AI capabilities, the built-in AI features available on modern operating systems are becoming increasingly powerful. Many use cases can now be handled fully on-device. Below, we’ll explore an example using iOS and SwiftUI. If you want to jump straight to the code, click here.

Enhancing Text-Based Adventure Games with AI

In our example, we have a text-based adventure game. Like classic point-and-click adventure games, the player is presented with multiple options and must choose one. This model has worked well for decades, offering players choices that shape their journey through the game. The ability to make decisions increases immersion and engagement.

However, there are limitations. Traditional user interfaces require predefined choices due to screen size constraints, typically limiting the player to just three or four options.

A Traditional Approach:

Imagine you're playing a detective game, investigating a crime scene at a train station. Historically, the game would present you with fixed options, such as:

1. Inspect the tracks

2. Examine the train car

3. Go to the ticket office

   
   

While this system works, it restricts player freedom.

A More Immersive AI-Driven Approach:

With AI, we can create a more dynamic experience. Instead of predefined choices, the player can type any action they wish. Behind the scenes, the game matches their input against a larger set of possible actions. For example, in addition to the above options, the player might also be able to:

4. Question the witnesses

5. Walk around the station

6. Visit the street-level shops

7. Follow a suspicious character

   
   

By allowing free-form text input, players are no longer constrained by a small set of choices, making the game world feel more open-ended and immersive.

Implementing Natural Language Processing (NLP)

Using Natural Language Processing (NLP), we can interpret user input and match it to predefined actions. The AI assesses the intent behind the input and selects the most relevant option. If the input does not match any available action—such as "Fly through the train station and into space"—the game can respond with something like, "You can’t do that here."

However with new NLP implementation a user could enter in: "Jump down onto the railway track and look for more clues". This would map to action: "Inspect the tracks".

Code Implementation:

In our example, we define a class to store the possible actions. Then, we present a view that displays the game’s narrative and accepts user input. Finally, our NLP logic processes the input and finds the best-matching action.

GameTypes.swift - basic types for the example code

struct GameScene {
	let description: String
    let validActions: [String]
}

struct GameWorld {
    let scenes: [String: GameScene]
}

GameEngine.swift - this loads the world and the scenes and has a processAction function which takes the input matches it to a valid action.

class GameEngine: ObservableObject {
	@Published var gameWorld: GameWorld?
	@Published var currentScene: GameScene?
	
	init() {
		loadGameWorld()
	}
	
	func loadGameWorld() {
		let scenes: [String: GameScene] = [
			"train_station": GameScene(
				description: "A crime has just been committed at the city train station! You have been tasked to solve it. You go to the station immediately and are welcomed by the police",						
				validActions: ["Inspect the tracks", "Examine the train car", "Go to the ticket office", "Question the witnesses", "Walk around the station", "Visit the street-level shops", "Follow a suspicious character", ...]
			),
			...
		]
		
		gameWorld = GameWorld(scenes: scenes)
		currentScene = gameWorld?.scenes["train_station"]
		print("Game world initialised successfully!")
	}
	
	func processAction(_ input: String) -> String {
        guard let scene = currentScene else { return "You are lost in the void." }
        if let matchedAction = TextMatcher.bestMatch(for: input, in: scene.validActions) {
            return "You chose: \(matchedAction)"
        } else {
            return "You can’t do that here."
        }
    }
}

TextMatcher.swift:

import NaturalLanguage

class TextMatcher {
    static let embedding = NLEmbedding.wordEmbedding(for: .english)

    static func averageVector(for phrase: String) -> [Double]? {
        guard let embedding = embedding else { return nil }
        let tokens = phrase.lowercased().split(separator: " ").map(String.init)

        let vectors = tokens.compactMap { embedding.vector(for: $0) }
        guard !vectors.isEmpty else { return nil }

        let vectorLength = vectors[0].count
        var average = [Double](repeating: 0, count: vectorLength)

        for vector in vectors {
            for i in 0..<vectorLength {
                average[i] += vector[i]
            }
        }

        for i in 0..<vectorLength {
            average[i] /= Double(vectors.count)
        }

        return average
    }

    static func bestMatch(for input: String, in options: [String], threshold: Double = 0.7) -> String? {
        guard let inputVector = averageVector(for: input) else { return nil }

        var bestScore: Double = 0.0
        var bestMatch: String?

        for option in options {
            if let optionVector = averageVector(for: option) {
                let score = cosineSimilarity(inputVector, optionVector)
                if score > bestScore {
                    bestScore = score
                    bestMatch = option
                }
            }
        }

        return bestScore >= threshold ? bestMatch : nil
    }

    static func cosineSimilarity(_ v1: [Double], _ v2: [Double]) -> Double {
        let dotProduct = zip(v1, v2).map(*).reduce(0, +)
        let magnitude1 = sqrt(v1.map { $0 * $0 }.reduce(0, +))
        let magnitude2 = sqrt(v2.map { $0 * $0 }.reduce(0, +))
        return dotProduct / (magnitude1 * magnitude2)
    }
}

This code defines a TextMatcher class that uses Apple’s NaturalLanguage (NL) framework to compare text similarity based on word embeddings. Here’s a breakdown of what it does:

Key Features:

1. Uses Pre-trained Word Embeddings

   • NLEmbedding.wordEmbedding(for: .english) loads an English word embedding model, where words are represented as numerical vectors.

2. Computes the Average Vector for a Phrase (averageVector(for:))

   • Tokenizes the input phrase into words.

   • Converts each word into a numerical vector using the embedding model.

   • Computes the average of all word vectors to get a single vector representing the phrase.

3. Finds the Best Matching Phrase (bestMatch(for:in:threshold:))

   • Converts the input phrase and each candidate option into vectors.

   • Compares them using cosine similarity, a metric that measures how similar two vectors are.

   • Returns the best match if it meets a similarity threshold (default: 0.7).

4. Computes Cosine Similarity (cosineSimilarity(_:_:))

   • Measures how similar two vectors are based on their angles in multi-dimensional space.

   • A value closer to 1 means higher similarity.

How to Support Us

This content will always remain free. If you find it valuable, please consider sharing it with others. You can also support us by downloading our apps or reading our books and leaving an honest review. We’d love to hear your thoughts and feedback!

Download Falling Sky from the Apple App Store today: https://apps.apple.com/app/id6446787964

Read our books: https://www.amazon.co.uk/stores/Matthew-J-Swarbrick/author/B0CRRLRJCV

Follow us on X: https://x.com/_kingdomarcade