Hollow Knight and Silksong are hard for an agent in a specific way: success needs long-horizon reasoning about a boss’s attack pattern, from nothing but the pixels on screen. Reward is sparse, episodes are long, and a mistake ends the run.
The agents here learn navigation, combat and evasion from raw pixel observations plus a handful of game-state variables — player health, boss health. The question we set out to answer was whether better state representations and modern architectures buy performance, robustness and sample efficiency on a game this unforgiving.
Approach
The work builds on model-based RL, and specifically on STORM — a stochastic Transformer world model that encodes frames with a categorical VAE and predicts future dynamics in latent space.
Three things in that design matter for a game like this. Latent imagination lets the agent practise inside its own model instead of burning real episodes. Transformer sequence modeling carries context far enough back to cover a boss cycle. And a stochastic world representation leaves room for an environment that does not repeat itself exactly.
Outcome
My contribution
State representation, world-model training, and evaluation.
Reference
Zhang, W., Wang, G., Sun, J., Yuan, Y., & Huang, G. (2023). STORM: Efficient Stochastic Transformer based World Models for Reinforcement Learning. Advances in Neural Information Processing Systems (NeurIPS 2023). PDF