Research Internships at Microsoft provide a dynamic environment for research careers with a network of world-class research labs led by globally-recognized scientists and engineers, who pursue innovation in a range of scientific and technical disciplines to help solve complex challenges in diverse fields, including computing, healthcare, economics, and the environment.

The Speaking the World into Existence research effort aims to apply the recent progress in large language models (LLMs) to prompt-based creation of interactive 3D scenes. Integrating LLMs with a game engine should enable not only faster development of 3D content in various domains, such as gaming, mixed reality applications or animated films, but also allow for spontaneous user-generated content in the course of an interactive experience.

For this project, we aim to build a tool for rapid prototyping of games, by employing generative AI to transmute natural language exchanges among a group of designers into playable content. We are in particular interested in utilizing a variety of large AI models to allow self-improving generation of game level layouts. While the primary target of this project is the creation of a prototyping tool, we expect that these techniques could be applied to the whole pipeline of creating interactive 3D content.