During summer of 2018, I worked at Sony Pictures as a game development intern. This was part of a three-month summer program where thirteen student interns prototyped a game or interactive experience to be played in virtual reality or augmented reality. In groups of two or three, we were assigned projects with different themes and objectives. My team was to create a mobile augmented reality app for promoting an upcoming Sony movie. Its main purpose was marketing the movie and to allow users to share the result of an augmented photo, but we decided to take it a step further and to add more interactive elements to turn it into a game, so that it is more than a photo booth app.
For the first few days, we brainstormed different ideas for the game. Since the main purpose is marketing, the game should be easy to pick up and suitable for a wide audience. After some initial brainstorming, we settled on building a gesture-based minigame based on a scene from the movie. The player will be in the perspective of the main character, who has to break out of a lab by fighting guards. What makes the game special is the interaction. To shoot at the enemies, the player has to wave a hand in front of the camera to trigger an attack. This was to mime the way the main character attacks in the movie.
Both my teammate and I are programmers, and we split the software development work into two parts. My teammate is a graduate student in game design, so he was responsible for programing gameplay and designing specific mechanics of the game, for example, the number of enemies and their range of attack, as well as any special visual effects that happens when the player hits something. Meanwhile, I took up the machine learning part. For a computer to recognize a hand from a camera, it needs to go through some training to learn what a “hand” looks like. Over a few hours, I gathered hundreds of thousands of hand images and fed them to the computer. It will then find similarities between those images to figure out patterns that determine what a hand is.
During the program, all thirteen interns shared a basement office and we often went to lunch as a group. This fostered communication between teams and encouraged us to ask each other for help. After a long day, some of us would stay in the office to play virtual reality games. At the end of the program, each group gave a presentation about their game and hosted a live demo in a large room.