Pointing interaction March 31, 2010
Posted by Cesar in thinking me.Tags: 360, Apple, ds, interaction, iphone, mouse, pointing devices, project natal, ps3, razer, wacom, wii
add a comment
Once again interaction is the subject of the blog. I suspect I mentioned some of this stuff before, when talking about immersion, but it’s been on my mind lately so I thought I could share.
What I have been rummaging is how important pointing is in human communication. It might just be the most important gesture we use when trying to express something. We point our fingers at people when we refer to them, at objects to draw attention, at streets and buildings to give directions and even at ourselves to indicate personal feelings or achievements.
But we don’t stop there. If you think of pointing in an ample way, as indicating a position in space, pointing is also the most natural drawing strategy we have. We sketch on the sand and paint with our fingers. And if we need more precision, we create tools to do the job. Pens, pencils, compasses, rulers, all of them tools to make pointing more precise. Of course the goal is to generate a visual representation of something, but we always want to point on the process.
Pointing can also be very aggressive too: guns are pointed at the target before the shot, swords pointed at the heart to threaten. The list could go on and on.
In computers we tried not to point too much and at the beginning we just moved cursors up and down with our arrow keys (on a second thought, isn’t that pointing too?). But after some skepticism, the whole world succumbed to the mouse, invented at the Stanford Research Institute and popularized by Apple in 1984 with the Macintosh. The mouse has always been recognized as a pointing device and during the years it increased enormously in precision and functionality.
But using the mouse is not as natural as using our hands. Or pencils. So there goes humanity again developing tablet pens and displays, touch screens and what not. Maybe by now, if you remember I am a gamer, you can see where I am going. No? What if I say a few weeks ago I gave up fighting and got an iPhone?
Yes, there we go again to input methods and interaction in video games.
Touch screens are not new and neither are stylus pens. However, when touch devices became more popular and portable, the stylus became a hassle. Storing the pen inside the device, like it happens for example with the Nintendo DS, is a valid option. But when the device becomes truly portable and is used everywhere, like in a smart phone, it doesn’t work so well. Why? Because getting it out of the device to answer a call is very annoying. And because it is easy to lose.
Before I got the iPhone, I had an LG Dare. It is a decent phone, with a touchscreen and a stylus that I could attach to it. But every time I got a call, I would leave the stylus where it was and use my fingers. My first instinct was to actually use my fingers all the time, except that in the LG Dare you sometimes can’t do s*** without the stylus.
The iPhone is different. It was designed to be used with the fingers, it doesn’t even have a stylus. I was very impressed at how this apparently small change made such a big difference. Everything feels more natural.
And when I thought of that, I immediately remembered my talk about video game immersion and the division between transparent and engaging controls. Using the touchscreen with the fingers is very transparent.
Now let’s thing again about the new control technologies coming out for consoles this year, joining the Wii in the innovative input group. Project Natal is definitely the most transparent of them hands down. Nothing feels as natural and unobtrusive as moving our own bodies. However, does it offer a good pointing device? Probably not as good as Playstation Move (or Wii MotionPlus for that matter).
And therein, as the bard would tell us, lies the rub (this reminds me of Inside Man. Great movie). There’s a lot of expectation associated to Natal. but we don’t know exactly what to expect yet. If they succeed at making with image recognition a top-notch pointing system, there’s no discussion and similar systems will be the future of gaming. However, I’m not sure that will hold. Just like touch screens lose precision when used with fingers, pointing will lose precision without a device manufactured exclusively with that purpose.
So I guess what I am trying to put into perspective is the balance between transparency and pointing precision. The difference in this balance is most likely going to dictate the games that come out for each device. And the question is: given the importance of pointing and of transparency, which one do you prefer? My answer is always the same: just to be on the safe side, I’ll probably choose both and find out playing.
See you space cowboys…
Project Natal update: looking good! March 6, 2010
Posted by Cesar in gaming me.Tags: camera, computer vision, lag, lionhead, molyneux, project natal
add a comment
Peter Molyneux, the famous Lionhead designer behind the Fable series, recently talked about improvement in Natal technology. In a previous post, I mentioned Project Natal was dropping the internal chip and the camera had disturbing lags that were preventing optimal gameplay.
It seems Microsoft is taking care of that and the new version of the device is a lot better. I also noticed it is not mentioned anywhere that the hardware reached it’s final state. That means Microsoft is still working to make it even better, which is great.
So one of the concerns, the lag, is apparently going to be resolved by the end of the year, when Project Natal hits the market.
I would love to see the internal chip go back into the product, but that’s not going to happen. However, while it is true that the lack of the internal processor will limit updates to already existing games, the absence of camera lag ensures Project Natal will be as revolutionary as it can be for the new games. I can’t wait to play with it.
See you space cowboys…
Project Natal: changes and concerns January 10, 2010
Posted by Cesar in thinking me.Tags: camera, chip, computer vision, games, microsoft, project natal, range sensors
add a comment
Gamesindustry.biz recently announced Microsoft is dropping the internal chip from Project Natal. Even though there’s some talk about the move being related to making updates easier, it is clearly a move to drop costs. It is understandable, as the device must ship at a reasonable price to succeed.
However, from my previous experiences with computer vision solutions, this might represent a big drawback. Natal is not supposed to be a simple toy, targeting unique, exquisite experiences. It is supposed to act as a reliable replacement for controllers in many ways. Computer vision algorithms, like the ones necessary to process both texture and range data the sensors provide, require significant processing, specially if reliability and response time are big issues (as it is clearly the case).
That means not only old games will not get updates (because there won’t be a processor budget to spare for the vision algorithms), but also that new games using the technology will have to reduce processing somewhere else in order to make the system as responsive and reliable as it has to be.
Not all is lost though and I still have hope. I don’t think Microsoft would make this decision without some confidence most of the appeal will still be there. What I can say is that the presence of the range sensors (as opposed to a simple camera) means a lot of the algorithms can be much simpler than in texture only solutions. Tasks like background subtraction, for example, are almost free when range data is available. And pose detection, be that of the head, hands or the whole body, is also simplified, since the range information makes things less ambiguous.
But there’s also no question a dedicated chip would make the impact bigger and increase the usability of the device. Developers will now have to decide between keeping Natal functionality or improving AI and other gameplay areas.
In a related note, the article also mentions Microsoft is struggling with a 100 ms delay in the system. That’s a very common issue with imaging devices: just turn on your webcam and notice the lag. In a way that’s an even more serious problem than the drop of the chip, since a hardware delay cannot be fixed by software optimization alone, meaning no matter how simple the game is, the delay will still be there. But, again, there’s hope. The fact that it is a well known problem that MS is working to solve means it will probably not be there when Natal is released by the end of the year. Or at least that’s what I hope.
See you space cowboys…
Less buttons, more immersion: addendum December 14, 2009
Posted by Cesar in thinking me.Tags: motion controller, project natal, ps3, wand, wii
1 comment so far
Hi folks. I know, after three huge posts, another one about this is definitely too much. But for completeness sake, I just have to do it.
What motivated the whole immersion subject was my interest in Project Natal. So that’s what I mentioned every time I talked about it. But it wouldn’t be right to neglect other efforts on the same field. My curiosity about Project Natal made me take a closer look into both the new PS3 motion controller and the already released Wii MotionPlus. Although they have a different concept, Natal being controller free and the others being wand based, all of them try to achieve the same goal, which is to increase immersion by either making controls more transparent or more engaging.
I think the philosophical prize has to go to Microsoft, not using controllers is the ultimate goal. But it is unquestionable that the wand makes some things much easier, specially for hardcore games. Pointing things, be they pens or guns, and swinging objects, golf clubs or swords, are part of a huge segment of games. And the mechanic is easier to simulate with a wand (although not impossible without it I must say).
I can actually see them going in different directions with each technology. No matter how they fight for controller supremacy, the winners are the players.
See you space cowboys…
Less buttons, more immersion: Final act December 13, 2009
Posted by Cesar in thinking me.Tags: blazblue, blizzard, chess, controls, interactivity, mgs4, mw2, pacman, project natal
4 comments
Finally, we go back to the first post to talk about the third pillar of game immersion: interactivity. While it is true that raw interactivity is part of play, the quality of interactivity is a different subject. And, like I said before, it is what makes games different. There’s an interesting social component to it, but I’ll leave it for another day and focus on controls. I think control strategies in games follow roughly the same rules as graphics and sound: they must be good enough for our brain to accept them and fool us into believing them. And again, we compare them with what we know, be it reality or another experience. For some games it is a breeze. Grab any d-pad and play Pacman. If feels so natural! There’s no thinking involved. No need for instructions. You press up Pacman moves up; you press left, he moves left. Easy. Now think about Rock N’ Roll Racing: I don’t know about you but on games like this I always struggle at the beginning, until I get a grasp on the clockwise/counterclockwise turning. And then it is easy. My point is that there’s a learning curve involved and practice usually makes perfect. Once you control, you are ready to get involved.
Oh, but it is not that easy. When you play chess (I’m talking actual wood board and pieces, not on the PC), control is an abstraction kill. Your big hand grasping the tower to kill the bishop has very little to do with immersion. But it is easy to do. So I would break controls in two categories: transparent and engaging. I call transparent the controls whose goal is simply to not interfere with the rest of the experience. Like moving a chess piece with your hand. You don’t have to think about it. Just do it and move on. In these games, the controls are not meant to be part of the fun. Replace it with anything else that gets the job done and no one will mind.
Engaging controls are different. They are meant to be enjoyed as part of the game. Like shooting on MW2, hitting and performing air combos in BlazBlue, or CQC in MGS4. The examples are everywhere. In these games, it is not enough to be easy. In fact being difficult is sometimes part of the fun. But they have to make sense and feel natural. If, even after you mastered the controls, they still feel wrong, the fabric of game reality is torn apart and the nuisance breaks immersion.
Now let’s look back at Project Natal. Just like the Wii controls did, it will allow for interaction to be more transparent when it has to be. That means more complex games will be played without the controls being a hassle. In other words, transparency will be increased. For engaging controls, we can’t deny it should allow for a more natural and intuitive experience. When the gap to real movement gets narrower, more intricate games become more involving. One way or another, immersion takes a leap.
So there you go. These are in my opinion the three pillars of video game immersion. But I could be wrong! If you have any thoughts on the matter, I would love to hear it.
See you space cowboys…
Less buttons, more immersion November 23, 2009
Posted by Cesar in gaming me, thinking me.Tags: 360, buttons, computer vision, controllers, Demon's Souls, fifa 10, games, gears of war, immersion, lumines, metal gear solid, modern warfare 2, perception, project natal, starcraft, tetris, wii
4 comments
Immersion is one of the greatest goals every game tries to achieve. It is more obvious in what I call character based games. When you play Gears of War, you feel you are in Marcus Fenix’s shoes. When you play MGS4 you feel the drama of the dying hero as if it were your own. In Modern Warfare 2, every drop of blood on the screen makes you worry and get cover.
But it is also true for impersonal games; that is, the ones whose focus is not on characters. It is easier to notice it when I replace the word immersion for involvement. When playing Lumines or Tower Bloxx for example, you feel involved in the game: frustrated after a mistake, excited when a new level is reached, defiant when the score of a friend is beaten. Note impersonal and casual games are not the same thing at all. Chess is impersonal as are, heroes aside, most RTSs: replace one Zealot for another, there’s no difference at all. All sports games are like that too, with the exception of modes like EA’s Be A Pro.
Now, there’s a multitude of factors that contribute to how immersive a game is. I recently mentioned one talking about Demon’s Souls: challenge. A challenged player is an involved player. There’s also the connection between player and game at the fundamental design level: some people like puzzles, some like shooters. I will not get in the merits of each one, but if you like a type of game, you will be more into it. Style also plays a big role: when graphics and sound suit the game and your mood, they also improve immersion. In fact, I would say immersion is the reason behind most graphical updates in the games industry. From polygon count and texture size to shaders, in the search for immersion graphics chips are always evolving to provide better and more artistic or realistic visuals.
From here we get to the other facet of immersion I want to talk about: controls. No matter what game you are playing, connecting to it requires controlling it without trouble. You only get into Tetris when you learn how to move and rotate the pieces the way you want, you only appreciate Geometry Wars after getting used to moving the ship with the analog stick, you only feel like Sam Fisher when pulling all his moves gets easy, you only enjoy Fifa 10 when passing, dribbling and shooting becomes second nature.
Over the years, games became more and more complex. And with game complexity came complex controls. The Atari had 1 button. The NES had 2. The Genesis had 3. In the current generation, both PS3 and 360 have 4 face buttons, 2 shoulder buttons, 2 shoulder triggers and 2 clickable analog sticks. And I am not counting dpads, select and start buttons.
Most gamers are used to it. Hell, controllers could have more buttons and that wouldn’t be a problem, not to me. But with the last generation of consoles, we saw a big move in the opposite direction coming from Nintendo. The Wii has less buttons and makes up for it trying to detect something everybody knows how to do: move and point. When I first heard about it, I was very interested, both as a gamer and as a robotic perception researcher. To me, that meant games would become even more immersive, shooters would feel even more realistic.
It is curious that while it is true the Wii controllers increased immersion, that change did not affect most hardcore games. Nintendo correctly (from a business perspective) focused on using the more approachable controls to bring a new crowd to the video games era. And it worked very well for them.
Nintendo’s approach was so right that others have been following it ever since. After the success of the Wii, many game platforms started exploring new and more natural input methods. Touch screens and accelerometers became very popular.
But we will soon reach a new apex. Something I personally have been waiting for since I started studying computer vision. And Microsoft is the one about to pull it off: no controls. No buttons at all. If you haven’t heard of Project Natal before, go check it out. It is awesome.
The idea of a vision system in games is not new, the PS2 had the EyeToy. But there were many technical limitations: from sensor capability (one still eye won’t give proper perception of depth for example) to processing power, as robust computer vision algorithms require a whole lot of processing. Project Natal solves these problems in a very interesting way: a single camera is used for “texture” detection. And instead of stereo vision, they achieve 3D perception with a depth sensor. As for the processing power, Project Natal’s device features a custom processor, which is certainly there to reduce the load on the 360 hardware.
Like Nintendo did, Project Natal’s first efforts will probably aim the casual market and bring more gamers to the table. But that does not change the fact that immersion in video games will take a big leap. Imagine playing Lumines by grabbing the blocks and rotating them with your hands. Or simply using your empty hands to select your playlist. Wouldn’t it be cool? Heck, in Minority Report Tom Cruise needed cool glowing gloves to do what we are about to get with our bare hands. The future is here, my friends.
Anyway, as the number of buttons get close to the limit, after all we only have 10 fingers, new input methods are here to stay. I don’t have a Wii. But I will need a bigger living room when Project Natal becomes available.
See you space cowboys…
gaming me 