Wednesday, March 29, 2017

On Releases and Glitches

I've been watching the launch of Mass Effect Andromeda with more than a passing bit of interest, even though a) I'm not even finished with the first Mass Effect game, much less the entire trilogy, and b) I don't really have the money to drop on a new game.*

Still, the armchair quarterback in me has been following along with the hype and inevitable problems at the game's launch.

You know the old adage "Fool me once, shame on you, fool me twice, shame on me"? This definitely applies to software releases these days. Even the supposed gold standards of software development and release, Blizzard and Apple, have had their share of software launch bugs.

This makes me wonder why someone would even bother buying the game at launch, much less pre-ordering, when you know that bugs will frequently be the reward of playing the game first. Another way of putting it is "Why pay to be a beta tester?"

Sure, you may get extra goodies such as an extra in-game item or the soundtrack**, but is it truly worth the headache of dealing with a game that is frequently in need of major patches to even make it enjoyable?***


In the case of Mass Effect Andromeda, there are bugs, and there are features.

The bugs are the obvious items: system crashes, graphical glitches, selections that don't work, etc. You know, the usual stuff.

But features, those are design decisions that may seem like bugs but aren't.

There are animation glitches in Andromeda, no doubt, but the overall look and feel of the animation is not a bug or a glitch. That was a design decision.

I'm reminded of the behind the scenes extras in the DVD release of The Incredibles. In the video article, they were talking to Pixar developers and engineers about the technical leaps they had to make for The Incredibles to work. As it was Pixar's first animated movie with an almost exclusively human cast****, they had to expand their technical capabilities to get certain aspects of animating humans right. At one point during production, one of the engineers had to go to John Lasseter (the head of Pixar) and tell him that "at the moment, hair is still pretty much theoretical." The concept of having hair move properly when animating a human --whether that hair is dry or wet or in a convoluted design-- confounded the developers for a long while.

And in the work surrounding Mass Effect Andromeda, the scale of the game meant that Bioware likely had to determine what priorities the developers worked on, and what aspects of development they were going to use an off-the-shelf or generic solution for.

This isn't exactly a newsflash to people who have worked in software development; in my time at a software shop we handed translation from our software's format to other formatting standards --akin to converting from JPG to PNG and back-- to a third party. The trick was to integrate the third party's software into our existing package seamlessly, and that was not as simple a task as you might think. The number of bugs that resulted from that integration was... pretty large at times. A lot of times it had nothing to do with the third party software at all, but with coding in a completely unconnected part of the software.*****

What does all this have to do with the facial and character animation? My speculation is that part of the Andromeda animation wasn't a high enough priority to deal with as an internal project, and so Bioware used an off the shelf product to handle the animations. And the issues with the facial and character animation could be due to a) integration with the main software, b) the third party software needing tweaks to work better with the overall product, or c) the third party software is being asked to handle something that might be beyond its current capability.

Or maybe a combination of all three.

But this isn't just my speculation, here's an Animstate Roundtable which included professional animators discussing this very issue, pointed out by an article from PCGamesN and Gamasutra. The entire roundtable is interesting, but this part I found resonated with me the most:

"Simon: Before I speculate on what the cause of these animation issues are, I think it’s important for people to understand some of the numbers behind a game like this. I don’t have exact figures from ME:A, but we do know that Mass Effect 3 had over 40,000 lines of dialogue and Dragon Age had about 60,000. If we split the difference at 50,000 and conservatively estimate that each line averages out to about three seconds, that puts us at around 41 and a half hours of dialogue. That’s about 21 feature films worth of just talking. Most of the major animated feature films have a team of about 70+ animators working for two or more years to complete just one movie. A game like Mass Effect might have somewhere between five and ten focused on more than 20X the content in the same amount of time. To add to that, we need to also factor in localizing (translating) the game into at least 4-5 additional languages.

Now, it’s just not possible to keyframe that amount of content to any acceptable level of quality, so teams looking at that much scope try to find procedural solutions. I know in the past they’ve used an off the shelf solution called FaceFX, which analyzes the audio tracks and creates animation based on the waveforms, projection, etc. At a base level, it can read as a very robotic performance and I suspect that is what we’re seeing in some of the footage. You can work with the audio and the procedural tools to polish the performances in various ways of course, but when you’re staring down thousands of minutes of performance to clean up, your definition of “shippable” is a sliding bar that moves relative to team capacity and your content lock date. If it were my team and project, I would try to gather metrics on which scenes were the most watched based on playtest and use whatever polish time I had with those as a priority, letting the lesser seen ones go with a default pass." --From ROUND TABLE – MASS EFFECT: ANDROMEDA


Back to the original thought behind this post, why bother buying a game at launch if you know there's glitches and/or features that need to be cleaned up? Part of that is, I suppose, faith in the development house to get the job done right. Or if not done right initially, then to fix the problems quickly. Reputable development houses don't just sit on problems, they fix them.

And another part of this is the reality behind software development. It is much more complex than, say, building a fence or even a car, and constant tweaks in response to unforeseen problems is pretty much par for the course.

And finally, there's also the recognition that very few software development houses announce a release when they feel it's ready --okay, it's Blizzard-- and that when a release date is presented to the public there becomes an enormous amount of pressure to meet that date. The company doesn't want to lose face to its investors, the investors are constantly asking each quarter "What have you done for me lately?", and the development staff doesn't want to disappoint the fans. For my money, Blizzard does it the right way, but even they aren't immune to the occasional bad release.

From my perspective, I have absolutely no need to rush in and buy something the moment it is released, so I'm content to wait. I did that once, when I bought the original AMD Athlon system back in 1999, and six months later I could have paid about $600 less for the same system. I learned my lesson that time, and I've not wavered from it.

*Yes, I'm quite aware that you don't need to have played the original ME trilogy to have played Andromeda, but it provides a good buffer to rushing out and buying the game from the get go. Besides, immersing yourself into the world of Mass Effect prior to playing Andromeda isn't necessarily a bad thing, even without the Geth or Reapers.

**Okay, I can understand the soundtrack enticement.

***I'm not a fan of the Assassin's Creed series, but the bugs of Assassin's Creed Unity are infamous among gamers.

****I kind of count the robot as a minor character.

*****Okay, I'm going to get a little technical here, but in C and C++, memory allocation is a huge thing. If you don't do it right, or you go beyond your allocated memory, you could end up overwriting whatever else is in memory. It's very powerful, but it is also dangerous. If you don't clean up your memory allocations, you end up with what are called "memory leaks". And eventually that will kill your performance and potentially cause crashes of software or the computer/server. The greater the complexity of the software, the harder it is to find these memory leaks by yourself and you have to rely upon --you guessed it-- third party software that shakes memory leaks out. But the fun doesn't stop there, because something might be working perfectly fine in its "leaky" state, and once you fix it the function/code stops working right. And then you have to go find out why that's the case, and maybe you find even more memory problems underneath it.

Java has this problem too, and that's why a lot of Java implementations --especially early ones-- have so many memory problems.


  1. My current theory is that--rather than bugs or poor implementation--this was ME:A's prototype facial animation system. The first or second version they built to get all the cutscenes playable while the rest of the game was being built/iterated upon. Obviously a prototype system wouldn't be the best, but Bioware wouldn't worry about it as it would be replaced, or maybe only used on super-low end computers.

    I think Bioware meant to use a better final animation system, but ran into issues. Maybe time, or budget, or technical problems. But for whatever reason, they weren't able to finish the final animation system, and were forced to ship with the prototype system, with whatever fast improvements they could make to it.

    1. That is also a plausible result, Rohan. Either way, the problem is fixable, but truth be told an extra month fixing things and getting the facial animation system right would be worth gold right now.