Gaming Horror – it's a Game Dev's Life

First off, some history. I hated one of my company’s rating system with every fiber of my body. The system had 5 options, from best to worst. I changed the ratings to protect the innocent:

- significantly above expectations
- above expectations
- met expectations
- below expectations
- significantly below expectations

When that system was first explained to us, and subsequently as well, a point was made to stress that “as expected” is really actually a rather good rating! And you do have to work a lot to achieve that rating. And of course, the SAE rating is “reserved for gods”. Those were actual words!

On the other end of the scale, if you got a SBE you’re basically out. It was said that two months in a row with SBE and you’re history, or three times within a year or two, something like that.

So the whole realistic range of ratings one could expect to get was actually limited to 3: AE, ME, BE.

Needless to say that system lent itself to creating rumors. Upper management all getting SAEs to save their bonuses or leads not having the guts to give an unpopular employee the deserved SBE rating were rumor mill food throughout. Many employees regardless of rating despised those ratings and the one on one meetings. Some departments even dropped them altogether while others were just not using them to actually discuss employee progress or issues – the ratings and reviews were viewed as annoying bureaucracy that you would just have to get over with every month. Since that rating system also determined how the bonuses were distributed among employees there was natural tendency for most to want higher ratings, whether deserved or not. Some got them, undeservedly, others didn’t, undeservedly. The ratings seemed to have depended more on the Lead and the employee’s character and liking than on actual work done and the quality of that work.

In turn, giving an employee a higher rating could jeopardize the Lead’s rating if he considers that only so much points are available for the studio, and higher ratings mean more points spent. It’s just a theory based on statistics. If more points are still available by not giving your employees a better rating, you yourself might have a better chance of getting a better rating because of the surplus of studio points that can still be spent. By trend, i would not be surprised to see the higher ups getting, on average, higher ratings than the workforce below them. This is simply greed at work both in the sense of what it’s affecting and where it’s having its effects. Overall, the system was unfair and brought with it all the negative effects it wanted to avoid.

So, while i was trying to setup a company with my colleagues, i devised a way to rate employees without using any numbers but instead, accepting the fact that evaluations are always something that is analogue and can’t really be measured with any accuracy (unless done or supervised by a trained HR representative with proper protocols and measure points). Definetely employee ratings can’t and shouldn’t be forced into a system that leaves you with only 5 or even just 3 choices.

…

Now, here’s my employee rating concept. First, you draw a half-circle with any drawing program so that it best fits the paper. From the center of the circle you draw a line (blue in the picture) straight upwards to the “middle” of the semi circle. I’ll attach a picture, excuse my terrible drawing skills (i’m a programmer):

The line intersecting the half-circle represents the team’s goal for a certain time period. You can label that any way you that makes sense to you (and your employee). Essentially you have two parameters to rate on: length (or amount, measured in distance from center) and direction. How exactly you define these is up to you and can be changed depending on your needs (hence: agile). You could even measure the relative distance between two points on the left and right side of the half-circle if you have something to rate that is mutually exclusive, as in the amount of work done in Project A vs the work done in Project B. Be creative and provide a meaningful and simple to understand context for measuring the work of your employees. I give you some examples further down.

When you rate an employee, you can only consider two things: for example how much work was done by the employee, and how much it helped get the team towards the goal (or the quality of the work done, or …). The amount of work you rate by putting an X closer to the circle border, the more work that employee has completed. You put the X closer towards the line if you think the employee’s work was very beneficial to achieving the team goal, or further away if it wasn’t (it could still be meaningful work though). By limiting the review to two items you communicate a clear focus of what’s (most) important to you and the company.

The ideal, perfect employee will have it’s X on the circle where the blue line intersects it. But that will rarely happen. Here you have the option to put the X still on the circle but more on the right or left side of the half-circle (which side can matter, later more on that). What you say by this: “I appreciate you’re working hard, but it’s not really helping us get toward the goal. Try to focus your efforts on work that helps us accomplish our goals”. The other extreme would be an employee that scores on the line but very close to the center of the circle. By that you mean: “Your work is always in line with our goals but your output could be much improved.” At this point try to find out why the work output is so little and offer help, training or try to work with daily goals.

The rating process should be done by the Lead/Manager and employee together – the employee first rates himself by placing an X with one color, then the Manager does his X rating with another color. That way significant misalignments in perception of reality are easily observed and can be discussed. If, during the discussion, either one changes their mind and would change the rating, throw away the paper and do it again until both are satisfied. It’s absolutely ok and encouraged that both X’s are still on different locations.

Make notes and put them on a separate sheet or on your computer but make sure ratings and notes can later be put back together (eg add month & year plus employee name to both sheets).

Now if you’ve collected several such sheets, and preferably they’re put together so you can flip through them (smaller sheets of paper work better) you can then “animate” the ratings of the previous months by quickly flipping through the papers. You may notice:

- seemingly random and wild fluctuations
- a steady incline or decline
- relative stagnation

Plus you have on record that an employee rates his own work usually higher or lower than the manager. This might be important information since in my experience, over- and underestimating one’s own work are often causes of numerous other issues and tend to hinder people from achieving great results. But even worse are misperceptions of the manager. These misperceptions can even make people depressive, frustrated, aggressive, or simply assholes – regardless of whether it’s their own perception of reality or that of the manager’s which is wrong.

It also gives you a chance to consult with others if you see such misalignment in the ratings because sometimes, it’s quite simply the manager’s limited point of view about an employee’s work, or the employee is not particular good at selling himself. It may be a simple matter of lacking PR skills, not playing the corporate politics game or the manager not having been present during the greatest accomplishments of the employee.

So for a yearly review you have a cute animation of the ratings and you can take into account things like continuous improvement or decline, highly fluctuating ratings etc and ponder respectively discuss the reasons for that.

I believe rating people without numbers takes away a lot of stress and anxiety that people put on themselves when it comes to rating oneself or others, especially if it has to be in concrete numbers and more so if the scale gives only limited options. I’ve seen Managers write down “in-between” scores, like a “ME+” in order to show it was a little more than expected but not enough to get the higher rating. However, such nuances are lost when you enter them into an electronic system that doesn’t allow such fractional ratings. Rating someone is hardly fair. But it could be made much more playful by using pens and ratings without a scale – since this is a highly subjective thing and the point is not the rating but the discussion that revolves around it.

The ratings can also be personalized depending on what you expect of the employee. By using a Team Goal you can bring and keep your employees focused on completing the team’s goals. Especially if you have issues of the team not really sticking their noses together. On the other hand it could simply be a subjective measure of the quality of the work that was done instead of using the Team Goal, apply that quality goal to all employees if you have overall quality issues with the team’s work. This way you can show that you have an eye on that particular issue.

In addition you can do a nice thing with these ratings: often employees tend to have more than one area of expertise. A programmer might also be a Lead, an artists might also do level design, a database editor might also keep your server in shape – and so on. You can rate the main two jobs (and no employee should ever have more than two “main” jobs btw) separately by choosing one side of the semi-circle to stand for one job, the other side for another. So you can rate people highly for doing efficient and error free database editing but you also determine that you finally need to hire an efficient System Administrator at your company and relieve the database editor of that task.

There may also be employees who enter a new field, or want to transition towards a specific area of expertise. Let’s assume you have a programmer who’d much rather do game design instead. So one side of the semi-circle is for programming work, the other for game design work. You work out a schedule for transitioning that person’s work and over time you should see the amount of work done on one side go down in favor of the other side’s work done going up. However, if that new side’s line of work does not have the expected quality (the team goal) and it doesn’t improve as much as you expected, you can slow down the transition, help that person improve in the new area, or just cancel the transitioning altogether.

This picture might be read as such: Employee has done good work on the programming side overall but due to his lack of experience he rated his quality and amount of design work much higher than the manager. Both should discuss the discrepancy between their ratings. Did the manager overlook something? Is the employee overconfident about the work he delivered?

I know a system like that is hard if not impossible to implement in a large corporate environment. But the idea was really evaluating employees of smaller teams/studios.

Any thoughts?