maxlevel
ContactLog inStart free
BlogRuleset design

Choosing XP values without wrecking the behaviour you wanted

An XP value is a price, and pricing an action changes what it means to the person doing it. A method for setting numbers that hold up, and what the research says about when they backfire.

2 August 202614 min read
Four bars rising left to right on a dark umber background lit by a soft ember glow, labelled 10, 25, 40 and 90. The tallest bar is filled with the ember gradient. A line beneath reads: an XP value is a price.

Every XP value is a price

Put a number on an action and you have priced it. People respond to prices, and not always the way you hoped.

There is a field experiment that makes this uncomfortably clear. In A fine is a price, Uri Gneezy and Aldo Rustichini followed ten Israeli day-care centres for twenty weeks. From week 5, six of them began charging parents NIS 10 whenever they collected a child more than ten minutes late. The other four carried on as before. Late collections at the fined centres roughly doubled. Then the fine was dropped after week 16, and the rate stayed exactly where it was.

Charging for lateness turned it into something you could buy. Plenty of parents bought it. Withdrawing the charge did not give them their guilt back.

XP does the same thing, just aimed at behaviour you want more of. Attach 15 XP to "log a customer call" and the call is now worth 15 XP. Whatever it was worth before, socially or professionally, has been written over. Mostly that is the point. But go in knowing it is a one-way door: you can change the number later, you cannot make the action unpriced again.

LATE COLLECTIONS PER WEEK Weeks 1-4 No fine Weeks 5-16 NIS 10 fine, roughly 2x baseline Weeks 17-20 Fine removed
One bar per period, width proportional to its length in weeks. The dashed line marks the level before the fine existed; withdrawing the fine did not bring the rate back down to it.

Rank first, number second

The usual approach is to open a table and start typing numbers into the XP column, one row at a time. Those numbers never end up agreeing with each other.

Sort the actions first. Write down everything you want to count, then order it by a single question: how badly do you want one more of this today? Once the order holds up, assigning values is arithmetic. Make the gaps between the numbers match the gaps in the ranking.

Take a household chore rota. Emptying the dishwasher, 10. Cooking the evening meal, 25. The weekly shop, 40. Deep-cleaning the oven, 90. Nobody believes the oven is nine times more useful than the dishwasher. It is nine times more avoided, and avoidance is what you are actually pricing.

Two limits are worth keeping in mind as you spread the values out. Your cheapest action still has to be worth logging, or people quietly stop bothering and the data goes thin. And if one action dwarfs everything else, you have built a single-action ruleset with decorations hanging off it.

Empty the dishwasher 10 XP Cook the evening meal 25 XP Do the weekly shop 40 XP Deep-clean the oven 90 XP RANKED BY RELUCTANCE, NOT BY USEFULNESS
Bar length is the XP value. The spread is a picture of how much the household dreads each job.

Price the outcome, not the keystroke

Anything you can count, somebody can manufacture. Pay for keystrokes and keystrokes are what you will get.

Wells Fargo is the expensive version of this lesson. The Consumer Financial Protection Bureau's September 2016 enforcement action found that staff had opened roughly 1.5 million unauthorised deposit accounts and around 565,000 unauthorised credit-card applications, chasing sales targets and the bonuses attached to them. Penalties came to 185 million dollars, split between the CFPB, the Office of the Comptroller of the Currency, and the City and County of Los Angeles. The bank had designed the incentive. What it had not done was watch how people were hitting it.

WHAT AN UNMONITORED TARGET BOUGHT Sales targets and bonuses 1.5m unauthorised deposit accounts 565,000 unauthorised card applications $185m in penalties, September 2016 Figures as found by the CFPB, September 2016.
The Wells Fargo chain: a target that was measured but not monitored, and the count it produced.

That is the far end of the scale, but the mechanism is completely ordinary. Ordóñez, Schweitzer, Galinsky and Bazerman went through the research in Goals gone wild and kept finding the same cluster of side effects: tunnel vision on the goal, more unethical behaviour, warped risk-taking, a corroded culture, less intrinsic motivation. Their suggestion is to treat goals like prescription medication. Dose carefully, and watch for reactions.

For a ruleset that means the easiest thing to fake gets the smallest number. Where you can, count something the world had to change for. A meeting that exists in a calendar. A delivery someone signed for. A bin actually out on the kerb. It is why maxlevel leans on QR codes and NFC tags rather than only a form in the app: standing next to a thing is harder to fake than typing that you did. Harder, not impossible. A photograph of a QR code still scans.

Count the click Cheap to produce. The ruleset gets gamed, and the number stops meaning anything. Count the outcome Costly to produce, because the world had to change for it. The ruleset survives.
The difference between a ruleset that counts effort signals and one that counts things the world had to change for.

Recognition and payout are different levers

XP, levels, badges, your spot on the leaderboard: all feedback. Reward-store items: payment. They sit next to each other in the product menu and they do not behave the same way at all.

Deci, Koestner and Ryan pooled 128 experiments to pull the two apart. Tangible rewards came out at d = -0.34 on free-choice intrinsic motivation. Expected tangible rewards were worse, at -0.36. Performance-contingent rewards, where the payout depends on doing the thing well, did the least damage at -0.28. Verbal reward and positive feedback went the other way entirely, at +0.33.

So save the reward store for work nobody was ever going to do out of enthusiasm. The cold-call list. The oven. Do not hang a payout in front of the part of someone's job they already enjoy, because that is the exact scenario those effect sizes describe. And when you do make a reward contingent, hang it on doing the work well rather than on ticking it off.

EFFECT ON FREE-CHOICE INTRINSIC MOTIVATION (D) Positive feedback +0.33 Performance-contingent rewards -0.28 All tangible rewards -0.34 Expected tangible rewards -0.36 ENHANCES UNDERMINES
Effect sizes from 128 experiments. Bars are drawn by magnitude; direction is given by the sign.

Your first ruleset will be wrong

Hamari, Koivisto and Sarsa read through 24 empirical studies and concluded that gamification works, with caveats. Two are relevant here. Some of the lift may just be novelty. And results swing hard on context and on who the users are, which makes a ruleset that works on one sales floor a hypothesis anywhere else, not a template.

The same review flags something more awkward. When gamified elements were taken away, the engaged users were the ones who suffered. Losing points you earned stings more than never having earned them. So you are stuck: you will need to re-price, and re-pricing badly costs more than the mistake you are correcting.

A season boundary is the way out. Freeze the values for the lenght of a season, watch what happens, then change them when the season ends. A reset at that point is already expected, so nobody reads it as aimed at them personally. Cut the weekly shop from 40 XP to 15 XP in week six and it is a punishment. Do it in the first hour of a new season and it is just the rules of a new game.

Seasons are not in maxlevel yet. We want them out soon, because this is the problem they solve. Until then you can run one by hand: pick the end date before you launch the ruleset, tell everyone what it is, and hold the values until you get there. Announcing the date up front is most of what makes it work.

SEASON 1 Weekly shop = 40 XP Values held fixed SEASON 2 Weekly shop = 15 XP Values held fixed RE-PRICE HERE
The same cut lands very differently depending on which side of the boundary it happens.

What the software will not do

maxlevel will run your ruleset. It will not tell you the ruleset is wrong.

The engine handles the mechanical half. Actions in, XP out, levels, badge progress, leaderboard positions. None of that involves an opinion, and none of it changes between Tuesday and Thursday. Deciding the oven is worth 90 and the dishwasher 10, then deciding in March that you were wrong, stays with you.

Which is why the ranking deserves an afternoon rather than ten minutes. Nothing downstream can repair a badly ordered list.

YOU DECIDE Which actions count What each one is worth When to reset MAXLEVEL DOES Take the logged action Apply the rules Add the XP Judgement, every time. Deterministic, every time.
The engine side never changes its mind. The left-hand box is where all the difficult work lives.

Sources

  • Gneezy, U. and Rustichini, A. (2000). A fine is a price. The Journal of Legal Studies, 29(1), 1–17. PDF
  • Consumer Financial Protection Bureau (2016). Consumer Financial Protection Bureau fines Wells Fargo $100 million for widespread illegal practice of secretly opening unauthorized accounts. Press release, 8 September 2016
  • Ordóñez, L., Schweitzer, M., Galinsky, A. and Bazerman, M. (2009). Goals gone wild: the systematic side effects of over-prescribing goal setting. Harvard Business School working paper 09-083. Abstract
  • Deci, E., Koestner, R. and Ryan, R. (1999). A meta-analytic review of experiments examining the effects of extrinsic rewards on intrinsic motivation. Psychological Bulletin, 125(6), 627–668. PDF
  • Hamari, J., Koivisto, J. and Sarsa, H. (2014). Does gamification work? A literature review of empirical studies on gamification. 47th Hawaii International Conference on System Sciences. PDF
← All posts