Evaluation System

[like@boss]

one map is the same for all players so difficulty doesn't matter. It affects all players the same.

Each map takes a different amount of time to complete. I could perform a ridiculously hard task and save .700 on a short run or I could just do something fairly easy and save 2 whole seconds on a rather long run. Why should I be rewarded to such a great extent just because the mapper decided the map would be very long? Which took more skill? Which discovery is worth more? The missing factor here is the map itself.

It seems to me that this is becoming far more complicated than it actually needs to be. Anyone that wants to win DFWC must complete all rounds. At the end of the comp the player with lowest average rank obviously performed the best overall. This way no map is allowed to have a larger factor in terms of "value" (as a longer map would potentially be). Rank 1 on a short map is factored as same value as rank 1 on a longer map. In the case of two players having the same average rank then I'd just extend competition time.

[Arcaon]

one map is the same for all players so difficulty doesn't matter. It affects all players the same.

Each map takes a different amount of time to complete. I could perform a ridiculously hard task and save .700 on a short run or I could just do something fairly easy and save 2 whole seconds on a rather long run. Why should I be rewarded to such a great extent just because the mapper decided the map would be very long? Which took more skill? Which discovery is worth more? The missing factor here is the map itself.

It would seem to me that this is becoming far more complicated than it actually needs to be. Anyone that wants to win DFWC must complete all rounds. At the end of the comp the player with lowest average rank obviously performed the best overall. This way no map is allowed to have a larger factor in terms of "value" (as a longer map would potentially be). Rank 1 on a short map is factored as same value as rank 1 on a longer map. In the case of two players having the same average rank then I'd just extend competition time.

Unfortunately, that doesn't always work that well in practice.

Compare these two players:
(46 + 1 + 1 + 5 + 2 + 2 + 1) / 7 = 8.29
(11 + 6 + 3 + 4 + 17 + 6 + 2) / 7 = 7

I feel like Player 1 should have a better score than Player 2 here.

Those are kop4ik's and nebuLa's ranks from DFWC 2010.

[<hk>]

That's the problem with humans. Their experience of fairness differs.

"Average Rank"(ARS) and "Point Adding System"(PAS) are pretty much the same thing. Apparently you didn't see that.

In ARS your time is transformed into a "Rank" which is just a fancy name for "score".
Rank 1 -> 1 point
Rank 2 -> 2 points
...

In PAS the time is transformed into a score that is close to the time.
Instead of Rank 1 you get your amount of milliseconds of points.
Think of it as a more precise rank. It doesn't just look at (A is better than B) but at how much the player is better.
You can even transform them. Instead of 10000ms, 12000ms, 32000ms, 33000ms ... you get Rank1, Rank1.2, Rank3.2, Rank 3.3, (very simplified example)
On top of that, ARS has the disadvantage every players score being dependent on other players performance. If everyone else sucks, the amount of points I get changes. Imho it should stay the same no matter how good/bad the others play. Players should not interact in the ranking. AVR is unfair.

The Idea was to have 6 rounds. 3 with standard rank filtering to get rid of most of the players and another 3 with "Normalised Point Adding System" in which no players are dropped.

The whole Idea is to force players to score good times on ALL maps. Not just the last map. Every frame should count.

[like@boss]

<snip>
Those are kop4ik's and nebuLa's ranks from DFWC 2010.

He may be a more skilled player but the onus is on him and everyone else to prove it over 7 rounds. I don't think it's right to just giveup on a map because you don't like it. That's partly why I agree with with you that results based off rank foster more competition. IMO, he's got nobody to blame but himself... 46 is a joke

[Arcaon]

The Idea was to have 6 rounds. 3 with standard rank filtering to get rid of most of the players and another 3 with "Normalised Point Adding System" in which no players are dropped.

The one quoted in this post?

Let's cross our fingers that the results are the predicted ones then...

Edit: I still like Rane's idea of using the time of the player at the 50th percentile as normalizing point 2, rather than 150% of rank 1's time. It yielded better results for the previous DFWC.

[<hk>]

The one quoted in this post?

no. There some weird formula is used.

Let's cross our fingers that the results are the predicted ones then...

What is that supposed to mean?
Everything that happens here is that there is just more resolution in ranks. It's the inherently the same as ARS. All the problems apply to ARS as well.

Edit: I still like Rane's idea of using the time of the player at the 50th percentile as normalizing point 2, rather than 150% of rank 1's time.

Rankings should not be based on the performance of others. What's the middle anyway? The middle in ranks? the middle in time difference from last to first? The latter is a rather bad idea.
Is the middle player really the best choice?

The purpose of the 2nd normalisation point is to help bad players a bit. You also get more resolution in ranks. That's not essential though.

[Arcaon]

The one quoted in this post?

no. There some weird formula is used.

Well, that "weird formula" does exactly what you had proposed in the quoted post.

Let's cross our fingers that the results are the predicted ones then...

What is that supposed to mean?
Everything that happens here is that there is just more resolution in ranks. It's the inherently the same as ARS. All the problems apply to ARS as well.

I'm still unsure whether you're talking about this system or not:

Winner gets 100 points. (Normalising point 1)
Winning time+50% is 0 points (Normalising point 2)
Everyone slower than the winning time + 50% is a loser and doesn't get any points.
In between we just use their time relative to the best player. and normalise it in the same way.
Every one would get a point value between 100 and 0.

(It does sound a lot like a "Normalised Point Adding System" that you talked about in the previous post.)
If so, Round 4 in the previous DFWC would've been an example of a circumstance that would skew the results significantly. But I will not comment any more on that for the moment, as I'm slightly confused.

Rankings should not be based on the performance of others.

Isn't that the whole point of a ranking system? Without some magical program that accurates calculates how many points a time deserves from a map, I can't see any other solution.

What's the middle anyway? The middle in ranks? the middle in time difference from last to first? The latter is a rather bad idea. Is the middle player really the best choice?

50th percentile = middle player.
Here's an example; P1 uses the middleman's time as normalizing point 2, while P2 uses the best time + 50% as normalizing point 2.

Code: Select all

#   Name        Time     P1     P2

ROUND 0:
1.  Arcaon      17.984   100    100
3.  Bazz        18.200   92     97.6
5.  glm         18.312   87.8   96.4
46. Nt-kop4ik   19.016   61.6   88.5

ROUND 4:
1.  glm         02.840   100    100
2.  Nt-kop4ik   02.992   95.2   89.3
12. Arcaon      03.816   69.2   31.3
19. Bazz!       04.312   53.5   0

I might have missed what point system you're referring to more specifically, dunno.

[Arcaon]

Points awarded for ranks, in order: 20, 15, 12, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1.

Results for the last DFWC (last round first):

Code: Select all

kop4ik      20   15   15   9    20   20   0       99
Arcaon      12   12   2    20   10   10   20      86
Bazz        -    20   0    12   8    9    12      61
nebuLa      15   8    0    10   12   8    3       56
ZERG        10   3    3    0    0    0    0       16

It works, and it's extremely simple.

[Arcaon]

I took it upon myself to improve the system, but still keep it simple.

Floor(h * 2^(((w + 1) - Rank)/w))

- h is half the score awarded for a rank 1.
- w is how widely the scores are distributed; i.e. a low w-value might give 60/100 points for rank 2, while a high one might give 90/100. 2 seems to yield the most credible rankings (70 for rank 2, 50 for rank 3), and a higher one would place less importance on getting top ranks.

Floor(50 * 2^(((2 + 1) - Rank)/2)) for DFWC 2010:

Code: Select all

1. Nt-kop4ik              465
2. Arcaon                 372
3. glm                    262
4. Bazz!                  242
5. ZyaX                   231
6. nebuLa                 192
7. Player1                143
8. Player2                75
9. uN-DeaD!ZERG!AMT       42
10. ALIEN                 35
11. n1k                   35
12. BG-F@T@L              34
13. uN-DeaD!eThaD         31
14. GT-StolfER            24
15. uN-DeaD!Enter         23
16. cLd                   21
17. Nt_BeST!              20
18. bg-inferno            17
19. jason                 12
20. kiddy                 12
21. fakir                 11
22. fps_psych             8
23. fps-otoRp             7
24. Nightmare             6
25. DiaMo                 3
26. flo                   2
27. GT-rch                2
28. tyaz                  2
29. [fps]Exception        1
30. AKA                   1
31. cjsHiTMaN             1
32. khetti                1
33. kreator               1
34. Mystic                1
35. uN-DeaD!aurum         1

[lith]

Beautiful, just beautiful.