[Pachi] Regression ?

lemonsqueeze lemonsqueeze at free.fr
Thu Apr 7 18:13:03 CEST 2016


Hi,

I checked, with this kind of scenario it converges pretty quickly:
For 2 players with same strength, probability of getting wr <= 27% is
2% for 20 games,  0.8% for 30 games and 0.1% for 40 games.

So far i got:

S GAMES WINRATE S.D.    PAIRING
/ 34        0.441   0.085   19-7.5-1-pachi_org-pachi_32e02d4
/ 41        0.244   0.067   19-7.5-1-pachi_org-pachi_ac859d8

So i think it's safe to say ac859d8 is bad.

The other way is more tricky though:
For 2 players with say 25% wr, probability of getting wr >= 40% is
10% for 20 games, 5% for 30 games and 2% for 40 games.

So not completely sure 32e02d4 is good yet.

I couldn't find anything obviously wrong with my branch so tried 
rebasing it on top of older branch i'm using of testing, and here i got:

S GAMES WINRATE S.D. PAIRING
/ 35        0.4     0.083   19-7.5-1-pachi_upsar-pachi_dcnn_base

Looks like it's not bad by itself but something that got merged recently 
breaks it, or it's triggering a latent bug that was already there. 
Either way it looks pretty bad. I'm tempted to run the thing through 
valgrind just to make sure there's not some memory corruption going on. 
Wasn't there a way to disable the tree's custom memory allocator and use 
malloc() instead ?


On 04/04/2016 09:07 PM, uurtamo . wrote:
> Small sample statistics can work here. It's out of fashion these days,
> but there was a lot of good work that you can use.
>
> s.
>
> On Apr 4, 2016 11:49 AM, "lemonsqueeze" <lemonsqueeze at free.fr
> <mailto:lemonsqueeze at free.fr>> wrote:
>
>
>
>     On 04/04/2016 08:32 PM, Petr Baudis wrote:
>
>         On Mon, Apr 04, 2016 at 07:33:13PM +0200, lemonsqueeze wrote:
>
>             Oh wow, looks like 32e02d4 is good, but ac859d8 is bad (!)
>             Either i missed something or something funny is going on ...
>
>             S GAMES WINRATE S.D.    PAIRING
>             ? 13        0.615   0.135   19-7.5-1-pachi_org-pachi_32e02d4
>             / 19        0.211   0.094   19-7.5-1-pachi_org-pachi_ac859d8
>
>
>         Well, these are *really* small sample sizes. :)  But it could be
>         carrying
>         some information.
>
>         Are you compiling without Cafe in these tests?
>
>
>     Yes, this is without caffe compiled in (code straight from repo)
>     I hope it's just bad luck because code-wise it makes no sense to me =)
>
>     The strange thing is that 20% winrate keeps popping up pretty
>     consistently (11+15+19=45 games so far). Esp full-strength games,
>     i'd expect less variance if there's really a strength difference,
>     but maybe i'm mistaken. I'll try to play more games to find out...
>
>     _______________________________________________
>     Pachi mailing list
>     Pachi at v.or.cz <mailto:Pachi at v.or.cz>
>     http://rover.ms.mff.cuni.cz/mailman/listinfo/pachi
>


More information about the Pachi mailing list