The Men In Suits

[peterbirks]

About nine years ago I think I said that I trusted Paradise Poker to be honest more than I did my bank. And it was something I meant. I generated good income for them. There were probably only about 20,000 active players (I still remember the 50 millionth hand promotion, which at the time got a record 3,000 people logged in at the same time -- compared with the 180,000 you get these days at the weekend on Pokerstars) and Paradise treated you well. You dealt with real people who understood poker.

But, times changed. First there was the Ultimate Bet "superuser" scandal, and then the concept of "poker skins" developed. Not only did this change the economic dynamic of online poker sites (where there is one host of the technical stuff, and a number of "agents" who run the front ends for the punters) but it also brought in a few fly-by-nights. As is always the case when there's money around, some shady characters emerged.

But Pokerstars remained aloof from this, and it profited from it. First it overtook Party Poker as the biggest site, and then it moved on to dominate. From bing the place where people went for tournaments, it became the home of choice for the 2+2 multi-table grinder.

Then, when the UIGEA laws came in, Pokerstars did not bottle it, as did the by-now listed Party Poker (the men in suits got in there early and did their best to totally fuck it up). And Pokerstars did prosper goodly.

But, around the end of last year, things seemed to change. Basically, Pokerstars had become too big, and it wanted to become bigger. It was targeting the emerging nations, a fertile collection of fields as far as it was concerned. Many of the poker guys who had been there a long time began to feel that they were outnumbered by the new "numbers" guys who were coming in. The men in suits had taken over.



I think this is inevitable to most growing companies, and the result is usually disastrous. Look what a mess Apple got into when the men in suits took over. Only when the original "inventors" came back did the original spark return.

I don't know why Lee Jones jacked it in at Pokerstars, but I do know that the man loves poker, and, like most lovers of poker, he is not a "big business" guy. Lee Jones would never have made it at IBM, unless he had joined it in the 1920s.

There are, I think, three kinds of employees:
1) employees who like being in small companies because they can "make a difference".
2) employees who like being in big companies because "if you fuck up, no-one notices", and
3) people who want to work in the public sector.

Discarding the last, that leaves two distinct types of private sector wannabees, and these two types will never get on.

Pokerstars has moved into the "big" side of the fence, and employees who are instinctively on the "small" side have gradually been crowded out by the typical "big company" employee.

The good news is that it hasn't taken too long for the shit to hit the fan. One reason for this is that the poker "consuming" community is less forgiving than virtually any other community. And it also happens to have a small core who, if a company fucks up, will be shown to have fucked up. It's a classic case of the people on the outside being cleverer and more competent than the people on the inside.

Two scandals have broken almost simultaneously.

First, "Chiren80", a multi-tabler who came from the World of Warcraft sector (where he was "Athene") and was given a "Pokerstars team player" badge (equals a ton of rakeback), got kind of fed up with spending his entire life 12- to 24-tabling in order to keep the extremely valuable Supernova Elite six-star status. Being quite a bright chap, he spotted that the new "micromania" promotion dreamt up by some incompetents in the Pokerstars marketing department (see "big employee" syndrome, above) had a bit of a flaw in it. Basically, the victory points being awarded at this level were extremely generous. Since Chiren got 100% rakeback, he didn't need to worry that much about the rake. All he had to do was play a ton of hands against another Supernova Elite player, bashing the money back and forth (and settling up any difference at the end) and generating victory points (those things which define your player level on Pokerstars) in the process. In a sense, it saved Chiren a lot of time, and time (as I well know), is the most valuable thing when it comes to maintaining a high player level on Pokerstars.

I should point out that this is not about pennies. I am a Supernova and maintaining even that level is a struggle -- it equals about $18k a year in rake. For Chiren, he needs to pay about $180k a year in rake.

But the benefits are large -- I'm chalking up about $700 to $800 in benefits from $1250 a month in rake. This is four times what I would get at the "base" Bronze Star level. And Chiren's benefits are even higher than that.

Not being stupid, Chiren asked Stars if he could play a "hyper-aggressive" style at lower stakes. Here comes incomeptent level two at Stars, because they said, Yes, he could. No alarm bells ranng because, well, Chiren understands poker, understands points accumulation (shit, he came from World of Warcraft!) and can spot a flaw in the rules a mile away.

So he sits down and plays against one other player in this "hyper-aggressive" style, generating gazillions of victory points for both of them. As soon as anyone else sat down at the table, they sat out.

Where Chiren went wrong was that the people on 2+2 aren't stupid, and they spotted it.

http://forumserver.twoplustwo.com/28/internet-poker/pokerstars-pro-chiren80-dumping-back-forth-vpps-829120/

Initially Chiren tried to spin the same line that he had played with Pokerstars (and which Pokerstars believed), but that got nowhere. 2+2 readers are a little bit sharper than that.

And, lo and behold, Pokerstars eventually posts to the forum that Chiren would not be doing this any more.

+++++

The second balls-up by Stars was even more hilarious and worrying. Malloc posted that he had noticed that three players who had played a ton of hands with very similar styles (9%/9% shortstackers in Full Ring Games) had moved down in stakes at the same time, and then, a month or so later, had moved down in stakes again.

Now, any person who plays a lot of online poker would say that this screams "Bot".


This was the first response from the now famous "Pokerstars Jeff". I somehow get the feeling that Jeff is one of the "big company" employees that I talked about earlier.

We have separate complex systems in place to automatically look for collusion, and for bots. Each is comprised of far more powerful servers than you can buy in a typical business/end-user machine, and each is dedicated solely to sifting through mountains of data to seek out potential suspects in their respective realms of responsibilities.

Exactly what they do and what they look at, we're just not at liberty to say.

Regards,

Jeff
PokerStars Game Security, Senior Bot Specialist

Unfortunately, Pokertableratings, without access to the hole cards, was able to produce proof, within 24 hours, that there was indeed a 10-man bot-ring at work, and that it had been at work for 8 million hands or so. The stats that it showed were not only convincing, but they were easy to find -- including the most damning 0% check-call and 0% check-raise on flop, plus 0% bet-raise(by opponent)-call.


http://www.pokertableratings.com/blog/2010/07/bot-ring-discovered-on-poker-stars/


One of the comforting thing about big-company employment is that everyone else around you doesn't give a shit either -- which is why incompetent people in big companies survive so long -- it's a conspiracy of silence going right the way up the tree.

Unfortunately in the land of online poker, that doesn't apply. It's not just your boss who you have to keep happy, and brown-nosing doesn't work with the hounds of death that are 2+2 contributors.

For a while all was silence from Pokerstars -- fair enough, in a sense, in that it was night-time in the Isle of Man and then the weekend.

Over the night of Friday the 10-man bot team continued playing. I suspect that they had already withdrawn most of their $60k-plus-benefits of winnings.

The 2+2 thread was running very hot on early Saturday morning my time (late night the US, mid-morning in Russia and old USSR states, where these bots were playing from).


Then it got even odder. Pokerstars Jeff, clearly unaware of the "when you are in a hole, stop digging" rule, posted the following:

Hello 2+2,

PokerStars would like to update you on the status of this case, and correct several misconceptions that have been perpetuated throughout this thread.

First, as of the afternoon of July 17, we have indeed removed ten players from the site for violations of our terms of service. We cannot give additional details about which accounts, or the specific reasons for their removal.

Second, we would like to clarify something that the OP has left out of this entire discussion, which is that there were two separate investigations here. PokerStars absolutely did *not* exonerate these players from being bots at any time. We conducted a review of the three users listed in the OP for collusion, and it was negative, and this was reported to the OP, who posted an incorrect "PokerStars said "All Clear'" here. A review of the players for being bots was concurrently being conducted, and was in progress until today, July 17th. The OP was notified of this concurrent bot review being in progress, and did not inform this thread of that fact.

We would like to address the fact that the play continued throughout the night of July 16th, after a third party source posted an independent review identified 7 additional players. PokerStars had also identified 7 additional players very early in our investigation -- prior to the third party site's identification of them. We were conducting a very in depth search to locate other potential accounts, to ensure we had located all of the accounts prior to closing them down concurrently. Shutting down only part of a ring in mid-investigation can spook additional, undiscovered accounts into cashing out prior to locating them. As of this afternoon, July 17, that review concluded that the ten accounts were the complete extent of the matter.

Next, please note that third party sites did not locate this ring unassisted. They had a baseline of three identical player profiles from which to start, which is a far more trivial task than saying "here's a collection of 4 million players... now, find me 3 who are identical.... without a baseline from which to search." One is a linear search (searching for a particular pattern) and the other is exponentially more difficult based upon the number of players. PokerStars located the rest of the ring faster than did the third party site.

...........

Finally, to address why these players played for a relatively extended period without detection, we can say that upon conclusion of this review that the bots were fairly sophisticated in certain aspects, including their human behavior modeling and their stealth capabilities. Our tools for bot detection are sophisticated, but they are not perfect. No site can guarantee you that they are completely free of bots. What PokerStars can and does pledge is that we use our extensive tools behind the scenes to detect bots as best we can... and they are very effective. We detect and remove most bots well before they even leave the development stage, and well before they could play long enough to come to the attention of players or third party databases the way these players did.

PokerStars takes its commitment to bot detection and removal extremely seriously, and will continue to do so, both behind the scenes with our tools (which are under constant review and improvement), and in response to player-generated suspicions. Most player reports of bot suspicions are false alarms, but we still investigate each one thoroughly, and when they actually do locate a previously undetected bot, appropriate action will always be taken.

Best Regards,

Jeff
PokerStars Game Security

Most of this post was greeted with a combination of disbelief, abuse and sarcasm (plus, of course, the inevitable Photoshops and photos of Heath Robinson-types and 50-year old computers hard at work trying to detect bots).

But the worst part of it was this was actually libellous (well, I think that it was libellous). It's stated quite explicitly that Malloc had lied in his original post and misrepresented Pokerstars' response to his enquiry -- that Stars had only said that these players were not colluding, but that a concurrent investigation into whether they were bots was continuing.

Malloc was having none of this:

Nowhere in that email does it say that an additional, parallel investigation into the possibility of bots is being undertaken.

Congrats, Jeff, you've massively pissed off another Supernova (2 years running) by lying to me and about me.

Which prompted this prompt response, presumably with two Isle of Man lawyers holding guns to the back of his head:

My sincere apologies, malloc. I did not intend to call you out or "piss you off". It is clear that you were not informed of a continuing investigation on the 13th, and haven't participated in this thread since I Emailed you on the 16th.

With sincere apologies,

Jeff
PokerStars Game Security

Oh my, this is almost textbook stuff of a big company fucking up. (And, should I add, all credit for malloc, who simply typed "thanks for the clarification" and took it no further. I would have been on the phone to my lawyers.)

++++

In a way this reminds me of AIG, a company that became the biggest insurer in the US and whose employees developed a reputation for arrogance along the lines of "we're the biggest". The employees from the old days were replaced by jobsworths who liked the comfort of big companies. It all went wrong for AIG. Pokerstars really ought to be careful that it doesn't head down the same route. One thing is definitely clear -- the people who are currently in charge are not the right people, and it's the senior management who should be replaced, not the likes of Pokerstars Jeff, who is merely a "big company" kind of employee who got himself into a situation where he was quite obviously completely out of his depth.

Don't sack him. Sack the people who put him there.

___________

Comments

[jiggery-pokery.livejournal.com]

How disappointing that the reaction to Chiren is so negative!

There are, from time to time, extremely lucrative schemes in the world of frequent flyer miles connection, and the person who devises them and demonstrates them normally gets vast kudos for this. In one sense I suppose that (a) this exploit relies on the fact that he is a Pokerstars team player and thus the activity is +EV (or, at least, +utility) for him when it wouldn't be +utility for someone trying it without being a Pokerstars team player, and thus everyone else who doesn't have that status is just jealous, and (b) poker players have a more direct notion of the concept of a zero-sum game than flyers, even though frequent flight currency gets devalued from time to time in a comparable way to the increase of rake, and thus is not so far from frequent play currency after all.

A more down-to-earth example considers occasions when Tesco give away lots of Clubcard points for buying unlikely yoghurts, etc., which can be redeemed at four times their nominal value with some retailers and thus the original purchase is immediately +EV. This probably illustrates how overpriced most of the things being given away as rewards are in the first place, rather than anything else.

[peterbirks.livejournal.com]

Well, I actually thought of writing on this very topic, but I had gone on long enough already. Essentially, poker players are biased with extreme prejudice against chip-dumping. Now, actually , Chiren wasn't really chip-dumping; he was VPP generating. It just happened to involve what looked like chip-dumping. But there are many terms and conditions in playing online, and generating VPPs through "fake" playing is very much against the rules. This was why Chiren pretended he was just playing "hyper aggressive". He didn't mention to Pokerstars that he knew his opponent (3J55) in real life. neither did he mention that he and 3J55 had a "split the EV swings" agreement (one paid the other depending on who was running relatively better compared to EV, or who was running relatively less badly!). All of this is grey area stuff where games players would probably say "perfectly fine", but poker players, who, believe it or not, have rather strict unwritten "codes of conduct", would say "no, it's out of order". But other poker players (Brunson, for example, who played as part of a team with Amarillo Slim and Sailor Roberts back in the 1950s) would be rather more hesitant to cast the first stone.

Of course, the real people who "should be punished" are in the marketing department, not Chiren and 3J55. But Chiren clearly knew that he was on dodgy ground. Else, why deceive Pokerstars about his real intention?

PJ

[jellymillion.livejournal.com]

The Chiren thing is silly - he should be given a pat on the back, the rules of the promotion should be changed and he should perhaps be invited onto a games-lawyer vetting panel to check future proposals for evidence of brain-death.

I'd love to know how the bot detection stuff works - there are a number of techniques I'd enjoy testing, had I access to the data.

Of course, the problem with sacking the people who should be carrying the can is that they're the sackers, not the sackees - they pay people to carry cans for them.

[peterbirks.livejournal.com]

I suspect that 'Stars "Fraud detection" techniques are now based on those of the banks, rather than having been built by poker players. So, first they look for all of those things that banks and insurers look for when it comes to fraud -- same IP addresses (same physical addresses), links between surnames, and so on. Bots are not so high up the investigation chain because, quite simply, it makes Stars money rather than costs them money. I have in the past said that I don't think bots should be outlawed, but I do think that they should be identified. Even better, have bot v bot tables, which individuals would be allowed to join if they so desired. It's not the botulism that annoys me, it's the concealment. Computer chess programs are easier to play against when you know that they are computer chess programs. Bots are the same. The trick is identifying them as such.

PJ

[real-aardvark.livejournal.com]

Dunno about banks, mate, but I've built a Credit Risk system for Visa in my time, and it paid no attention whatsoever to all that crap.

Of course, a credit-card risk analysis system is nicely constrained by the fact that, well, you need a credit card number to commit fraud. Building a history off this is naturally quite simple. All you have to do then is to match it against a Model (not my capitalisation; Visa's) which tracks against generic behavioural profiles.

(In the case of credit cards, it's quite easy to build a basic Model. At one end you've got unusual, but possibly legitimate, behaviour -- buying a diamond-studded camel saddle, for example, if you live in Neasden -- and at the other end you've got typical fraudster behaviour, a simple example of which is to use the credit card to buy ten quids' worth of petrol fifteen minutes after you've ripped off £1000. There are subtleties, but you don't need to know them, and I can't tell you because of black helicopters and such.)

Crossing over to poker sites, I don't see a huge behavioural difference. Naturally, you're going to want the Model to be Bayesian (I know how much you love that stuff), and naturally you're going to want to compare profile against profile, rather than a single history against a single profile. Catching a bot ring using this sort of technique should be fairly straight-forward: in fact, in the case you quote, almost automatic and instantaneous (if there's any such thing as "instantaneous" in a Bayesian world).

The Model would, I estimate, require around 50,000 lines of Java or C# or C++, and configuration files -- to be updated as necessary -- stretching to maybe 5000 lines. Given the huge granularity of poker transactions (I mean, you have to play the whole hand, not just stick a piece of plastic in a slot), I estimate from experience that you could probably fit the whole thing on a mid-range Solaris with report-time of, say, a couple of minutes from detection. You're looking at £100,000 for hardware and a couple of thousand man-days at £300 per (contract rates, UK) ... actually, now I come to think about it, £700,000 is quite a lot of money.

Particularly if bots don't affect your bottom line directly.

[real-aardvark.livejournal.com]

That diamond-studded camel saddle, btw? I didn't make it up. (Although obviously it didn't happen in Neasden.)

One of the 'Stans, I forget which, had a median credit card transaction of $5000. Now, considering that the median credit card transaction (yr 200ish) across the G20 was between $10 and $30, that's quite the disparity. Part of it was explicable by the number of worthies in said 'Stan who qualified for a credit card, but from the point of view of fraud, that's somewhat of a petitio principii.

In an echo of Pokerstars' "Not really our problem, mate," Visa International (as then was) contacted the financial authorities of the 'Stan in question and pointed out that, according to the Model, 99% of their credit card transactions were fraudulent. The Bank of 'Stan's response?

"It's just the way we do business here."

[slowjoe.livejournal.com]

Great post, BTW. Has the chip dumping thing been blocked? Otherwise, I might farm some VPPs myself.

The pokertableratings detection was essentially trivial.

They calculated the difference between pairs of players by taking a set of standard stats (VPIP, PFR, Flop CBET, Yadda Yadda) and calculating the difference, then squaring that difference and summing them. They called it the Euclidean distance or difference or something.

That's the clone-detection problem solved, then, until the bot writers start including mixed strategies deliberately. But how the hell were Pokerstars trying to detect bots previously? Clearly, they weren't looking for clones at all.

Men in suits

[e584.livejournal.com]

I don't really buy the big firm/small firm angle, to be honest - what little awareness I had of AIG came from the early 2000's and I thought Hank Greenberg came across as a nasty piece of work - I see it as the same arrogance that brought Enron about.

And the pokerstars contact that I have had makes me think that the the employees there are less sector 7g drones as they are just genuine people who are a) not rocket scientists and b) not really poker players. It's not ideal, but I'd still prefer a big site over a small one from the pov of fraud/security.

I do agree re the skins angle though - I find scary the idea of the poker economy not being 6 people round a table playing their own bankrolls with the sites taking their 5% $3 cap, or whatever, but instead some people round a table with rakeback accounts through an affiliate, half of whom are staked by a third party, some are in 'makeup' (which sounds a bit like indentured servitude tbph) playing on an interface provided by one company through to an engine run by another, either or both of whom are sponsoring name players to play in said games. Too complicated by half.

Re: Men in suits

[peterbirks.livejournal.com]

There are clearly degrees of "big company/small company" preference (after all, I work for a big company because I used to work for a small company that got bought out!)

Your analysis of the Stars staff is probably about right, and I would not want it to be imagined that I am backing small poker companies over Stars. The collusion on small sites is almost certainly worse (I hardly play on any small sites now, so I can't be sure), and bots have been operating on Pacific for ages in the short-stack games (see Tipolina for a good example).

But Stars really did put its foot in it s couple of times here, and I don't think that the "Stars of old" would have made the same mistakes.

PJ