Official Superfinal poster by Santiago Mendez (click for full size)
The authors of the two programs shared their thoughts in a 30 min interview with TCEC. The following is a reconstructed transcript of a Skype conference call held November 21, 2017 between Nelson Hernandez of TCEC, Robert Houdart (developer of Houdini), Mark Lefler (programmer of Komodo) and GM Larry Kaufman (developer of Komodo).
Official website / Replay games / Twitch TV channel
Nelson: One thing I wanted to clear up was whether you had ever seen each other before this call?
Robert: Well, not directly. Of course, I have seen pictures of Mark and Larry, but I haven’t spoken to them live.
Nelson: Well, in that case this is indeed an historic occasion. Gentlemen, my first question is the most obvious and what everyone wants to know. Who is going to win the Superfinal?
[awkward silence, bemused expressions, shrugs, chuckles]
Robert: Can we toss a coin for that?
Mark: For a question like that the answer is ‘yes’. [head shaking no, hands indicating ‘I don’t know’]
Nelson: I was hoping we would see a scene similar to the Liston-Clay fight in 1964 where the future Muhammad Ali stated that he was “The Greatest”. But I guess that was not to be. My next question is what you thought of the first two stages. Did you learn anything? Did anything surprise you?
Robert: Yes, well, for me one of the things I learned was in the match between Komodo and Houdini which Komodo won. It was a very surprising position where for several moves Komodo saw the win, a very clear win, and Houdini was blind, completely blind, so that was a very interesting position. I haven’t yet had time to analyze it but I certainly will because there is something I can certainly learn from that particular position.
And the other thing we learned is that that the top programs, which are Stockfish, Komodo and Houdini, are very close in strength. We knew that before the event, but the first two stages have confirmed that. It was very exciting right to the end of the Stage 2, to see which engine would get into the Superfinal and which of the three would get eliminated. So that was very exciting to see.
Mark: Yeah, only half a point difference, right? Stockfish only barely didn’t make it. Pretty tough.
Nelson: Yes, just one more win in Stage 2 and it would have finished first. Larry, do you have anything to add?
Larry: One thing I suspect as a reason that Stockfish didn’t make the final was because both Komodo and Houdini have a fairly strong form of contempt that helps us beat up on the weaker engines, and Stockfish doesn’t do that, and I imagine that cost them at least a half-point they needed.
Robert: [grinning] I was hoping we could do this conversation without mentioning contempt but that failed already after two minutes. It’s one of the most popular topics on the chat that is on the TCEC web site.
Nelson: Oh, it’s true, everyone looks for some way to make sense of results, to attribute the whole reason for an outcome to one thing. As we speak Game #6 of the Superfinal is in progress and it looks like Houdini might have a 4-2 score, which is an excellent start. Do you have any early impressions or surprises?
Robert: It’s looking good, yeah. But that’s just the throw of the dice. You can start well or you can start badly. It might have been the opposite. Komodo nearly won the second game [Game #2 replay]; it had a two pawn advantage and nearly converted the position. So it might have been 3.5-2.5 for Komodo and as it happens it is probably going to be 4-2 for Houdini. But that could change in four games’ time; we could see the opposite result and we could be back to level in the match.
Larry: Well, I noticed in the two games that were played without book in both cases for quite a number of moves black played exactly the lines that I have often played in tournaments myself, so it was for both colors, not just Komodo that did it, it was also when Houdini was black. People might say I was feeding my own moves to Houdini.
Robert: [laughs] True. That is one of the surprising things about the modern top engines, that they can play really good openings right from the start. You can start from the starting position and get a really good variation of a really good opening played by the engines. It’s not like 20 years ago where engines would play really weird openings and play really weird developing moves. That is no longer the case.
Larry: We have been seeing Komodo on its own, without a book, play the Najdorf Sicilian, which of course many people would say might be the best opening in chess for both sides.
Nelson: Mark, there’s something I’ve been meaning to ask. Clearly, Larry is a grandmaster so most of the ideas have to come from him, but you’re the programmer, is that right?
Mark: Yeah, I do the programming. We both have ideas, but whenever we come up with, say, an evaluation feature it is always Larry that comes up with the initial numbers to try, and we don’t have to tune them all that much. It seems to work fairly well if the idea is going to work.
It’s easier for me to come up with very simplistic ideas, so I can suggest a few things, but Larry is really the expert. He’s a numbers guy, and he’s also been doing this since the 1960s.
Nelson: That leads me into my next question, which is when you started chess programming and what motivated you to make this your avocation?
Larry: Well, in my case when I was a student at MIT [Massachusetts Institute of Technology] back in the 1960s. They were developing a chess program, a guy named [Richard D.] Greenblatt, was developing this program called Mac Hack, which was the first chess program to be good enough to actually compete in human tournaments. So they hired me as the chess guy for the project. I wrote the opening book, I played test games where I had to give a queen handicap. Things have changed a bit. [chuckles] And yeah, I was pretty well involved with it and that’s how I got into computer chess and Mac Hack got rated at about 1540, I think, which doesn’t sound like much now but at that time was vastly better than any other computer could have done.
Nelson: So you have been involved in computer chess for fifty or nearly fifty years?
Larry: Yeah, but I wasn’t involved continuously the whole time. But, yeah.
Nelson: Wow. How about you, Mark?
Mark: Well, I have always been interested in games and things like that. I remember going to a university library when I was in high school to read some books about computer chess that had been published throughout the years. I read everything from David Levy [ICGA] and a lot of compendiums of things for years and years. And I wrote a chess program initially in BASIC when I was in college, on an Ohio Scientific Challenger 1P, which was a 6502 machine that I could afford–the Apple was too expensive.
Since then it has been off and on. Another thing I work on is Zillions of Games, a universal game-playing program. You can create a set of rules of how the pieces move, and the board topology, and it creates an evaluation and search function from there and you can immediately play it as a game. Say you want to play chess but you want to checkmate the queen—just a couple of line changes and you’re done.
Nelson: Robert?
Robert: Well, I like chess and I like programming, and I’ve been doing that since I was a teenager, 14 or 15 years, so it was in the 1980s, so I’ve been doing this for over thirty years now. So it’s a logical step; I’ve always been interested in chess engines. The first engine I looked at was in 1984 or 1985. It was called Psion Chess. It was on a ZX81, a small computer with one kilobyte of memory. And there was already a pretty decent chess program on it, and so I decompiled it and took a look at it, because at the time using a computer was programming, actually; there were no programs unless you made them yourself. And so I had a look at that program and that was the start. Since then I’ve been as well playing chess as programming, and programming chess engines. It’s a kind of virus that gets onto you and that never releases you.
Nelson: Let me ask you something that might constitute a trade secret. I am sure you watch each other’s programs in action and detect things that casual observers might never notice or recognize. What quality of your program do you think may be superior to your opponent in the Superfinal?
Mark: I’ll let Larry take this one.
Larry: Well, okay. Basically, we have much more comparison with Stockfish because Stockfish is open source so we can easily compare our ideas and see what works better or worse. I don’t really know the inside workings of Robert’s program, but what I can tell you is that my belief is that Komodo is better in most things than Stockfish. But there is something holding us back that has to do with search depth. I don’t—we’ve been trying to figure it out for years, I don’t know what it is, but there is some reason we are not able to get the same search depth as Stockfish even if we tried to copy all their algorithms. We’ve tried experiments where we’ve tried to make Komodo act like Stockfish but it doesn’t work, and I don’t know why, but I feel that if we ever figure that out we’ll just be clearly #1. But almost every time we tried any idea from Stockfish in Komodo, nine times out of ten it makes Komodo weaker. I feel we’re doing most things better, but obviously not everything, or else we’d be stronger. I can’t say as much about Houdini because we don’t spend so much time comparing Houdini since we don’t know what’s going on with Houdini.
Robert: Ummm… let me think. I think the feature in which Houdini is better than both Komodo and Stockfish, I think, is mobility. It is piece mobility. Might be surprising, but I think the piece mobility is something which is different in Houdini than in other programs. We have some special handling of it and it results in a bit more aggressive chess actually, because you can see that when you look at the programs playing. You see that Houdini is a slightly more aggressive, tends to keep queens on the board, for instance, and I didn’t program anything particular for that, but it’s just the result of the rest of the program. It happens that because of that special mobility we can create a bit more…contempt-like…if I can again use that word, contempt-like behavior without actually having contempt. Behavior that favors keeping queens on the board, favors keeping pieces on the board, and favors aggressive play that tries to win the game. So that, I think, is one of the bigger differences between Houdini and the other programs.
Nelson: The next question may seem a bit far-fetched but I have to ask. You are all aware how society is evolving into a kind of technocratic meritocracy. You see that a lot nowadays in California, for example. It seems to me that people like yourselves, who create genuinely world-class products, are the cognitive rock stars of this age. What I am asking is how much longer can it be before people with your rare talents are on the cover of the Rolling Stone and you have an agent, an entourage and groupies?
[Larry grins, Robert chuckles, Mark makes funny faces]
Robert: It is true that, in a way, chess is still rather prestigious in a very special way. Everybody knows chess, even people who don’t actually play it, they know chess. Chess games are very often used in movies, even if the games they choose are rather bizarre—
Nelson: Yes, always with the black square in the upper left corner or the king and queen on the wrong squares.
Robert: the boldest checkmate that a super-genius player doesn’t see…but otherwise I don’t really think that we, the chess developers, will ever feature on the Rolling Stone. I really doubt that. I mean, really.
Nelson: Let’s turn for a moment to human chess. Did you all see the recent interview with world chess champion Magnus Carlsen in Germany, where one of the leading topics of discussion was computer chess?
[Robert nods, Larry and Mark have not seen it]
In case you viewers didn’t see it, Magnus said that chess programs are stronger than human beings because of their obviously superior calculation ability, but oftentimes programs make moves that give evidence that they are clueless because their algorithms which stand in for human judgment are deficient. I think none of us would disagree with that. Am I right?
Robert: Uh-huh. Well, less and less, to be honest. Ten years ago what Carlsen said was obviously true, but today it is becoming less clear whether that is still the case.
Nelson: Okay. Then Magnus went on to say something interesting that made all of us in TCEC smile a bit. He was asked if he studied games played between chess engines. He said no. He said games played between chess engines were rubbish and that finding useful ideas within them was like finding a needle in a haystack. None of you match his skill level but you are all relatively good chess players. What do you think? Are engine games rubbish?
Larry: I think what he means is that they aren’t terribly useful for the purpose of coming up with ideas for a human to use against another human. Doesn’t mean that the games aren’t incredibly high-quality. They are too subtle I guess is the point. The reason for the moves are usually nothing that is directly usable by a human player.
Nelson: Robert?
Robert: Yep. I think that is the right interpretation. But the first thing to say is: Magnus doesn’t really like chess engines. [chuckles] He is not a big fan of chess engines.
Nelson: Yeah, I got that. He seemed really uncomfortable in that interview.
Robert: Yeah… and so it’s clear that from his point of view he can’t gain a lot by studying games played by chess engines because they are at a too-high level, actually. I think they are—in terms of precision of the moves, they are probably like 300 or 400 Elo above him and so it is very difficult for him to gain some—
[Internet transmission garbled the rest of his answer, and for rest of transmission all audio has a pronounced echo]
Nelson: Robert, your answer broke up for several seconds. Could you repeat the last 15 seconds or so of your answer?
Robert: That’s fine. I think it is more important for Carlsen to focus on his own methods for finding new ideas rather than relying on chess engines to help him achieve success against human opponents.
Nelson: I know I am not the subject of this interview but I think I should weigh in on this question myself. Like you say, I think he’s right from the perspective of a human player. But I think if you look at a chess game as nothing more than a sequence of positions which contain data points, and you collect a lot of these data points, a human might have a statistical basis for exploring new ideas.
Nelson Hernandez of TCEC (upper left), Robert Houdart developer of Houdini (lower left), Mark Lefler programmer of Komodo (upper right) and GM Larry Kaufman developer of Komodo (lower right)
I have one final question which you are all uniquely qualified to talk about. What changes do you foresee coming to computer chess in the next five years and what ramifications might they have? Mark, you first.
Mark: I think there is a lot that can still be done in terms of data mining. Taking a game and trying to extract information suggesting evaluation terms or pruning ideas or extensions, things like that. They have already started doing that.
We’re also really interested in Monte Carlo. I mean, what do you do when you now have a 44-core server? What is it going to be next year? At some point more cores don’t help very much. There is a website by Andreas Strangmuller who has done a lot of experiments. We have gone one to two, two to four, up to 32 processors, Stockfish and Komodo might gain 15 to 20 Elo or something going from 16 to 32, Stockfish even less. What do you do with all this hardware to use it more effectively? In Monte Carlo [garbled] statistics that might increase your winning chances. I think those are things you could work on.
Robert: Well, I think we are all waiting for artificial intelligence to pop up in chess after having seen the success of the artificial intelligence approach of Google for the Go game. And so basically what I would expect if some of these giant corporations would be interested is that in the next five years chess also might see that kind of development. For example the artificial intelligence for the evaluation of a position, it could produce some very surprising results in chess. And so, we’re probably waiting for that and then we can retire our old engines. Look at the AlphaChess engine that will be 4000 Elo. [chuckles]
Nelson: Yep, at that point we can all fade back into history. Larry, anything to add?
Larry: Well, I also followed closely the AlphaGo situation. The guy who is the head of it at Google Mind is a chess master himself, Demis Hassabis. Although Go is thought to be a much harder game than chess to beat the best humans at, and they have certainly proven that they can do that, it is so far yet to be proven that a learning program such as the latest one from DeepMind [can replicate that in chess]. Their latest learning program beat the pants off all other, previous Go programs. But that does not apply to chess. Nobody has a self-teaching chess program that can fight with Houdini or Komodo. That’s a fantasy. Maybe that’s the challenge, to get Google to prove that it applies to chess too. But who knows.
Nelson: All right, gentlemen, thank you all for attending the first-ever TCEC Programmer’s Summit. We have made history here. May the best program win the Superfinal and may the draw-rate be comfortably below 90%.
More on the official TCEC website and the Twitch TV channel
