Official Superfinal poster by Santiago Mendez
TCEC live with analysis / Replay all the games / Video conference with the authors / Twitch TV channel
Eight decisive games in the Superfinal so far closely matches expectations. The diverse and interesting openings by Jeroen Noomen have added excitement into the high-level match. Engines play each opening with both colors and it is up to them to prove that a win can be achieved. So far neither Komodo nor Houdini have managed to score a win with black.
Here are the eight decisive games for replay with analysis:
Houdini – Komodo, Sicilian: Taimanov, 6.Be3 a6 7.Qd2 1-0
Houdini – Komodo, King’s Indian: Fianchetto, Classical, 9.h3 Qb6 10.c5 1-0
Houdini – Komodo, French: Advance, Euwe, 6.a3 1-0
Houdini – Komodo, Budapest: 3…Ng4 4.e3 1-0
Houdini – Komodo, Philidor: Improved Hanham, 7.Re1 1-0
Komodo – Houdini, Spanish: Modern Steinitz, Rubinstein Variation 1-0
Houdini – Komodo, Spanish: Modern Steinitz, Rubinstein Variation 1-0
Houdini – Komodo, Colle: 3…e6 4.Bd3 c5 1-0
Komodo Team Reports Compiler Glitch – Version Update Rejected
As per the rules, before the start of the Superfinal both participants had the chance to update their engines. Komodo submitted a new development version, upgrading from Komodo 1959.00 to Komodo 1970.00. At the beginning of the Superfinal many noticed that Komodo’s speed (in nodes per second) was lower than previously seen. Tournament director Anton Mihailov, with the help of the server administrator Martin Thoresen, double- and triple-checked that the engine was installed correctly. The details were sent to the Komodo team and everyone agreed there was no problem during the Superfinal setup.
On November 25 the Komodo team, Mark Lefler and GM Larry Kaufman, noted they believed the source of the nodes per second problem had been discovered. Here is their full explanation, a request to update Komodo, and the resolution of that question.
Statement by team Komodo
During Game 1 of the Superfinal Anton Mihailov, TCEC’s tournament director, emailed us noting significantly lower nodes per second from the version of Komodo we had used in Stage 2, 1959.00. Anton checked TCEC’s server for correct settings and found nothing amiss there.
We began our investigation by making Windows versions of many of the promotions between version 1959.00 and 1970.00 (the Superfinal version), using 23 threads on a 24 core machine. In this testing we found a roughly 8% drop in speed when running 1960.00 versus 1959.00, and similar speed drops in all the versions tested up through 1970.00. We examined the code change between 1959.00 and 1960.00 which was a minor addition to some backward pawn score (two or three lines of code).
Finding nothing in the code that could explain a significant slowdown, we recompiled the source code turning off PGO (profile-guided optimization). This did not help. Then we recompiled turning off LTO (Link Time Optimization). This resulted in restoration of the lost speed. The compiler we used throughout was MinGW 4.7.3.
It is not completely clear what exactly happened between Stage 2 version 1959.00 and 1960.00. There was no evidence of this speed reduction in 1959.00 or Komodo 11.2.2, our latest public release. We speculate that the speed reduction happens once some size of compiled code is exceeded.
It is important to point out that the approximately 8% speed reduction we noted on our best hardware (24 cores) is apparently as high as 23% on TCEC’s 44-core machine based on Komodo’s relative nodes per second vs. Houdini in Stage 2. A difference of this magnitude is what Anton noted in his original email signalling that there might be a problem.
In summary, there is indeed a slowdown in the version now running in TCEC, which appears to be due to a compiler bug. We believe that the compiler bug is more significant than any program changes since 11.2.2.
We accept full responsibility for not discovering the problem before the start of the final. Although the bug has probably cost us some points it probably does not fully explain the current five point score deficit.
We do not claim to be entitled to a program substitution at this point. Although the bug was not “our” bug, we could in theory have rented a 44-core machine for a test, or caught the much smaller slowdown on our own hardware. But I would point out that when Stockfish made a substitution last season it was due to their own bug, not a compiler bug. But this is a new season with new rules.
However, if TCEC offers to allow us to use a recompiled (with LTO off) 1970.00, or the Stage 2 compile of 1959.00, or even version 1937.00 of Stage 1 (in that order of preference) for remaining games (presumably after consulting with Houdini developer Robert Houdart) we would be happy with that substitution. If Robert wants to submit a better compile of the same version at the same time for any reason that would be fair. I think we may be too far behind to change the final match result but perhaps we could be competitive in the remaining games. If you want to wait for an even number of games or even for the halfway point in the match to allow a change that would also be fine.
Regardless of the decision, I think the viewing public is entitled to know the reason for the slowdown. It’s obvious that there is a significant slowdown and we cannot deny knowing the reason for it. I don’t think it hurts TCEC in any way to provide an explanation of an obvious problem. We’re certainly not blaming anyone other than the authors of the compiler.
Larry Kaufman and Mark Lefler, authors of Komodo
The rules of TCEC state that no engine update is possible during a stage except if an engine is behaving in a manner suggestive of critical bugs. Tournament director Anton Mihailov explained the rules and the procedure following Komodo’s request, “a critical bug is a situation where an engine is unstable in a high percentage of the games, is crashing, or is unable to produce sensible chess. Komodo’s case is far from that and, despite the lower speed, it presents very strong chess games. Thus, per the rules, Komodo cannot be updated.” However, there was a precedent during Season 8 wherein Stockfish was updated during the stage due to a bug that was not considered critical. This precedent has since become an unwritten rule of TCEC and Anton Mihailov recaps the case, “the only way an engine can get updated in the middle of a stage without presenting a critical bug is if and only if 100% of the participants in a given stage agree for another engine to use another version.”
Thus Komodo’s request was forwarded to the other participant in the Superfinal, Robert Houdart, the author of Houdini. His response:
Houdini replies to Komodo’s request
After due consideration, I cannot agree with the substitution of Komodo.
My reasons are the following:
1) About half the match has already been played – 46 games out of 100; it’s a bit too late to substitute an engine for any other reason than a critical bug.
In the current case the engine doesn’t have any critical issues; it’s just a bit slower than expected.
2) There is no precedent in TCEC for allowing an update of an engine halfway in a stage, only because there appears to be an issue which slightly reduces its strength.
I estimate that the reported 23% speed drop is equivalent to about 9 Elo at the TCEC superfinal time control and number of cores (assuming that a doubling of the speed creates about 30 Elo, which is a reasonable estimate). One could argue that this is “much ado about nothing”…
3) Houdini has been playing the whole TCEC Season 10 with the release version of Houdini 6 (in essence a version which is now 2½ months old). This is a deliberate choice; it assures that TCEC receives a high-quality, well tested build. With that in mind, it doesn’t make sense to allow other engines the luxury of correcting non-critical bugs in the middle of a TCEC stage.
Robert Houdart, Houdini Chess Engine
As per the rules and the precedent set in Season 8, tournament director Anton Mihailov has therefore ruled that Komodo will continue playing the rest of the Superfinal with the current version and no update will be allowed. He further noted, “during the ten season history of TCEC, the competition has developed one of the most comprehensive rule sets in the computer chess world. However, right after Season 10, wording of some rules will be made clearer and more detailed. As the TCEC championship grows bigger and evolves, so will the rules set.”
