The Bain Experiment

The purpose of this experiment was to test Dan Heisman’s basic tactics training ideas, using the problems from John Bain’s Chess Tactics for Students.  In this experiment, I not only greatly improved at solving the problems that I was practicing, but also at solving problems that I had never seen before.  Bain's book has 14 chapters:

Chapter  Problems   Motif
   1       2-31     Pins
   2      33-62     Back Rank
   3      64-93     Knight Forks
   4      95-124    Double Attack
   5     126-155    Discovered Checks
   6     157-186    Double Checks
   7     188-217    Discovered Attacks
   8     219-248    Skewers
   9     250-279    Double Threats
  10     281-310    Promoting Pawns
  11     312-341    Removing the Guard
  12     343-372    Perpetual Check
  13     374-403    Zugswang
  14     405-434    Identifying Tactics

(The first problem in each chapter is an illustration that duplicates the first problem of that chapter.)  The book is formatted as a workbook.  Beneath each diagram, the book says who it is to move, and gives hint for finding the solution.  There are no solutions, as such, but the hints are so detailed that a solution is hardly necessary.  The reverse side of each diagram is set aside for writing the solution, so I was able to cut out the diagrams along with their hints, doing only minor damage to some of the hints on the other side.  I folded back the hints so that they were not visible, and marked each diagram with W+, W=, W#, B+, B=, or B#, according to the result required, taking care not to look at the diagrams. (This caused some problems.  There are a few cases where the book says e.g. “win a rook”, and it is possible to win material in another way.  I gave myself the benefit of the doubt in these cases.)

The book says that the problems within each chapter are in order of difficulty.  I constructed six batches A-F of problems from Chapters 1 to 13.  Batch A consisted of problems 2, 8, 14, 20, 26..., plus problems 33, 39, 45, 51, 57..., and so on.  Batch B consisted of problems 3, 9, 15, 21, 27..., plus problems 34, 39, 46, 52, 58..., and so on. The remaining batches were constructed in the same way.  This method ensured that, as nearly as possible, each batch had the same number of problems with the same level of difficulty from each chapter.  If the book’s ordering by level of difficulty was perfect, there would be a very slight increase in difficulty from batch to batch.  Any remaining variation in difficulty between the batches can reasonably be ascribed to random factors.  I set Chapter 14 aside, because sorting the problems by motif would have given me prior knowledge, and I had a suspicion (which seems to be right) that they are duplicates of problems from previous chapters.  I thoroughly shuffled each batch - which was almost entirely ineffective - so I scrambled the order by sorting them into two piles and then into three.  I discarded problem 26, because it worked only if the opposing side blundered; and problem 95, because it was a complete dud.

The early part of my schedule was:

Day 1: A+B, A+B
Day 2: A+B, C+D
Day 4: A+B, C+D
Day 6: C+D
Day 8: A+B, C+D
Day 9: E+F

The first four repetition intervals for batches A+B were ½ day, 1 day, 2 days and 4 days, and the first three repetition intervals for batches C+D were 2 days, 2 days and 2 days.  I measured the time taken to solve each problem with a stopwatch, rounding the times to a tenth of a second.  In the diagrams below, I counted the times of any incorrect solutions as taking more than 30 seconds, whatever the actual time spent.  The cumulative distributions of solution times for the first four passes through batches A+B were:

I was clearly faster at the outset than Heisman’s typical student, and improved more rapidly.  However, a large proportion of Bain’s problems were either simple examples from Reinfeld, or simplified versions of more complicated ones, so I had an unfair advantage here!  (N.B. You can click on the diagrams to enlarge them.)  The histogram of solution times for the first five passes through A+B was:

Note the very rapid initial improvement from Pass 1 to Pass 2, which was carried out on the same day.  I got 83% in under 5 seconds on the fifth pass.  (N.B.  For the diagrams in this section, 0-5 stands for 0-4.9 seconds, and similarly for the other “buckets”.)  The histogram of solution times for the first four passes through C+D was:

Note the slow initial progress with the two day repetition intervals.  Most of my difficulty was with D rather than C, so I did a quick untimed pass of D, before my first pass of E+F, to reduce my times for C+D closer to those of A+B.  (I had hoped that the experiment would show an improvement on my first pass from A to B, from C to D, and from E to F. An improvement was observed from A to B, but I did worse on D than C and on F than E, so I do not believe that we can draw any conclusions here.)

Both the ½ day, 1 day , 2 days, 4 days, and the 2 days, 2 days. 2 days repetition intervals worked well here.  The intervals of 1 day, 2 days, 4 days used in the Reinfeld Experiment should give much the same results as 2 days, 2 days, 2 days (see the earlier article on that experiment).  It is not clear whether the additional repetition at ½ day would still have a significant benefit after many more repetitions at progressively doubling intervals.

Not surprisingly, I improved at the problems that I was practicing - but what about problems that I had never seen before?  Here is a histogram of solution times for my first passes through A+B, C+D and E+F:

I was astonished!  On my first pass through E+F, I got over 85% in 15 seconds, and very nearly 50% in 5 seconds.  What was going on?  One theory is that I simply got faster as a result of practice.  Another is that I learned new patterns by repeatedly solving A+B that helped me with B+C, and that repeatedly solving B+C taught me still more patterns, that helped me further with E+F.  What does the data have to say?

The simplest hypothesis is that the solution times were all reduced by a common factor.  I found that dividing all the solution times for my first pass through A+B by 1.3 gave the closest match to the solution times for my first pass through C+D.  Similarly, dividing all the solution times for my first pass through A+B by 2.6 gave the closest match to those on my first pass through E+F.  (I used the method of least squares here.)  The histogram below compares the counts in each “bucket” for my real passes through C+D and E+F with those simulated by dividing the solution times of A+B:













The fit is about as good as it could be, given the statistical variability.  It is remarkable that my first pass of E+F was 2.6 times faster than my first pass of A+B.

What about the pattern matching theory?  Imagine that x% of the problems in A+B are duplicated in C+D, and that any internal duplication within C+D is at the same level as that in A+B.  On the first pass of C+D, I will have solved x% of the problems three times already, and the remainder will be new to me.  I can approximate my performance on the x% by using the histogram for the third pass of A+B.  My performance on the remaining problems within C+D can be approximated by the histogram for my first pass through A+B.  I used the method of least squares to find the value of x% which made this approximation as close as possible to the histogram for my first pass through C+D.  The best fit was with x% = 25%.  I also approximated my first pass through E+F using the histograms for my first and fifth passes through A+B - the best fit was obtained with x% = 48% - which is almost exactly twice the value for C+D, as it should be if duplicates are equally distributed throughout the batches.  The histogram below compares my real passes through C+D and E+F with the approximated ones:












Again, the fit is good.  Bain has many problems that are the same as another problem within the book, but with one less move at the beginning.  If I tackle the harder problem first, the easier one should show up as a duplicate, but if I tackle the easier one first, the harder one may not show up as a duplicate.

What conclusions can we draw?  The data is consistent with my matching patterns that I already knew 2.6 times faster.  Since Bain’s problems were either very simple - or examples from Reinfeld that I already knew - it is possible that I already knew all the patterns, and had just become faster at finding them.  However, I am sure that I am not 2.6 times faster at solving all tactics problems at this level of complexity.  I believe that most of my improvement was pattern specific.  This interpretation is supported by the fact that my improvement in going from my first pass of A+B to my first pass of E+F was almost exactly twice my improvement in going from my first pass of A+B to my first pass of C+D.  (N.B. If the experiment had been carried out on a set of problems that had the same statistical profile as simple tactics in real games, my improvement would be real, whether or not it resulted from pattern duplication.)  It could be objected that, despite my best efforts, E+F might be easier than C+D, which might in turn be easier than A+B.  It is not possible to completely eliminate possibilities like this from a single player experiment. Please feel free to repeat the experiment with the batches in the reverse order!

For an update, see my later article: Basic Tactics Revision.