Why bigger training sets must yield diminishing returns: An information-theoretic speed limit for AI

It is now well established that AI systems suffer diminishing performance returns as the number of training items is increased. Given that a fundamental limit on the performance of any system is the amount of Shannon information in its training set, we prove that adding one item to a training set of n items cannot provide more than ∆H = 0.5 log2 ((n + 1)/n) bits of additional information. Because ∆H shrinks rapidly as n increases, it is inevitable that the performance of any AI system suffers from diminishing returns as n is increased.