Around 1900, a German farmer made an extraordinary claim: he had taught a horse basic arithmetic, and even to read and spell! Indeed, in public demonstrations, the horse, called Clever Hans, was able to answer all of its owner’s questions correctly, by a sequence of taps of its hooves.

A轮1900年，德国农民做出了非凡的要求 ：他教过马基本的算术，甚至阅读和拼写！ 确实，在公开示威中，这匹名为“聪明汉斯”(Clever Hans)的马能够轻敲蹄子，就能正确回答主人的所有问题。

Oskar Pfungst, a psychologist of the time at the University of Berlin, did not quite believe the animal really possessed such human-like capabilities and designed a set of experiments to prove the farmer’s claim wrong. In 1907 he published his famous report on the matter, making a surprising observation: Clever Hans would only give the correct answer if the answer was known by the experimenter asking the question!

柏林大学当时的心理学家奥斯卡·普丰格斯特(Oskar Pfungst)不太相信这种动物确实具有像人类一样的能力，并设计了一系列实验来证明农民的说法是错误的。 1907年，他发表了有关该问题的著名报告 ，做出了令人惊讶的观察：聪明的汉斯只会在问问题的实验者知道答案的情况下给出正确答案！

The horse had not actually learned to solve the question by calculating or reading, but rather it recognized subtle cues in the experimenter’s body language to figure out what answer was expected of it.

这匹马实际上并没有学会通过计算或阅读来解决问题，而是认识到了实验者肢体语言中的细微线索，以期得出对它的期望。

To this day, the phenomenon of (human or animal) test subjects’ actions in psychological studies being affected by the experimenter’s expectations is known as “Clever Hans phenomenon”. Typically, special care is taken to prevent the subjects from giving the right answer without solving the real task.

为了这一天 ，现象(人或动物)考试科目在心理学的研究行动正受到实验者的期望被称为‘聪明的汉斯现象’。 通常，要特别注意防止受试者在解决实际任务时给出正确的答案。

Wait. I thought this article was about deep learning?

等待。 我以为这篇文章是关于深度学习的？

Indeed, in modern-day, cutting-edge deep learning systems (and even simple regression) we encounter the same problem! The Clever Hans phenomenon, or in traditional statistics literature spurious correlations, can occur when there is a feature in the data that is highly correlated with the correct outcome (like the experimenter’s body language), but is not the cause for the answer being correct (e.g. a correct calculation).

确实，在当今的尖端深度学习系统(甚至是简单的回归 )中，我们遇到了同样的问题！ 当数据中的某个特征与正确的结果高度相关(例如实验者的肢体语言)，但不是答案正确的原因时，就可能发生“聪明汉斯”现象，或在传统的统计文献中出现虚假的相关性。例如正确的计算)。

A recent Nature article by Lapuschkin et al. beautifully illustrates that this effect is really occurring in modern deep learning models, e.g. for image recognition or playing ATARI games. (See below for more examples.)

Lapuschkin等人在 Nature上最近发表的一篇文章。 漂亮地说明了这种效果确实发生在现代深度学习模型中，例如用于图像识别或玩ATARI游戏。 (有关更多示例，请参见下文。)

So, why should we care what information the model is using? The model is outputting the correct answer after all.

那么，为什么我们要关心模型正在使用哪些信息？ 该模型毕竟输出正确的答案。

Well, for your training and test data it does.

好吧，对于您的培训和测试数据而言，它确实可以。

But is it still correct at deployment? Can it be fooled easily because it relies on the wrong features?

但是在部署时仍然正确吗？ 难道因为它依赖错误的功能而容易上当？

For example, imagine you have an algorithm distinguishing Wolves and Huskies, but it mainly tells them apart by the presence of snow in the image. Or, imagine you want to identify trains and boats (e.g., given the PASCAL VOC 2012 dataset), but your algorithm actually learns to recognize water and rails. This would spell disaster for any serious application. You can actually find plenty of examples in the literature if you pay attention:

例如，假设您有一个区分狼和爱斯基摩犬的算法，但主要是通过图像中雪的存在来区分它们 。 或者，假设您想识别火车和轮船(例如，给定PASCAL VOC 2012数据集 )，但是您的算法实际上学会了识别水和铁轨 。 这将对任何严重的应用造成灾难。 如果您注意以下内容，实际上可以在文献中找到很多示例：

• Identifying “father” based on the presence of a child.

  根据孩子的存在来识别 “父亲”。

• Identifying melanoma based on surgical markings.

  根据手术标记识别黑色素瘤。

• Learning dumbbells as joint concept with the hands holding them.

  学会哑铃作为联合概念用拿着他们的手。

• Classifying Asians dressed in red as ping-pong ball.

  将穿着红色的亚洲人分类为乒乓球。

• Associating neck-tie with male, and rosy cheeks with wearing lipstick.

  关联 领带与男性 ，与唇膏 脸颊红润 。

• Predicting age based on smiling and wearing glasses.

  根据微笑和戴眼镜 预测年龄。

Now, you could still argue that it’s not all that bad; after all the network is solving the task correctly, just that the problem is not posed in a good way — just fix the data! And if you do, you’re in good company. Léon Bottou, an acclaimed researcher currently at Facebook AI, thinks that fixing our problem statement can go a long way.

现在，您仍然可以辩称，这还不算太糟。 毕竟，网络可以正确地解决任务，只是问题没有以好的方式提出-只需修复数据即可！ 而且，如果您这样做，则说明您相处融洽。 目前在Facebook AI上广受赞誉的研究员LéonBottou认为，修正我们的问题陈述可能会走很长一段路。

So what’s the big deal? Why are Clever-Hans-type mistakes so bad that we have to care about them independently of typical classification mistakes?

那有什么大不了的？ 为什么“聪明汉斯”式错误如此严重，以至于我们必须独立于典型的分类错误来关心它们？

The fact is, that that it is really not easy to find out whether your network is making Clever-Hans-type mistakes, precisely because they are not reflected in the classification error. Basically, you have to rely on having a bias-free dataset or a test-set of sufficient quality. Not very reassuring, given the examples above.

事实是，要确定您的网络是否犯了Clever-Hans类型的错误，这真的不容易，这恰恰是因为它们未反映在分类错误中。 基本上，您必须依靠拥有无偏差的数据集或足够质量的测试集。 鉴于上述示例，并不是很令人放心。

If you are employing interpretability methods, and know what to look for, you might find Clever-Hans-type mistakes by just looking at what features the network is actually using. But for big datasets, this is infeasible. Researchers are starting to address the issue in a scalable way, but there is still a long way to go.

如果您正在使用可解释性方法 ，并且知道要查找的内容，则可能仅通过查看网络实际使用的功能来发现Clever-Hans类型的错误。 但是对于大型数据集，这是不可行的。 研究人员开始以可扩展的方式解决该问题 ，但是还有很长的路要走。

As researchers and practitioners, it is important to have an awareness of Clever-Hans-type mistakes and to know where to start (trying) to uncover them. Beyond that — good luck out there!

作为研究人员和从业人员，重要的是要了解聪明汉斯式的错误，并知道从哪里开始(尝试)发现错误。 除此之外，祝您好运！

I hope you learned something useful!

希望您学到了有用的东西！