Ian Goodfellow Interview吴恩达与伊恩·古德费罗的访谈

[Andrew] Hi, Ian. Thanks a lot for joining us today.

[Ian] Thank you for inviting me, Andrew. I am glad to be here.

[Andrew] Today, you are one of the world's most visible deep learning researchers. Let us share a bit about your personal story. So, how do you end up doing this work that you now do?

[Ian] Yeah. That sounds great. I guess I first became interested in machine learning right before I met you, actually. I had been working on neuroscience and my undergraduate adviser, Jerry Cain, at Stanford encouraged me to take your Intro to AI class.

[Andrew] Oh, I didn't know that. Okay.

[Ian] So I had always thought that AI was a good idea, but that in practice, the main, I think, idea that was happening was like game AI, where people have a lot of hard-coded rules for non-player characters in games to say different scripted lines at different points in time. And then, when I took your Intro to AI class and you covered topics like linear regression and the variance decomposition of the error of linear regression, I started to realize that this is a real science and I could actually have a scientific career in AI rather than neuroscience.

[Andrew] I see. Great. And then what happened?

[Ian] Well, I came back and I was the TA to your course later.

[Andrew] Oh, I see. Right. Like a TA.

[Ian] So a really big turning point for me was while I was TA-ing that course, one of the students, my friend Ethan Dreifuss, got interested in Geoff Hinton's deep belief net paper. I see. And the two of us ended up building one of the first GPU CUDA-based machines at Stanford in order to run Watson machines in our spare time over winter break.

[Andrew] I see.

[Ian] And at that point, I started to have a very strong intuition that deep learning was the way to go in the future, that a lot of the other algorithms that I was working with, like support vector machines, didn't seem to have the right asymptotics, that you add more training data and they get slower, or for the same amount of training data, it's hard to make them perform a lot better by changing other settings. At that point, I started to focus on deep learning as much as possible.

[Andrew] And I remember Richard Reyna's very old GPU paper acknowledges you for having done a lot of early work.

[Ian] Yeah. Yeah. That was written using some of the machines that we built. Yeah. The first machine I built was just something that Ethan and I built at Ethan's mom's house with our own money, and then later, we used lab money to build the first two or three for the Stanford lab.

[Andrew] Wow that's great. I never knew that story. That's great. And then, today, one of the things that's really taken the deep learning world by storm is your invention of GANs. So how did you come up with that?

[Ian] I've been studying generative models for a long time, so GANs are a way of doing generative modeling where you have a lot of training data and you'd like to learn to produce more examples that resemble the trading data, but they're imaginary. They've never been seen exactly in that form before. There were several other ways of doing generative models that had been popular for several years before I had the idea for GANs. And after I'd been working on all those other methods throughout most of my Ph.D., I knew a lot about the advantages and disadvantages of all the other frameworks like Boltzmann machines and sparse coding and all the other approaches that have been really popular for years. I was looking for something that avoid all these disadvantages at the same time. And then finally, when I was arguing about generative models with my friends in a bar, something clicked into place, and I started telling them, You need to do, this, this, and this and I swear it will work. And my friends didn't believe me that it would work. I was supposed to be writing the deep learning textbook at the time, I see. But I believed strongly enough that it would work that I went home and coded it up the same night and it worked.

[Andrew] So it take you one evening to implement the first version of GANs?

[Ian] I implemented it around midnight after going home from the bar where my friend had his going-away party.

[Andrew] I see.

[Ian] And the first version of it worked, which is very, very fortunate. I didn't have to search for hyperparameters or anything.

[Andrew] There was a story, I read it somewhere, where you had a near-death experience and that reaffirmed your commitment to AI. Tell me that one.

[Ian] So, yeah. I wasn't actually near death but I briefly thought that I was. I had a very bad headache and some of the doctors thought that I might have a brain hemorrhage. And during the time that I was waiting for my MRI results to find out whether I had a brain hemorrhage or not, I realized that most of the thoughts I was having were about making sure that other people would eventually try out the research ideas that I had at the time.

[Andrew] I see. I see.

[Ian] In retrospect, they're all pretty silly research ideas.

[Andrew] I see.

[Ian] But at that point, I realized that this was actually one of my highest priorities in life, was carrying out my machine learning research work.

[Andrew] I see. Yeah. That's great, that when you thought you might be dying soon, you're just thinking how to get the research done.

[Ian] Yeah. Yeah. That's commitment.

[Andrew] Yeah. Yeah. Yeah. So today, you're still at the center of a lot of the activities with GANs, with Generative Adversarial Networks. So tell me how you see the future of GANs.

[Ian] Right now, GANs are used for a lot of different things, like semi-supervised learning, generating training data for other models and even simulating scientific experiments. In principle, all of these things could be done by other kinds of generative models. So I think that GANs are at an important crossroads right now. Right now, they work well some of the time, but it can be more of an art than a science to really bring that performance out of them. It's more or less how people felt about deep learning in general 10 years ago. And back then, we were using deep belief networks with Boltzmann machines as the building blocks, and they were very, very finicky. Over time, we switched to things like rectified linear units and batch normalization, and deep learning became a lot more reliable. If we can make GANs become as reliable as deep learning has become, then I think we'll keep seeing GANs used in all the places they're used today with much greater success. If we aren't able to figure out how to stabilize GANs, then I think their main contribution to the history of deep learning is that they will have shown people how to do all these tasks that involve generative modeling, and eventually, we'll replace them with other forms of generative models. So I spend maybe about 40 percent of my time right now working on stabilizing GANs.

[Andrew] I see. Cool. Okay. Oh, and so just as a lot of people that joined deep learning about 10 years ago, such as yourself, wound up being pioneers, maybe the people that join GANs today, if it works out, could end up the early pioneers.

[Ian] Yeah. A lot of people already are early pioneers of GANs, and I think if you wanted to give any kind of history of GANs so far, you'd really need to mention other groups like Indico and Facebook and Berkeley for all the different things that they've done.

[Andrew] So in addition to all your research, you also coauthored a book on deep learning. How is that going?

[Ian] That's right, with Yoshua Bengio and Aaron Courville, who are my Ph.D. co-advisers. We wrote the first textbook on the modern version of deep learning, and that has been very popular, both in the English edition and the Chinese edition. We've sold about, I think around 70,000 copies total between those two languages. And I've had a lot of feedback from students who said that they've learned a lot from it. One thing that we did a little bit differently than some other books is we start with a very focused introduction to the kind of math that you need to do in deep learning. I think one thing that I got from your courses at Stanford is that linear algebra and probability are very important, that people get excited about the machine learning algorithms, but if you want to be a really excellent practitioner, you've got to master the basic math that underlies the whole approach in the first place. So we make sure to give a very focused presentation of the math basics at the start of the book. That way, you don't need to go ahead and learn all that linear algebra, that you can get a very quick crash course in the pieces of linear algebra that are the most useful for deep learning.

[Andrew] So even someone whose math is a little shaky or haven't seen the math for a few years will be able to start from the beginning of your book and get that background and get into deep learning?

[Ian] All of the facts that you would need to know are there. It would definitely take some focused effort to practice making use of them.

[Andrew] Yeah. Yeah. Great. If someone's really afraid of math, it might be a bit of a painful experience. But if you're ready for the learning experience and you believe you can master it, I think all the tools that you need are there. As someone that worked in deep learning for a long time, I'd be curious, if you look back over the years. Tell me a bit about how you're thinking of AI and deep learning has evolved over the years.

[Ian] Ten years ago, I felt like, as a community, the biggest challenge in machine learning was just how to get it working for AI-related tasks at all. We had really good tools that we could use for simpler tasks, where we wanted to recognize patterns in how to extract features, where a human designer could do a lot of the work by creating those features and then hand it off to the computer. Now, that was really good for different things like predicting which ads a user would click on or different kinds of basic scientific analysis. But we really struggled to do anything involving millions of pixels in an image or a raw audio wave form where the system had to build all of its understanding from scratch. We finally got over the hurdle really thoroughly maybe five years ago. And now, we're at a point where there are so many different paths open that someone who wants to get involved in AI, maybe the hardest problem they face is choosing which path they want to go down. Do you want to make reinforcement learning work as well as supervised learning works? Do you want to make unsupervised learning work as well as supervised learning works? Do you want to make sure that machine learning algorithms are fair and don't reflect biases that we'd prefer to avoid? Do you want to make sure that the societal issues surrounding AI work out well, that we're able to make sure that AI benefits everyone rather than causing social upheaval and trouble with loss of jobs? I think right now, there's just really an amazing amount of different things that can be done, both to prevent downsides from AI but also to make sure that we leverage all of the upsides that it offers us.

[Andrew] And so today, there are a lot of people wanting to get into AI. So, what advice would you have for someone like that?

[Ian] I think a lot of people that want to get into AI start thinking that they absolutely need to get a Ph.D. or some other kind of credential like that. I don't think that's actually a requirement anymore. One way that you could get a lot of attention is to write good code and put it on GitHub. If you have an interesting project that solves a problem that someone working at the top level wanted to solve, once they find your GitHub repository, they'll come find you and ask you to come work there. A lot of the people that I've hired or recruited at OpenAI last year or at Google this year, I first became interested in working with them because of something that I saw that they released in an open-source forum on the Internet. Writing papers and putting them on Archive can also be good. A lot of the time, it's harder to reach the point where you have something polished enough to really be a new academic contribution to the scientific literature, but you can often get to the point of having a useful software product much earlier.

[Andrew] So read your book, practice the materials and post on GitHub and maybe on Archive.

[Ian] I think if you learned by reading the book, it's really important to also work on a project at the same time, to either choose some way of applying machine learning to an area that you are already interested in. Like if you're a field biologist and you want to get into deep learning, maybe you could use it to identify birds, or if you don't have an idea for how you'd like to use machine learning in your own life, you could pick something like making a Street View house numbers classifier, where all the data sets are set up to make it very straightforward for you. And that way, you get to exercise all of the basic skills while you read the book or while you watch Coursera videos that explain the concepts to you.

[Andrew] So over the last couple of years, I've also seen you do one more work on adversarial examples. Tell us a bit about that.

[Ian] Yeah. I think adversarial examples are the beginning of a new field that I call machine learning security. In the past, we've seen computer security issues where attackers could fool a computer into running the wrong code. That's called application-level security. And there's been attacks where people can fool a computer into believing that messages on a network come from somebody that is not actually who they say they are. That's called network-level security. Now, we're starting to see that you can also fool machine-learning algorithms into doing things they shouldn't, even if the program running the machine-learning algorithm is running the correct code, even if the program running the machine-learning algorithm knows who all the messages on the network really came from. And I think, it's important to build security into a new technology near the start of its development. We found that it's very hard to build a working system first and then add security later. So I am really excited about the idea that if we dive in and start anticipating security problems with machine learning now, we can make sure that these algorithms are secure from the start instead of trying to patch it in retroactively years later.

[Andrew] Thank you. That was great. There's a lot about your story that I thought was fascinating and that, despite having known you for years, I didn't actually know, so thank you for sharing all that.

[Ian] Oh, very welcome. Thank you for inviting me. It was a great shot.

[Andrew] Okay. Thank you.

[Ian] Very welcome.

[Andrew] 嗨，Ian。非常感谢你今天加入我们。

[Ian] 感谢你的邀请，Andrew。我很高兴能来到这里。

[Andrew] 如今，你是世界上最知名的深度学习研究者之一。让我们分享一些你的个人故事吧。那么，你是如何走上现在这条研究道路的呢？

[Ian] 是的，这听起来很不错。我想我第一次对机器学习产生兴趣是在遇见你之前。我一直在研究神经科学，我的本科导师Jerry Cain鼓励我参加你在斯坦福大学的人工智能入门课程。

[Andrew] 哦，我不知道这件事。好的。

[Ian] 所以我一直认为人工智能是个好主意，但在实践中，我认为主要的想法是像游戏AI，人们为游戏中的非玩家角色编写了很多硬编码规则，以便在不同的时间点说出不同的脚本台词。然后，当我参加了你的人工智能入门课程，你讲解了线性回归和误差方差分解等主题时，我开始意识到这是一门真正的科学，我可以在人工智能领域而不是神经科学领域拥有一个科学的职业生涯。

[Andrew] 我明白了。太好了。那后来发生了什么？

[Ian] 嗯，后来我回来担任了你课程的教学助理。

[Andrew] 哦，我明白了。对，就像教学助理一样。

[Ian] 所以对我来说一个很大的转折点是，当我担任这门课程的教学助理时，其中一名学生，我的朋友Ethan Dreifuss，对Geoff Hinton的深度信念网络论文产生了兴趣。我明白了。我们两个最终在斯坦福大学建立了最早的基于GPU CUDA的机器之一，以便在寒假期间运行Watson机器。

[Andrew] 我明白了。

[Ian] 那时，我开始强烈地感觉到深度学习是未来的发展方向，其他很多算法，比如支持向量机，似乎没有正确的渐进性，当你增加更多的训练数据时它们会变慢，或者对于同样数量的训练数据，很难通过改变其他设置来显著提高它们的表现。从那时起，我开始尽可能多地专注于深度学习。

[Andrew] 我记得Richard Reyna非常早期的一篇关于GPU的论文中提到了你，因为你做了很多早期的工作。

[Ian] 是的，是的。那是使用我们构建的一些机器撰写的。是的，我构建的第一台机器是Ethan和我在他妈妈家里用我们自己的钱建造的，后来我们使用实验室的资金为斯坦福实验室建造了最初的两到三台。

[Andrew] 哇，太棒了。我从没听说过这个故事。很棒。然后，今天，真正席卷深度学习界的一件事是你发明的生成对抗网络（GANs）。那么你是怎么想到这个的？

[Ian] 我研究生成模型已经很久了，所以GANs是一种生成建模的方法，当你有很多训练数据并且你想学习生成更多类似于交易数据的示例，但这些示例是虚构的，以前从未以那种形式出现过。在我提出GANs之前，有几种其他流行的生成模型方法已经流行了好几年。在我整个博士生涯中研究所有这些其他方法之后，我非常了解所有其他框架如Boltzmann机器和稀疏编码以及其他多年来非常流行的方法的优缺点。我在寻找一种同时避免所有这些缺点的方法。最后，当我在一个酒吧里和朋友们争论生成模型时，灵光一闪，我开始告诉他们，你需要做这个、这个和这个，我发誓它会奏效。而我的朋友们不相信它会奏效。我当时应该正在写深度学习教科书，我明白了。但我坚信它会奏效，所以我当晚回家就把它编码实现了，结果它奏效了。

[Andrew] 所以只用了一个晚上你就实现了第一版GANs？

[Ian] 是的，我从酒吧回家后大约在午夜实现的，那时我的朋友正在举办他的告别派对。

[Andrew] 我明白了。

[Ian] 第一版就奏效了，这非常非常幸运。我不需要搜索超参数或其他任何东西。

[Andrew] 我在哪里读到过一个故事，说你有一次濒死经历，这重新坚定了你对AI的承诺。告诉我那个故事。

[Ian] 是的。实际上我并没有濒死，但我一度以为我会死。我头痛得非常厉害，一些医生认为我可能脑出血。在我等待MRI结果以确定是否有脑出血的时候，我意识到我脑海中的大部分想法都是确保其他人最终会尝试我当时的研究想法。

[Andrew] 我明白了。我明白了。

[Ian] 回想起来，那些研究想法都挺傻的。

[Andrew] 我明白了。

[Ian] 但那时我意识到，这实际上是我生命中最重要的优先事项之一，就是进行我的机器学习研究工作。

[Andrew] 我明白了。是的。这很好，当你以为自己可能很快就会死去时，你却在想如何完成研究。

[Ian] 是的。

[Andrew]是的。这就是承诺。

[Ian] 是的。

[Andrew]是的。所以今天你仍然处于GANs和生成对抗网络活动的中心。那么告诉我你如何看待GANs的未来。

[Ian] 现在，GANs被用于很多不同的事情，比如半监督学习、为其他模型生成训练数据，甚至模拟科学实验。原则上，所有这些任务都可以由其他种类的生成模型来完成。所以我认为GANs现在正处于一个重要的十字路口。目前它们有时效果很好，但要真正发挥它们的性能更像是一门艺术而非科学。这或多或少就像十年前人们对深度学习的总体感觉。那时我们使用深度信念网络作为基本构件，它们非常挑剔。随着时间的推移，我们转向了像修正线性单元和批归一化这样的东西，深度学习变得更加可靠。如果我们能使GANs变得像深度学习那样可靠，那么我认为我们会继续看到GANs在今天所有使用它们的地方取得更大的成功。如果我们无法弄清楚如何稳定GANs，那么我认为它们对深度学习历史的主要贡献将是展示人们如何完成所有这些涉及生成建模的任务，最终我们将用其他形式的生成模型取代它们。所以我现在大约40%的时间都在致力于稳定GANs。

[Andrew] 我明白了。很酷。好的。哦，所以就像大约十年前很多人加入深度学习领域一样，比如你自己，最终成为了先驱者，或许今天加入GANs的人，如果一切顺利，也可能成为早期的先驱。

[Ian] 是的。已经有很多人都是GANs的早期先驱，我认为如果你想给出GANs到目前为止的任何历史记录，你真的需要提到其他团队，比如Indico、Facebook和伯克利大学，因为他们在很多不同的事情上都做出了贡献。

[Andrew] 所以除了你的研究，你还与人合著了一本关于深度学习的书。这本书进行得怎么样？

[Ian] 没错，与我的博士联合导师Yoshua Bengio和Aaron Courville一起写的。我们写了第一本关于现代深度学习版本的教科书，这本书非常受欢迎，无论是英文版还是中文版。我认为两版总共卖出了大约70,000本。我从学生那里得到了很多反馈，他们说从中学到了很多。我们与其他书籍略有不同的是，我们从深度学习所需的数学基础开始进行非常集中的介绍。我认为我从你在斯坦福的课程中学到的一点是，线性代数和概率非常重要，人们会对机器学习算法感到兴奋，但如果你想成为一名真正优秀的从业者，你必须首先掌握整个方法背后的基本数学。所以我们确保在书的开头非常集中地介绍数学基础知识。这样，你就不需要提前学习所有的线性代数，你可以快速掌握对深度学习最有用的线性代数部分。

[Andrew] 那么即使某人的数学基础有些薄弱，或者已经几年没接触数学了，他们也能从你的书开头开始学习这些背景知识并进入深度学习领域吗？

[Ian] 你需要知道的所有事实都在那里。当然，要实践使用它们确实需要一些专注的努力。

[Andrew] 是的。是的。太好了。如果有人真的很害怕数学，这可能是一次痛苦的经历。但如果你准备好学习并且相信你能掌握它，我认为你需要的所有工具都在那里。作为一个长期从事深度学习工作的人，我很好奇，回顾这些年，你对AI和深度学习的思考是如何演变的？

[Ian] 十年前，我觉得作为一个社区，机器学习面临的最大挑战就是如何让它在AI相关任务中发挥作用。我们有非常好的工具可以用于更简单的任务，比如预测用户会点击哪些广告或进行各种基本的科学分析。但在涉及数百万像素的图像或原始音频波形的情况下，系统必须从头构建所有的理解，这真的很难。大约五年前，我们彻底克服了这个障碍。现在，我们处于一个有很多不同路径可以选择的阶段，任何想进入AI领域的人面临的最大问题可能是选择哪条路径走下去。你是想让强化学习像监督学习那样运作良好吗？你想让无监督学习像监督学习那样运作良好吗？你想确保机器学习算法是公平的，不反映我们希望避免的偏见吗？你想确保围绕AI的社会问题得到很好的解决，确保AI让每个人都受益而不是导致社会动荡和失业问题吗？我认为现在有非常多不同的事情可以做，既可以防止AI带来的负面影响，也可以确保我们充分利用它提供的所有好处。

[Andrew] 所以如今，有很多人想进入AI领域。对此你有什么建议吗？

[Ian] 我认为很多想进入AI领域的人开始认为他们绝对需要获得博士学位或其他类似的资格认证。我不认为这实际上是必要的。一种吸引注意的方式是编写好的代码并发布到GitHub上。如果你有一个有趣的项目解决了某个高层人员想要解决的问题，一旦他们找到你的GitHub仓库，他们会来找你并请你去那里工作。我去年在OpenAI或今年在Google招聘的很多人，我第一次对他们感兴趣是因为我看到他们在互联网上的开源论坛上发布了一些东西。写论文并发布到arXiv上也是个好办法。很多时候，要达到有足够的成果成为新的学术贡献给科学文献更难，但你通常更早就能达到拥有一个有用的软件产品的地步。

[Andrew] 所以阅读你的书，练习材料并在GitHub上发布，也许还可以在arXiv上发布。

[Ian] 我认为如果你通过阅读书籍来学习，同时进行一个项目是非常重要的，要么选择某种方式将机器学习应用于你已经感兴趣的领域。比如，如果你是一个野外生物学家并想进入深度学习，也许你可以用它来识别鸟类，或者如果你不知道如何在自己的生活中使用机器学习，你可以选择像制作街景门牌号码分类器这样的东西，所有数据集都已经设置好了，让你非常容易上手。这样你就可以在阅读书籍或观看解释概念的Coursera视频时练习所有基本技能。

[Andrew] 在过去的几年里，我还看到你做了更多关于对抗性示例的工作。告诉我们一些这方面的情况。

[Ian] 是的。我认为对抗性示例是一个新领域的开端，我称之为机器学习安全。在过去，我们看到计算机安全问题时，攻击者可以欺骗计算机运行错误的代码。这被称为应用级安全。还有一些攻击，人们可以欺骗计算机相信网络上的消息来自实际上不是他们声称的那个人。这被称为网络级安全。现在，我们开始看到，你也可以欺骗机器学习算法做它们不应该做的事情，即使运行机器学习算法的程序是正确的，即使运行机器学习算法的程序知道网络上所有消息的真实来源。我认为，在新技术发展的初期就构建安全是非常重要的。我们发现，先建立一个工作系统然后再添加安全性是非常困难的。所以我真的对这个想法感到兴奋，如果我们现在就投入并预见到机器学习的安全问题，我们可以确保这些算法从一开始就是安全的，而不是试图在几年后进行修补。

[Andrew] 谢谢。讲得非常好。关于你的故事有很多我觉得非常有趣，尽管认识你多年，但我实际上并不知道这些，所以谢谢你分享这一切。

[Ian] 哦，非常欢迎。谢谢你邀请我。这是一个很棒的机会。

[Andrew] 好的。谢谢你。

[Ian] 非常欢迎。