See all Lex Fridman transcripts on Youtube

Speaker 1

00:00

The thing I would very much like to talk about today is the state of the art in deep learning. Here we stand in 2019

Speaker 2

00:08

really at the height of some of the great accomplishments that have happened but also stand at the beginning. And it's up to us to define where this incredible data-driven technology takes us. And so I'd like to talk a little bit about the breakthroughs that happened in 2017 and 2018 that take us to this point.

Speaker 3

00:29

So This

Speaker 2

00:32

lecture is not on the state-of-the-art results on main machine learning benchmarks. So the various image classification, object detection or the NLP benchmarks or the GAN benchmarks. This isn't about the cutting-edge algorithm that's available on GitHub that performs best on a particular benchmark.

Speaker 2

00:56

This is about ideas. Ideas and developments that are at the cutting edge of what defines this exciting field of deep learning. And so I'd like to go through a bunch of different areas that I think are really exciting. Now of course this is also not a lecture that's complete.

Speaker 2

01:15

There's other things that I may be totally missing that happened in 2017 and 18 that are particularly exciting to people here, people beyond. For example, medical applications of deep learning is something I totally don't touch on. And protein folding and all kinds of applications that there has been some exciting developments from DeepMind and so on that don't touch on. So forgive me if your favorite developments are missing but hopefully this encompasses some of the really fundamental things that have happened both on the theory side, on the application side and on the community side of all of us being able to work together on this on these kinds of technologies.

Speaker 2

01:57

I think

Speaker 1

01:57

2018

Speaker 3

02:00

in terms of deep learning is the year of natural language processing. Many have described this year as the ImageNet moment in

Speaker 1

02:08

2012

Speaker 2

02:10

for computer vision when AlexNet was the first neural network that really gave that big jump in performance on computer vision and started to inspire people what's possible with deep learning, with purely learning-based methods. In the

Speaker 3

02:23

same way, there's been a series of developments from

Speaker 1

02:26

2016, 17,

Speaker 3

02:29

and led up to 18 with the development of BERT that

Speaker 2

02:36

has made on benchmarks and in our ability to apply NLP to solve various NLP tasks, natural language processing tasks, a total leap. So let's tell the story of what takes us there. There's a few developments.

Speaker 2

02:52

I've mentioned a little bit on Monday about the encoder-decoder recurrent neural networks. So this idea of recurrent neural networks encode sequences of data and output something. Output either a single prediction or another sequence. When the input sequence and the output sequence are not the same, necessarily the same size.

Speaker 2

03:19

They're like a machine translation. We have to translate from 1 language to another. The encoder-decoder architecture takes the following process. It takes in the sequence of words or the sequence of samples as the input and uses the recurring units whether it's LSTM or GRUs or beyond and encodes that sentence into a single vector.

Speaker 2

03:47

So forms an embedding of that sentence of what it represented, a representation of that sentence. And then feeds that representation in the decoder recurrent neural network that then generates the the sequence of words that form the sentence in the language that's being translated to. So first you encode by taking a sequence and mapping it to a fixed size vector representation and then you decode by taking that fixed size vector representation and unrolling it into the sentence that can be of different length than the input sentence Okay, that's the encoder-decoder structure for recurrent neural networks It's been very effective for machine translation and dealing with arbitrary length input sequences, arbitrary length output sequences. Next step,

Speaker 3

04:43

attention. What is attention? Well it's the next step

Speaker 2

04:48

beyond, it's an improvement on the encoder-decoder architecture. It allows the, it provides a mechanism that allows to look back at the input sequence. So as opposed to saying you have a sequence that's the input sentence and that all gets collapsed into a single vector representation, you're allowed to look back at the particular samples from the input sequence

Speaker 3

05:17

as part of the decoding process. That's attention. And you

Speaker 2

05:20

can also learn which aspects are important for which aspects of the decoding process. Which aspects of the input sequence are important to the output sequence. Visualize another way, and there's a few visualizations here that are quite incredible that are done by Jay Alomar.

Speaker 2

05:45

I highly recommend you follow the links and look at the further details of these visualizations of attention. So if we look at neural machine translation, the encoder RNN takes a sequence of words and throughout, after every sequence, forms a set of hidden representations, a hidden state that captures the representation of the words that followed. And those sets of hidden representations as opposed to being collapsed to a single fixed size vector are then all pushed forward to the decoder that are then used by the decoder to translate but in a selective way where the decoder here visualized on the y-axis the input language and on the x the output language. The decoder weighs the different parts of the input sequence differently in order to determine how to best translate, generate the word that forms the translation in the full output sentence.

Speaker 2

06:52

Okay that's attention. Allowing, expanding the encoded decoder architecture to allow for

Speaker 3

07:02

selective attention to the input sequence

Speaker 2

07:05

as opposed to collapsing everything down into a fixed representation. Okay, next step, self-attention. In the encoding process, allowing the encoder to also selectively look in forming the hidden representations at other parts of the input sequence in order to form those representations.

Speaker 2

07:30

It allows you to determine for certain words what are the important relevant aspects of the input sequence that can help you encode that word the best. So it improves the encoder process by allowing it to look at the entirety of the context that's self-attention Building on that, transformer it's using the self-attention mechanism in the encoder to form these sets of representations on the input sequence. And then as part of the decoding process, follow the same but in reverse with a bunch of self-attention that's able to look back again. So it's self-attention on the encoder, attention on the decoder and that's where the magic that's where the entirety of the magic is that's able to capture the rich context of the input sequence in order to generate in a contextual way the output sequence.

Speaker 2

08:30

So let's take a step back then and look at what is critical to natural language in order to be able to reason about words, construct a language model and be able to reason about the words in order to classify a sentence, or translate a sentence, or compare 2 sentences and so on. The sentences are collections of words, or characters, and those characters and words have to have an efficient representation that's meaningful for that kind of understanding and that's what the process of embedding is. We talked a little bit about it on Monday and so the traditional Word2Vec process of embedding is you use some kind of trick in an unsupervised way to map words into a compressed representation So language modeling is the process of determining which words follow each other usually So 1 way you can use in a skip-gram model taking huge data sets of words you know there's writing all over the place taking those data sets and feeding a neural network that in a supervised way looks at which words are usually follow the input So the input is a word the output is which word are statistically likely to follow that word and the same with the preceding word And doing this kind of unsupervised learning which is what Word2Vec does if you throw away the output and the input and just take in the hidden representation form in the middle that's how you form this compressed embedding a meaningful representation that when 2 words are related in a language modeling sense 2 words are related they're going to be in that representation close to

Speaker 3

10:21

each other and when they're totally unrelated have nothing to do with each other

Speaker 2

10:24

they're far away. ELMo is the approach of using bi-directional LSTMs to learn that representation. And what bi-directional, bi-directionally.

Speaker 2

10:34

So looking not just the sequence to lead up to the word but in both directions. The sequence that followed the sequence that before. And that allows you to learn the rich full context of the word. In learning the rich full context of the word.

Speaker 2

10:49

In learning the rich full context of the word you're forming representations that are much better able to represent the statistical language model behind the kind of corpus of language that you're looking at. And this has taken a big leap in ability to then further algorithms that then with the language model a reasoning about doing things like sentence classification, sentence comparison, so on, translation, that representation is much more effective for working with language. The idea of the OpenAI transformer is the next step forward is taking the same transformer that I mentioned previously, the encoder with self-attention, decoder with attention looking back at the input sequence, and using it, taking the language learned by the decoder and using that as a language model and then chopping off layers and training on a specific language task like sentence classification. Now BERT is the thing that did the big leap in performance.

Speaker 2

12:08

With the transformer formulation, there's always, there's no bi-directional element. There is, it's always moving forward. So the encoding step and the decoding step. With BERT, it's richly bidirectional.

Speaker 2

12:23

It takes in the full sequence of the sentence and masks out some percentage of the words, 15% of the words, 15% of the samples, the tokens from the sequence. And tasks the entire encoding self-attention mechanism to predict the words that are missing. That construct and then you stack a ton of them together. A ton of those encoders self-attention feedforward network, self-attention feedforward network together.

Speaker 2

13:01

And that allows you to learn the rich context of the language to then at the end perform all kinds of tasks. You can create first of all like ELMo and like Word2Vec create rich contextual embeddings. Take a set of words and represent them in the space that's very efficient to reason with. You can do language classification, you can do sentence pair classification, you could do the similarity of 2 sentences, multiple choice question answering, general question answering, tagging of sentences.

Speaker 2

13:37

Okay I lingered on that 1 a little bit too long, but it is also the 1 I'm really excited about and really if there's a breakthrough this year has been it's thanks to BERT. The other thing I'm very excited about is totally jumping away from the NeurIPS, the theory, those kind of academic developments in deep learning and into the world of applied deep learning. So Tesla has a system called Autopilot where the hardware version 2 of that system is a implementation of the NVIDIA Drive PX2 system which runs a ton of neural networks. There's 8 cameras on the car and a variant of the Inception network is now taking in all 8 cameras at different resolutions as input and performing various tasks like drivable area segmentation, like object detection and some basic localization tasks.

Speaker 2

14:58

So you have now a huge fleet of vehicles where it's not engineers some I'm sure are engineers but it's really regular consumers people that have purchased the car have no understanding in many cases of what a neural network's limitations and capabilities are, so on. Now it has, a neural network is controlling the well-being, it's decisions, it's perceptions, and the control decision based on those perceptions are controlling the life of a human being. And that to me is 1 of the great sort of breakthroughs of 17 and

Speaker 1

15:34

18.

Speaker 2

15:36

In terms of the development of what AI can do in a practical sense in impacting the world. And so 1 billion miles, over 1 billion miles have been driven in autopilot. Now there's 2 types of systems currently operating in Tesla's.

Speaker 2

15:53

There's hardware version 1, hardware version 2. Hardware version 1 was Intel Mobileye Monocular Camera Perception System. As far as we know that was not using a neural network and it was a fixed system that wasn't learning at least online learning in the Teslas. The other is hardware version 2 and it's about half and half now in terms of the miles driven.

Speaker 2

16:14

The hardware version 2 has a neural network that's always learning, there's weekly updates, it's always improving the model, shipping new weights and so on. That's the exciting set of breakthroughs. In terms of AutoML, the dream of automating some aspects or all aspects or as many aspects as possible of the machine learning process where you can just drop in a data set that you're working on and the system will automatically determine all the parameters from the details of the architectures, the size of the architecture, the different modules in that architecture, the hyper parameters used for training the architecture, running that, doing inference, everything. All is done for you, all you just feed it is data.

Speaker 2

17:03

So that's been the success of the neural architecture search in 16 and 17 and there's been a few ideas with Google AutoML that's really trying to almost create an API where you just drop in your data set and it's using reinforcement learning and recurrent neural networks to, given a few modules, stitch them together in such a way where the objective function is optimizing the performance of the overall system. And they showed a lot of exciting results. Google showed and others that outperform state-of-the-art systems both in terms of efficiency and in terms of accuracy. Now in 18 there have been a few improvements on this direction and 1 of them is Adonet where it's now using the same reinforcement learning AutoML formulation to build ensembles on neural networks So in many cases state-of-the-art performance can be achieved by as opposed to taking a single architecture is building up a multitude, an ensemble, a collection of architectures.

Speaker 2

18:07

And that's what is doing here is given candidate architectures, stitching them together to form an ensemble to get state-of-the-art performance. Now that state-of-the-art performance is not a leap, a breakthrough leap forward, but it's nevertheless a step forward and it's a very exciting field that's going

Speaker 3

18:28

to be receiving more and more attention. There's an area of machine learning that's heavily understudied and I think it's extremely exciting area. And if you look at

Speaker 1

18:42

2012

Speaker 2

18:44

with AlexNet achieving the breakthrough performance of showing that what deep learning networks are capable of. From that point on, from 2012 to today, there's been nonstop, extremely active developments of different architectures that, even on just ImageNet alone, on doing the image classification task, have improved performance over and over and over with totally new ideas. Now on the other side, on the data side there's been very few ideas about how to do data augmentation.

Speaker 2

19:20

So data augmentation is the process of, you know, it's what kids always do when you learn about an object, right? Is you look at an object and you kind of like twist it around, is taking the raw data and messing it with such a way that it can give you much richer representation of what this data can look like in other forms, in other contexts in the real world. There's been very few developments, I think still. And this auto-augment is just a step, a tiny step into that direction that I hope that we as a community invest a lot of effort in.

Speaker 2

20:06

So what Auto Augment does is it says okay so there's these data augmentation methods like translating the image, sharing the image, doing color manipulation like color inversion. Let's take those as basic actions you can take and then use reinforcement learning and an RNN again construct to stitch those actions together in such a way that can augment data like on ImageNet to when you train on that data it gets state-of-the-art performance. So mess with the data in a way that optimizes the way you mess with the data. So, and then they've also showed that given that the set of data augmentation policies that are learned to optimize for example for ImageNet, given some kind of architecture, you can take that learned set of policies for data augmentation and apply it to a totally different data set So there's the process of transfer learning So what is transfer learning?

Speaker 2

21:17

So we talked about transfer learning you have a neural network that learns to do cat versus dog, or no, learns to do a thousand class classification problem on ImageNet and then you transfer, you chop off a few layers and you transfer on the task of your own data set of cat versus dog. What you're transferring is the weights that are learned on the ImageNet classification task and now you're then fine-tuning those weights on the specific

Speaker 3

21:49

personal cat versus dog dataset you have. Now you could do the same thing here. You can

Speaker 2

21:56

transfer as part of the transfer learning process take the data augmentation policies learned on ImageNet and transfer those. You can transfer both the weights and the policies. That's a really super exciting idea, I think.

Speaker 2

22:13

It wasn't quite demonstrated extremely well here in terms of performance. So it got an improvement in performance and so on but it kind of inspired an idea that's something that we need to really think about. How to augment data in an interesting way such that given just a few samples of data we can generate huge data sets in a way that you can then form meaningful, complex, rich representations from. I think that's really exciting and 1 of the ways that you break open the problem of how do we learn a lot from a little.

Speaker 2

22:51

Training deep neural networks with synthetic data, this also really an exciting topic that a few groups, but especially NVIDIA has invested a lot in and here's a from a CVPR 2018 probably my favorite work on this topic is they really went crazy and said okay let's mess with synthetic data in every way we could possibly can. So on the left there is showing a set of backgrounds, then there's also a set of artificial objects, and you have a car or some kind of object that you're trying to classify. So let's take that car and mess with it with every way possible. Apply lighting variation to it with every way possible.

Speaker 2

23:35

Rotate everything that is crazy. So what NVIDIA is really good at is creating realistic scenes. And they said, okay, let's create realistic scenes but let's also go way above board and not do realistic at all. Do things that can't possibly happen in reality and so generally these huge data sets I want you to train and again achieve quite interesting, quite

Speaker 3

24:01

good performance on image classification. Of course they're trying to apply it

Speaker 2

24:06

to ImageNet and so on these kinds of tasks. You're not going to outperform networks that were trained on ImageNet but they show that with just a small sample from from those real images they can fine tune this network train on synthetic images totally fake images to achieve state of

Speaker 3

24:23

the art

Speaker 2

24:23

performance. Again, another way to generate, to get to learn a lot from very little by generating fake worlds synthetically.

Speaker 3

24:37

The process of annotation, which for supervised learning, is what you need to do in order to train the network. You need to be able to provide ground truth, you need to be able to provide ground truth, you need to be able to label whatever the entity that is being learned. And so for image classification that's saying what is going on in the image.

Speaker 3

24:57

And part of that was done on ImageNet by doing a Google search for creating candidates. Now saying what's going on in

Speaker 2

25:04

the image is a pretty easy task. Then there is the object detection task of detecting the bounty box. And so saying, drawing the actual bounty box is a little bit more difficult but it's a couple of clicks and so on.

Speaker 2

25:19

Then if we take the final, the probably 1 of the higher complexity tasks of perception of image understanding is segmentation. Is actually drawing either pixel level or polygons the outline of a particular object. Now if you have to annotate that, that's extremely costly. So the work with polygon RNN is to use recurring neural networks to make suggestions for polygons.

Speaker 2

25:47

It's really interesting. There's a few tricks to form these high resolution polygons. So the idea is it drops in a single point. You draw a bounding box around an object.

Speaker 2

26:01

You use convolutional neural networks to drop the first point and then use recurrent neural networks to draw around it. And the performance is really good. There's a few tricks and this tool is available online. It's a really interesting idea.

Speaker 2

26:14

Again, the dream with AutoML is to remove the human from the picture as much as possible. With data augmentation remove the human from the picture as much as possible for menial data. Automate the boring stuff and in this case the act of drawing a polygon tried to automate it as much as possible. The interesting other dimension along which deep learning is recently been trying to be optimized is how do we make deep learning accessible?

Speaker 2

26:50

Fast, cheap, accessible. So the Dawn bench from Stanford, the benchmark, the Dawn bench benchmark from Stanford, asked, formulated an interesting competition which got a lot of attention and a lot of progress. It's saying, if we

Speaker 3

27:08

want to achieve 93% accuracy on ImageNet and 94% on CIFAR-10 let's now compete, that's like the requirement. That's not compete how you can do it in the least amount of time and for the least amount of dollars. Do the training in the least amount of time and the training in the least amount of dollars.

Speaker 3

27:29

Like literally dollars you're allowed to spend to do this. And Fast AI, you know, it's a renegade group, awesome renegade group of deep learning researchers, have been able to train on ImageNet in 3 hours. So this is for training process for 25 bucks. So training a network that achieves

Speaker 1

27:51

93%

Speaker 2

27:52

accuracy for

Speaker 1

27:54

$25

Speaker 2

27:56

and 94% accuracy for $0.26 on CIFAR-10. So the key idea that they were playing with is quite simple but really boils down to messing with the learning rate throughout the process of training. So the learning rate is how much you based on the loss function, based on the error the neural network observes how much do you adjust the weights?

Speaker 2

28:21

So they found that if they crank up the learning rate while decreasing the momentum which is a parameter of the optimization process where they do it that jointly they're able to make the network learn really fast. That's really exciting and the benchmark itself is also really exciting because that's exactly for people sitting in this room that opens up the door to doing all kinds of fundamental deep learning problems without the resources of Google DeepMind or OpenAI or Facebook or so on, without computational resources. That's important for academia, that's important for independent researchers and so on.

Speaker 3

29:05

So GANs. There's been a lot of work on generative adversarial neural networks and in some ways there has not been breakthrough ideas in GANs for quite a bit. And I think big GAN from Google DeepMind ability to generate Incredibly high resolution images And it's the same GAN technique So in terms of breakthroughs innovations But scaled So

Speaker 2

29:41

increase the model capacity And increase the batch size the number of images that are fed

Speaker 3

29:48

to the network. It produces incredible images. I encourage you to go online and look at them.

Speaker 3

29:55

It's hard to believe that they're generated. So that was, so 2018 for GANs was a year of scaling and parameter tuning as opposed to break through new ideas Video to video synthesis This work is from NVIDIA is looking at the problem so there's been a lot of work on going from image to image. So from a particular image generating another image. So whether it's colorizing an image or just traditionally defined GANs.

Speaker 3

30:36

The idea with video-to-video synthesis that a few people have been working on but Nivea took a good step forward is to make the video, to make the temporal consistency, the temporal dynamics part of the optimization process. So make it look not jumpy. So if you look here at the comparison for this particular, So the input is the labels in the top left and the output of the NVIDIA approach is on the bottom right See it's very temporally consistent. If you look at the image to image mapping that's state of the art, pix2pix HD it's very jumpy, it's not temporally consistent at all.

Speaker 3

31:26

And there's some naive approaches for trying to maintain temporal consistency that's in the bottom left. So you can apply this to all kinds of tasks, all kinds of video to video mapping. Here is mapping it to face edges, edge detection on faces, mapping it to faces, generating faces from just edges. You can look at body pose to actual images.

Speaker 3

31:55

As input to the network you can take the pose of the person and generate the video of the person. Okay, semantic segmentation. The problem of perception sort of began with AlexNet and ImageNet has been further and further developments where the input, the problem is of basic image classification where the input is an image and the output is a classification what's going on in that image. And the fundamental architecture can be reused for more complex tasks like detection, like segmentation and so on.

Speaker 3

32:35

Interpreting what's going on in the image. So these large networks from VGG net, Google net, ResNet, SC net, DenseNet, all these networks are forming rich representation that can then be used for all kinds of tasks, whether that task is object detection. This here shown is the region-based methods where the neural network is tasked, the convolutional layers make region proposals, so a bunch of candidates to be considered and then there's a step that's determining what's in those different regions and forming bounding boxes around them

Speaker 2

33:12

in a for loop way and then there is the 1 shot method, single shot method where in a single pass all of the bounding boxes in their classes are generated and there has been a tremendous amount of work in the space of object detection. Some are single-shot methods, some are region-based methods and there's been a lot of exciting work but not, I would say, breakthrough ideas. And then we take it to the highest level of perception which is semantic segmentation.

Speaker 2

33:48

There's also been a lot of work there. The state-of-the-art performance is, at least for the open source systems, is DeepLab v3 plus

Speaker 3

34:00

on the Pascal VLC challenge. So semantic segmentation to catch it all up started in 2014 with fully convolutional neural networks chopping off the fully connected layers and then outputting the heat map very grainy, very

Speaker 2

34:20

low resolution. Then improving that with SegNet, performing max pooling with a breakthrough idea that's reused in a lot of cases is dilated convolution, 8-chars convolutions.

Speaker 3

34:34

Having some spacing which increases the field of view of the convolutional filter. The key idea behind DeepLab V3 that is the state of the art is the multi-scale processing without increasing the parameters. The multi-scale is achieved by quote-unquote, the atrius rate.

Speaker 3

34:58

So taking those atrius convolutions and increasing the spacing. And you can think of the increasing that spacing

Speaker 2

35:06

by enlarging the model's field of view and so you can consider all these different scales of processing and looking at the layers of features

Speaker 3

35:19

so allowing you to be able to grasp the greater context as part of the upsampling deconvolutional step.

Speaker 2

35:27

And that's what's producing the state of our performances and that's where we have the

Speaker 3

35:32

notebook tutorial on GitHub showing this deep lab architecture trained on Cityscapes. So Cityscapes is a driving segmentation data set that is 1 of the most commonly used for the task of driving scene segmentation. Okay, on the deep reinforcement learning front.

Speaker 3

36:08

So this is touching a bit, a bit on the

Speaker 1

36:11

2017,

Speaker 3

36:13

but I think the excitement really settled in 2018. As the work from Google and from OpenAI DeepMind. So it started in DQN paper from Google DeepMind where they beat a bunch of Atari games achieving superhuman performance with deeper enforcement learning methods that are taking in just the raw pixels of the game.

Speaker 3

36:38

So this same kind of architecture is able to learn how to beat these games. Super exciting idea that kind of has echoes of what general intelligence is, taking in the raw information and being able to understand the game, the sort of physics of the game sufficient to be able to beat it. Then in 2016, AlphaGo, with some supervision and some playing against itself, self-play. Some supervised learning on expert world champ players and some self-play where it plays against itself.

Speaker 3

37:15

It was able to beat the top, the world champion at Go. And then

Speaker 1

37:19

2017

Speaker 3

37:20

AlphaGo 0, a specialized version of AlphaZero, was able to beat the AlphaGo with just a few days of training and 0 supervision from expert games. So through the process of self-play, again this is kind of getting the human out of the picture more and more and more, which is why AlphaZero is probably or this AlphaGo 0 was the demonstration of the cleanest demonstration of all the nice progress in deep reinforcement learning. I think if you look at the history of AI, when you're sitting on a porch 100 years from now, sort of reminiscing back, AlphaZero will be a thing that people will remember as an interesting moment in time, as a key moment in time.

Speaker 3

38:16

And AlphaZero was applied in

Speaker 1

38:19

2017

Speaker 3

38:20

to beat, AlphaZero paper was in 2017 and it was this year played Stockfish in chess which is the best engine, chess playing engines, is able to beat it with just 4 hours of training. Of course the 4 hours, there's caveats because 4 hours for Google DeepMind is highly distributed training. So it's not 4 hours for an undergraduate student sitting in their dorm room.

Speaker 2

38:48

But meaning

Speaker 3

38:51

it was able through self-play to very quickly learn to beat the state-of-the-art chess engine and learn to beat the state-of-the-art shogi engine, Elmo. And the interesting thing here is you know with perfect information games like chess, you have a tree and you have all the decisions you could possibly make and so the farther along you look at along that tree presumably the better you do. That's how Deep Blue beat Kasparov in the 90s is you just look as far as possible down the tree to determine which is the action is the most optimal If you look at the way human grandmasters think it certainly doesn't feel like they're like looking down a tree There's something like creative intuition There's something like you could see the patterns in the board you can do a few calculations but really it's in the order of hundreds It's not on the order of millions or billions which is kind of the the stockfish, the state-of-the-art chess engine approach and AlphaZero is moving closer and closer closer towards the human grandmaster concerning very few future moves It's able through the neural network estimator that's estimating the quality of the move and the quality of the different the quality, the current quality of the board and the quality of

Speaker 2

40:11

the moves that follow it's able to do much much less look ahead. So the neural network learns the fundamental information. Just like when a grandmaster looks at a board, they can tell how good that is.

Speaker 2

40:25

So that's again, interesting. It's a step towards, at least echoes of what human intelligence is in this very structured, formal, constrained world of chess and Go and Shogi. And then there's the other side of

Speaker 3

40:40

the world that's messy. It's still games, it's still constrained in that way, but open AI has taken on the challenge of playing games that are much messier, that have this semblance of the real world and the fact that you have to do teamwork, you have to look at longtime horizons with huge amounts of imperfect information, hidden information, uncertainty. So within that world they've taken on the challenge of a popular game Dota

Speaker 1

41:11

2.

Speaker 3

41:13

On the human side of that There's the competition, the International, hosted every year, where in 2018 the winning team gets $11 million. It's a very popular, very active competition that's been going on for a few years. They've been improving and achieved a lot of interesting milestones.

Speaker 3

41:34

In 2017 they were 1v1 bot, beat the top professional Dota 2 player. The way you achieve great things is you try. In

Speaker 1

41:45

2018

Speaker 3

41:46

they tried to go 5v5, The OpenAI 5 team lost 2 games against the top 2, the top Dota 2 players at the

Speaker 1

41:55

2018

Speaker 3

41:56

International. And of course, their ranking here, the MMR ranking in Dota 2 has been increasing over and over but there's a lot of challenges here that make it extremely difficult to beat the human players and This is you know in every story Rocky or whatever you think about losing is an essential element of a story that leads to then a movie and a book and greatness. So you better believe that they're coming back next year and there's going to be a lot of exciting developments there. It's also Dota 2 and this particular video game makes it currently, there's really 2 games that have the public eye in terms of AI taking on as benchmarks.

Speaker 3

42:42

So we solved Go, incredible accomplishment. But what's next? So last year, the associate with the best paper in Europe's it was the heads up Texas no limit Hold'em. AI was able to beat the top level players.

Speaker 3

43:01

What's completely current, well not completely, but currently out of reach is the general, not heads up 1 versus 1, but the general team. Texas no limit hold them. There you go. And on the gaming side, this dream of Dota 2, Now that's the benchmark that everybody's targeting and it's actually incredibly difficult 1 and some people think it'll be a long time before we can win.

Speaker 3

43:25

And on the more practical side of things, the

Speaker 1

43:30

2018

Speaker 3

43:33

starting 2017 has been a year of the frameworks growing up, of maturing and creating ecosystems around them. With TensorFlow, with the history there dating back a few years, has really with TensorFlow

Speaker 1

43:53

1.0

Speaker 3

43:55

has come to be a mature framework PyTorch 1.0 came out in 2018 it's matured as well and now the really exciting developments in TensorFlow with eager execution and beyond that's coming out in TensorFlow 2.0 in

Speaker 1

44:12

2019. So really

Speaker 2

44:15

those 2 players have made incredible leaps

Speaker 3

44:20

in standardizing deep learning.

Speaker 2

44:25

And the fact that a lot of

Speaker 3

44:26

the ideas I talked about today and Monday and we'll keep talking about all have a GitHub repository with implementations in TensorFlow and PyTorch making it extremely accessible and that's really exciting. It's probably best to quote Jeff Hinton the quote-unquote godfather of deep learning 1 of the key people behind backpropagation said recently of backpropagation is my view is throw it all away and start again. He believes backpropagation is totally broken and an idea that is ancient and needs to be completely revolutionized.

Speaker 3

45:03

And the practical protocol for doing that is he said the future depends on some graduate student who's deeply suspicious of everything I've said. That's probably a good way to end the discussion about what the state of the art in deep learning holds. Because everything we're doing is fundamentally based on ideas from the 60s and the 80s and really in terms of new ideas there's not been many new ideas. Especially the state of the art results that I've mentioned are all based on fundamentally on stochastic gradient descent and back propagation.

Speaker 3

45:46

It's ripe for totally new ideas. So it's up to us to define the real breakthroughs and the real state of the art 2019 and beyond. So with that, I'd like to thank you And the stuff is on the website, deeplearning.mit.edu. You