（一）（一）（一）（一）（一）
Full Stack 深度学习 ｜ Full Stack Deep Learning 2022 p01 P01_Lecture_01-何时使用机器学习及课程愿景 -BV1k4YXznEjw_p1-
Hey everyone， welcome to the 2022 edition of Fulltack deep learning。 I’m Josh Tobin。
one of the instructors。 and I’m really excited about this version of the class because we’ve made a bunch of improvements and I think it comes at a really interesting time to be talking about some of these topics。
Let’s dive in today we’re gonna cover a few things First we’ll talk about why this course exists and what you might hope to take away from it then we’ll talk about the first question you should ask when you’re starting a new M project which is should we be using M for this at all and then we’ll talk through the highlevel overview of what the lifecycle of a typical ML project might look like。
which will also give you a conceptual outline for some of the things we’ll talk about in this class。
Really what is full stack deep learning about We aim to be the course and community for people who are building products that are powered by machine learning and I think it’s a really exciting time to be talking about ML power products because machine learning is rapidly becoming a mainstream technology and you can see this in startup funding in job postings as well as in the continued investment of large companies in this technology。
I think it’s particularly interesting to think about how this has changed since 2018 when we started。
Teaching the class in 2018， a lot of the most exciting ML powered products were built by the biggest companies。
You had self-driving cars that were starting to show promise。
you had systems like translation from big companies like Google that were really starting to hit the market in a way that was actually effective but the broader narrative in the field was that very few companies were able to get value out of this technology and even on the research side now G3 is becoming a mainstream technology。
but in 2018， G1 was one of the state of the art examples of language models and if you look at what it actually took to build a system like this it was the code in the standardization around it was still not there like these technologies were still hard to apply now on the other hand。
there’s a much wider range of really powerful products that are powered by machine learning Dolly is I think a great example image generation technology on the consumer side Tiktok is a really powerful example but it’s not just massive companies now that are able to build machine learning powered products Descript is an application that we at full stack deep learning use all the time。
In fact， we’ll probably use it to edit this video that I’m recording right now and startups are also building things like email generation So there’s a proliferation of machine learning powered products and the narrative that shifted I think a little bit as well which is that before these technologies were really hard to apply but now there’s standardization that’s emerging both around the technology stack transformers and NLP starting to seep their way into more and more use cases as well as the practices around how to actually apply these technologies in the world one of the biggest changes in the field in the past four years has been the emergence of this term called MLOs which we’ll talk a lot about in this class and so if you asked yourself why why is this changed so rapidly I think in addition to the field just maturing and research continuing to progress I think one of the biggest reasons is that the training of models is starting to become commoditized we showed a couple of slides ago how complicated code for GT1 was now using something like huggingface you can deploy a state of the art NLP model or computer vision model in one or two lines of code on top of that。
Aml is starting to actually work for a lot of applications I think four years ago。
were pretty skeptical about it。 now I think it’s a really good starting point for a lot of problems that you might want to solve and companies are starting to provide models really as a service where you don’t even have to download open source package to use it you can just make a network call and you can have predictions from a state of the art model and on the software side a lot of frameworks are starting to standardize around things like Kais and pytorch lightning So a lot of the like spaghetti code that you had to write to build these systems just isn’t necessary anymore And so I think if you project forward a few years what’s going to happen in M。
I think the history of the ML is characterized by rise and fall of the public perception of the technology these were driven by a few different AI winters that happened over the history of the field where the technology didn’t live up to its height live up to its promise and people became skeptical about it what’s going to happen in the future of the field I think what a lot of people think is that this time is different we have real applications。
Of machine learning that are generating a lot of value in the world。
And so the prospect of a true AI winter where people become skeptical about AI as a technology maybe feels less likely。
but it’s still possible。 a slightly more likely outcome is that the overall luster of the technology starts to wear off。
but certain applications are getting a ton of value out of this technology。
And then I think the upside outcome for the field is that AI continues to accelerate really rapidly and it becomes pervasive and incredibly effective。
And I think that’s also what a lot of people believe to happen。
And so what I would conjecture is that the way that we as a field avoid an AI winter is by not just making progress in research。
but also making sure that that progress is translated to actual real-wor products。
That’s how we avoid repeating what’s happened in the past that’s caused the field to lose some of its luster。
But the challenge that presents is that building m powered products requires a fundamentally different process in many ways than building the types of M systems you create in academia sort。
Process that you might use to develop a model in an academic setting。
I would call flat earth machine learning flat Earth M。
this is a process thatll probably be familiar to many people。 you start by selecting a problem。
you collect some data to use to solve the problem。
you clean and label that data you iterate on developing a model until you have a model that performs well on the dataset that you collected and then you evaluate that model and if the model performs well according to your metrics。
then you write a report， produce a Jupyter notebook a paper or some slides and then you’re done but in the real world。
the challenge is that if you deploy that model in production。
it’s not going to perform well in the real world for long necessarily and so ML powered products require this outer loop where you deploy the model into production。
you measure how that model is performing when it interacts with real users。
you use the real world data to build data flywheel and then you continue this as part of an outer loop。
some people believe that the Earth isn’t round just because you can’t see the outer loop in the M system doesn’t mean it’s not there And so this course is really about how to do this process of building ML powered products。
What we won’t cover as much is the theory in the math and the sort of computer science behind deep learning and machine learning more broadly。
there are many great courses that you can check out to learn those materials we also will talk a little bit about training models and some of the practical aspects of that but this isn’t meant to be your first course in training machine learning models again there’s many great classes for that as well but what this classes about is the unique aspects that you need to know beyond just training models to build great ML power products so our goals in the class are to teach you a generalist skill set that you can use to build an MLP product and an understanding of how the different pieces of ML power products fit together we will also teach you a little bit about this concept of MLOs but this is not an MLOs class Our goal is to teach you enough MLOs to get things done but not to cover the full depth of MLOs as a topic we’ll also share best practices from what we’ve seen to work in the real world and try to explain some of the motivations behind them and if you’re on the job marketer if you’re thinking about transitioning into。
A role in machine learning we also aim to teach you some things that might help you with ML engineering job interviews and then lastly in practice。
I think what we’ve found to be maybe the most powerful part of this is forming a community that you can use to learn from your peers about what works in the real world and what doesn’t we as instructors have solved many problems with M but there’s a very good chance that we haven’t solved one that’s like the one that you’re working on but in the broader full stackack deep learning community。
I would bet that there probably is someone who’s worked on something similar and so we hope that this can be a place where folks come together to learn from each other as well as just learning from us Now there are some things that we are explicitly not trying to do with this class we’re not trying to teach you machine learning or software engineering from scratch if you are coming at this class and you have an academic background in M but you’ve never written production code before or you’re a software engineer but have’ve never taken an M class before you can follow along with this class but I would highly recommend taking these prerequisites before you dive into the material here because I think get a lot more out of the class once you’ve learned the fundamentals。
Of each of these fields。 We’re also not aiming to cover the full breadth of deep learning techniques or machine learning techniques more broadly。
We’ll talk about a lot of the techniques that are used in practice。
but the chances are that we won’t talk about the specific model that you use for your use case It’s not the goal here。
We’re also not trying to make you an expert in any single aspect of machine learning We have projects and a set of labs that are associated with this course that will allow you to spend some time working on a particular application of machine learning。
but there isn’t a focus on becoming an expert in computer vision or NLP or any other single branch of machine learning and we’re also not aiming to help you do research and deep learning or any other M field and similarly Mops is this broad topic that involves everything from infrastructure and tooling to organizational practices and we’re not aiming to be comprehensive here The goal of this class is to show you end to end what it takes to build an ML powered product and give you pointers to the different pieces of the field that you’ll potentially need to go deeper on to solve the particular problem that you’re working。
So if you are feeling rusty on your prerequisites， but want to get started with the class anyway。
here are some recommendations for classes on ML and software engineering that I’d recommend checking out if you want to remind yourself of some of the fundamentals I mentioned this distinction between ML power products and MLOs a little bit and so I want to dive into that a little bit more MLOs is this discipline that’s emerged in the last couple of years really that is about practices for deploying and maintaining and operating machine learning models and the systems that generate these machine learning models in production and so a lot of MLOs is about how do we put together the infrastructure that will allow us to build models in a repeatable and governorable way how we’re able to do this at scale。
how we’re able to collaborate on these systems as a team and how we’re able to really run these machine learning systems in a potentially highscale production setting super important topic if your goal is to make ML work in the real world and there’s a lot of overlap with what we’re covering in this class but we see ML。
productss is kind of a distinct but overlapping discipline because a lot of what it takes to build a great MLP product goes beyond the infrastructure side and the sort of repeatability and automation side of machine learning systems and it also focuses on how to fit machine learning into the context of the product or the application that you’re building So other topics that are in scope of this M product discipline are things like how do you understand how your users are interacting with your model and what type of model they need how do you build a team or an organization that can work together effectively on machine learning systems How do you do product management in the context of M What are some of the best practices for designing products that use Ml as part of them things like data labeling capturing feedback from users etc and so this class is really focused on teaching you end to end what it takes to get a product out in the world that uses M and will cover the aspects of MLOs that are most critical to understand in order to do that a little bit about us as instructors。
I’m Josh Tobin on cofounder and CEO。machineine learning infrastructure startup called Gantry previously I was a research scientist openaiI and did my machine learning PhD at Berkeley and Charles and Sergey are my wonderful co-instructors who you’ll be hearing from in the coming weeks on the history of full stacky learning so we started out as a boot camp in 2018 Sergey and I as well as my grad school advisor and our closecollaborator Peter Ael had this collective realization that a lot of what we had been discovering about making M work in the real world wasn’t really well covered in other courses and we didn’t really know if other people would be interested in this topic so we put it together as a onetime weekend long boot camp we started to get good feedback on that and so it grew from there and we put the class online for the first time in 2021 and here we are So the way that this class developed was a lot of this is from our personal experience our study and reading of the materials in the field we also did a bunch of interviews with practitioners from this list of companies and at this point like a much longer list as well So we’re constantly out there talking to folks who are。
Doing this we’re building MLP products and trying to fold their perspectives into what we teach in this class。
Some logistics before we dive into the rest of the material for today。
first is if you’re part of the synchronous cohort。
all of the communication for that cohort is going to happen on discord So if you’re not on discord already then please reach out to us instructors and we’ll make sure to get you on if you’re not on Discord if you’re not checking it regularly there’s a high likelihood that you’re gonna miss some of the value of the synchronous course we will have a course project again for folks who are participating in the synchronous option which we’ll share more details about on Discord in the coming weeks and there’s also I think one of the most valuable parts of this class is the labs。
which have undergone like a big revamp this time around I want to talk a little bit more about what we’re covering there So the problem that we’re gonna be working on the labs is creating an application that allows you to take a picture of a handwritten page of text and then transcribe that into some actual text and so imagine that you have this web application where you can take a picture of your handwriting and then at the end you get the text。
Out of it。 And so the way that this is going to work is we’re going to build a web backend that allows you to send web requests。
decodedes those images and sends them to a prediction model an OCR model that will develop that will transcribe those into the text itself。
and those models are going be generated by a model training system that will also show you how to build in the class。
and the architecture that will use will look something like this will use state of the art tools that we think balance being able to really build a system like this in a principled way without adding too much complexity to what you’re doing All right so just to summarize this section。
machine learning power products are going mainstream and in large part this is due to the fact that it’s just much much easier to build machine learning models today than it was even four or five years ago。
And so I think the challenge ahead is given that we’re able to create these models pretty easily how do we actually use the models to build great products and that’s a lot of what we’ll talk about in this class And I think the sort of fundamental challenges is that there’s not only different tools that you need in order to build great。
Products but also different processes and mindsets as well。
And that’s what we’re really aiming to do here in FSDL。
So looking forward to covering some of this material and hopefully helping create the next generation of ML power products。
The next topic I want to dive into is when to use machine learning at all。
what problems is this technology useful for solving。
And so the key points that we’re gonna cover here are the first is that machine learning introduces a lot of complexity。
And so you really shouldn’t do it before you’re ready to do it。
and you should think about exhausting your other options before you introduce this to your stack on the flip side that doesn’t mean that you need to a perfect infrastructure to get started。
and then we’ll talk a little bit about what types of projects tend to lend themselves to being good applications of machine learning and we’ll talk about how to know whether projects are feasible and whether they’ll have an impact on your organization。
But to start out with when should you use machine learning at all。
So I think the first thing that’s really critical to know here is that machine learning projects have a higher failure rate than software projects in general the statistic that。
YouMost often floated around in blog posts or vendor pitches is that 87% this very precise number of machine learning projects fail。
I think it’s also worth noting that 73% of all statistics are made up on the spot So this one in particular I think is a little bit questionable whether this is actually a valid statistic or not anecdotally I would say that from what I’ve seen it’s probably more like 25% It still a very high number still very high failure rate。
but maybe not the 90 is% that people are quoting。The question you might ask is why is that the case right。
why is there such a high failure rate for machine learning projects？You know。
one reason that’s worth acknowledging is that for a lot of applications。
machine learning is fundamentally still research， So 100% success rate probably shouldn’t be the target that we’re aiming for。
but I do think that many machine learning projects are doomed to fail maybe even before they are undertaken。
And I think there’s a few reasons that this can happen。
So oftentimes machine learning projects are technically infeasible or they’re just scope poorly and there’s just too much of a lift to even get the first version of the model developed and that leads to projects failing because they just take too long to see any value。
Another common failure mode that’s becoming less and less common is that a team that’s really effective at developing a model may not be the right team to deploy that model into production。
And so there’s this friction after the model is developed where the model maybe looks promising in a Jupyter notebook。
but it never makes the leap to prod。 And so hopefully you’ll take things away from this class that will help you avoid being in this category。
Another really common issue that I’ve seen is when you as a broader organization are not all on the same page about what we would consider to be successful here。
And so I’ve seen a lot of machine learning projects fail because you have a model that you think works pretty well and you actually know how to deploy into production but the rest of the organization。
can’t get comfortable with the fact that this is actually going be running and serving predictions to users So how do we know when we’re ready to deploy And then maybe the most frustrating of all these failure modes is when you actually have your model work well and solves the problem that you set out to solve but it doesn’t solve a big enough problem and so the organization decides。
hey this isn’t worth the additional complexity that it’s going to take to make this part of our stack I think this is a point I want to double click on which is that really I think the bar for your machine learning project should be that the value of the project must outweigh not just the cost of developing it。
but the additional complexity that machine learning systems introduce into your software and machine learning introduces a lot of complexity to your software So this is kind of a quick summary of a classic paper that I would recommend reading。
which is the high interest credit card of technical debt paper the thesis of this paper is that machine learning as a technology tends to introduce technical debt at a much higher rate than most other software and the reasons。
The authors point to are one an erosion of boundary between systems。
so machine learning systems often have the property， for example。
that the predictions that they make influence other systems that they interact with if you recommend a user a particular type of content that changes their behavior and so that makes it hard to isolate machine learning as a component in your system。
It also relies on expensive data dependencies。 So if your machine learning system relies on a feature that’s generated by another part of your system。
then those types of dependencies the authors found can be very expensive to maintain it’s also very common for machine learning systems to be developed with design antipatterns somewhat avoidable but in practice very common and the systems are subject to the instability of the external world if your user’s behavior changes that can dramatically affect the performance of your machine learning models in a way that doesn’t typically happen with traditional software So the upshot is before you start a new M project。
you should ask yourself are we ready to use ML at all。
do we really need this technology to solve this problem and is it actually ethical to use M to solve this problem to know if you’re ready to use M some of the questions you might ask do we have a product at all。
do we have something that we can use to collect the data to know whether this is actually working are we already collecting that data and storing it in the same way if you’re not currently doing data collection then it’s gonna be difficult。
Build your first M system And do we have the team that will allow us to do this knowing whether you need Ml to solve a problem？
I think the first question that you should ask yourself is， do we need to solve this problem at all。
or are we just inventing a reason to use M because we’re excited about the technology。
have we tried using rules or simple statistics to solve the problem with some exceptions。
I think usually the first version of a system that you deploy that will eventually use M should be a simple rulebased or statisticsbased system because a lot of times you can get 80% of the benefit of your complex M system with some simple rules Now。
there’s some exceptions to this if the system is an MlP system or a computer vision system where rules just typically don’t perform very well。
But as a general rule， I think if you haven’t at least thought about whether you can use a rulebased system to achieve the same outcome。
then maybe you’re not ready to use M yet。 And lastly， is it ethical。
I won’t dive into the details here because we’ll have a whole lecture on this later in the course。
Next thing I want to talk about is if we feel like we’re ready to use M in our organization。
We know if the problem that we’re working on is a good fit to solving it with machine learning。
The sort of TlDR here is you want to look for like any other project prioritization。
you want to look for use cases that have high impact and low cost。
and so we’ll talk about different heuristics that you can use to determine whether this application of machine learning is likely to be high impact and low cost and so we’ll talk about heuristics like friction in your products。
complex parts of your pipeline places where it’s valuable to reduce the cost of prediction and looking at what other people in your industry are doing。
which is a very underrated technique for picking problems to work on and then we’ll also talk about some heuristics for for assessing whether a machine learning project is going to be feasible from a cost perspective overall prioritization framework that we’re going to look at here is。
Projects that you want to select are ones that are feasible。
So they’re low cost and they’re high impact。 Let’s start with the high impact set of things。
So what are some mental models you can use to find high impact Ml projects。
And these are some of the ones that we’ll cover。 So starting with a book called the Econs of AI。
And so the question of this book asks is， what problems does machine learning make economically feasible to solve that were maybe not feasible to solve in the past。
And so the sort of core observation in this book is that really at a fundamental level。
what AI does is it reduces the cost of prediction before maybe you needed a person。
and that person would take five minutes to create a prediction。 It’s very expensive。
It’s very operationally complex。 AI can do that in a fraction of a second for the cost of essentially running your machine or running your GPU Che prediction means that there’s going to be predictions that are happening in more places。
even in problems whereas too expensive to do before。
And so the upshot of this mental model for project selection is。
Think about projects where cheap prediction will have a huge business impact。
like where would you hire a bunch of people to make predictions that it isn’t feasible to do now。
the next mental model I want to talk about for selecting highimpact projects is just thinking about what is your products need And so I really like this article called three principles for designing ML power products from Spotify and in this article they talked about the principles that they use to develop the Discover weekly feature。
which is I think is like one of the most powerful features of Spotify and really the way that they thought about it is this reduces frid for our users reduces the friction of chasing everything down yourself and just brings you everything in a neat little package And so this is something that really makes their product a lot better。
And so that’s another kind of easy way to come up with ideas for machine learning projects Another angle to think about is what our types of problems that machine learning is particularly good at。
And one exploration of this mental model is an article called software 2。0 from Andre Carpathy。
which is also definitely worth a read。The kind of main thesis of this article is that machine learning is really useful when you can take a really complex part of your existing software system。
so a really messy stack of handwritten rules and replace that with machine learning。
replace that with gradient descent。And so if you have a part of your system that is complex。
manually defined rules， then that’s potentially really good candidate for automating with M。
And then lastly， I think it’s worth just looking at what other people in your industry are doing with M。
and there’s a bunch of different resources that you can look at to try to figure out what other success stories with MR。
I really like this article covering the spectrum of use cases of M and Netflix。
there are various industry reports。 This is a summary of one from algorithmthmia。
which kind of covers the spectrum of what people are using M to do。 and more generally。
I think looking at papers from the biggest technology companies tends to be a good source of what those companies are trying to build with M and how they’re doing it as well as earlier stage tech companies that are still pretty M forward and those companies I think are more likely to write these insights in blog posts than they are in papers And so here’s a list that I didn’t compile。
but I think is really valuable of case studies of using machine learning in the real world that are worth going。
If you’re looking for inspiration of what are types of problems you can solve and how might you solve them Okay so coming back to a prioritization framework。
we talked about some mental models for what M projects might be high impact and the next thing that we’re going to talk about is how to assess the cost of a machine learning project that you’re considering So the way I like to think about the cost of machine learning projects is there’s three main drivers for how much a project is going to cost The first and most important is data availability So how easy is it for you to get the data that you’re going to need to solve this problem the second most important is the accuracy requirement that you have the problem that you’re solving and then also important is the sort of intrinsic difficulty of the problem that you’re trying to solve So let’s start by talking about data availability the kind of key questions that you might ask here to assess whether data availability is going to be a bottleneck for your project is do we have this data already and if not how hard is and how expensive is it going to be to acquire how expensive is it not just to acquire but also to label if your labelers are really expensive。
Then getting enough data to solve the problem really well might be difficult。
How much data will we need in total， this can be difficult to assess a priori。
but if you have some way of guessing whether it’s 5000 or 10，000 or 100000 data points。
this is an important input and then how stable is the data So if you’re working on a problem where you don’t really expect the underlying data to change that much over time then the project is gonna be a lot more feasible than if the data that you need changes on a dayto-day basis So data availability Vidi is probably the most important cost driverri for a lot of machine learning powered projects。
because data just tends to be expensive。And this is slightly less true outside of the deep learning realm。
It’s particularly true in deep learning where you often require manual labeling。
but it also is true in a lot of other ML applications where data collection is expensive。
And lastly， on data availability is what data security requirements do you have if you’re able to collect data from your users and use that to retrain your model than that bodes well for the overall cost of the project if on the other hand。
you’re not even able to look at the data that your users are generating。
then that’s just gonna to make the project more expensive because it’s gonna to be harder to debug and harder to build a data flywheelMoving on to the accuracy requirement。
the kinds of questions you might ask here are。How expensive is it when you make a wrong prediction on one extreme you might have something like a self-driving car where a wrong prediction is extremely expensive because the prospect of that is really terrible。
On the other extreme is something like let’s say potentially a recommendationer system where if a user sees a bad recommendation once it’s probably not really going to be that bad。
maybe it affects their user experience over time and maybe and causes them to churn。
but certainly not as bad as a wrong prediction in a self-driving car。
you also need to ask yourself how frequently does the system actually need to be right to be useful。
I like to think of systems like Dolly2， which is an image generation system as like a positive example of this where you can if you’re just using Dolly2 as a creative supplement。
you can generate thousands and thousands of images and select the one that you like best for your use case。
So the system doesn’t need to be right more than like once every end times in order to actually get value from it as a user On the other hand。
if the system needs to be 10% reliable， like never ever make a wrong prediction。
In order for it to be useful then it’s just gonna to be more expensive to build these systems and then what are the ethical implications of your model making wrong predictions is like an important question to consider as well and then lastly on the problem difficulty questions to ask yourself is this problem well defineded enough to solve with M are other people working on similar things doesn’t necessarily need to be the exact same problem but if it’s a sort of a brand new problem that no one’s ever solved with M before that’s going to introduce a lot of technical risk Another thing that’s worth looking at if you’re looking at other work on similar problems is how much compute did it actually take them to solve this problem and it’s worth looking at that both on training side as well as on the inference side because if it’s feasible to train your model but it takes five seconds to make a prediction and for some applications that will be good enough and for some it won’t and then I think like maybe the weakest heuristic here but still potentially useful one is can a human do this problem at all a human can solve the problem then that’s a decent indication that a machine learning system might be able to solve it？
Well， but not a perfect indication， as we’ll come back to。
So I want to double click on this accuracy requirement。
Why is this such an important driver of the cost of machine learning projects？
The fundamental reason is that in my observation， the project cost tends to scale like super linearly in your accuracy requirement。
So as a very rough rule of thumb every time that you add an additional9 to your required accuracy。
So moving from 99。9 to 99。99% accuracy might lead to something like a 10x increase in your project costs because you might expect to need at least 10 times as much data if not more in order to actually solve the problem to that degree of accuracy required。
but also you might need a bunch of additional infrastructure monitoring support in order to ensure that the model is actually performing that accurately。
Next thing I want to double click on is the problem difficulty。
So how do we know which problems are difficult for machine learning systems to solve the first point I want to make here is this is like I think like a classically hard problem。
To really answer confidently。 And so I really like this comic for two reasons。
The first is because it gets at this core property of machine learning systems。
which is that it’s not always intuitive which problems will be easy for a computer to solve and which ones will be hard for a computer to solve in 2010 doing Gs lookup was super easy and detecting whether a photo was a bird was like research team in five years level of difficulty。
So not super intuitive as someone maybe be outside of the field。
the second reason I like this comic is because it also points to the sort of second challenge in assessing feasibility in M。
which is that this field just moves so fast that if you’re not keeping up with what’s going on in the state of the art。
then your understanding of what’s feasible will be stale very quickly。
building an application to detect whether a photo is of a bird is no longer a research team in five years problem。
It’s like a API call in 15 minutes tag problem。 So take everything I say here with a grain of salt because the。
asibility of ML projects is notoriously difficult to predict Another example here is in the late 90s the New York Times when they were talking about sort of AI systems beating humans at chess predicted that it might be 100 years before a computer beats human echo go or even longer and less than 20 years later。
machine learning systems from Deep mindd beat the best humans in the world at go。
these predictions are notoriously difficult to make。 but that being said。
I think it’s still worth talking about。 And so one heuristic that you’ll hear for what’s feasible to do with machine learning is this heuristic from Andrew。
which is that anything that a normal person can do in less than one second we can automate with AI。
I think this is actually not a great heuristic for what’s feasible to do with AI。
but you’ll hear it a lot。 So I wanted to talk about it anyway。
there’s some examples of where this is true So recognizing the content of images understanding speech。
potentially translating speech， maybe grasping objects with a robot and things like that are things that you could point to as evidence for Andrew’s statement being correct。
but I think there’s some really obvious counter。Examples as well。
Maine learning systems are still no good at things that a lot of people are really good at。
like understanding human humor or sarcasm， complex inhand manipulation of objects。
generalizing to brand new scenarios that they’ve never seen before。
this is a heuristic that you’ll see it’s not one that would recommend using seriously to assess whether your project is feasible or not There’s a few things that we can say are definitely still hard in machine learning。
I kept a couple of things in these slides that we talked about being really difficult in machine learning when we started teaching the class in 2018 that I think I would no longer consider to be super difficult anymore unsupervised learning being one of them。
but reinforcement learning problems still tend to be not very feasible to solve for real world use cases。
although there are some use cases where with tons of data and compute reinforcement learning can be used to solve real world problems within the context of supervised learning there are also still problems that are hard。
So things like question answering a lot of progress over the last few years still these systems aren’t perfect text summarization video prediction building 3D models。
Another example of one that I think I would use to say is really difficult。
but with Nerf and all the sort of derivatives of that I think is more feasible than ever。
real world speech recognition so outside of the context of a clean data set in a noisy room can we recognize what people are saying resisting adversarial examples doing math。
although there’s been a lot of progress on this problem as well over the last few months。
Solving world word problems or boguard problems。 This is an example， by the way。
of a boguard problem。 It a visual analogy type problem。
So this is kind of a laundry list of some things that are still difficult。
even in supervised learning。 And so can we reason about this。
what types of problems are still difficult to do。 So I think one type is where not the input to the model itself。
but the prediction that the model is making in the output of the model or that is like a complex or high dimensional structure or where it’s ambiguous So。
for example，3D reconstruction， The 3D model that you’re outputting is very high dimensional。
And so that makes it difficult to do for Ml video prediction。
not only high dimensional but also ambiguous just because。
you know what happened in the video for the last five seconds。
there’s still maybe infinite possibilities for what the video might look like going forward。
So it’s ambiguous and it’s high dimensional， which makes it very difficult to do with M dialogue systems。
again， very ambiguous， very openended， very difficult to do with M and openended recommender systems。
So a second category of problems that are still difficult to do with M are problems where you really need the system to be reliable。
machine learning systems tend to fail in all kinds of unexpected and hard to reason about ways。
So anywhere where you need really high precision or robustness is going to be more difficult to solve using machine learning So failing safely out of distribution。
for example， is still a difficult problem in M。 robustness to adversarial attacks is still a difficult problem in M and even things that are easier to do with low precision。
like estimating the position and rotation of an object in 3D space can be very difficult to do if you have a high precision requirement。
The last category of problems I’ll point to here is problems where you need the system to be able to generalize well to data that’s never seen before。
This can be data that’s out of distribution it can be where your system needs to do。
Something that looks like reasoning or planning or understanding of causality These problems tend to be more in the research domain today。
I would say one example is in the self-driving car world dealing with edge cases。
very difficult challenge in that field but also control problems in self-driving cars you know those stacks are incorporating more and more M into them whereas the computer vision and perception part of self-driving cars adopted machine learning pretty early the control piece was using more traditional methods for much longer and then places where you have a small amount of data again。
like if you’re considering machine learning broadly。
small data is often possible but especially in the context of deep learning。
small data still presents a lot of challenges。Summing this up like how should you try to assess whether your machine learning project is feasible or not。
First question you should ask is do we really need to solve this problem with M at all。
I would recommend putting in the work upfront to define what is the success criteria that we need and doing this with everyone that needs to sign up on the project in the end。
not just the M team。 let’s avoid being an M team that works on problems in isolation and then has those projects killed because no one actually really needed to solve this problem or because the value of the solution is not worth the complexity that it adds to your product then you should consider the ethics of using M to solve this problem and we’ll talk more about this towards the end of the course in the ethics lecture that it’s worth doing literature review to make sure that there are examples of people working on similar problems trying to rapidly build a benchmark data that’s labeled so you can start to get some sense of whether your models performing well or not and only then building a minimum viable model So this is potentially even just manual rules or simple linear regression deploying this into production if it’s feasible to do so or at least running this on your existing problem So。
Baseline and then lastly， it’s worth just restating making sure that you once you’ve built this minimum viable model that may not even use M just really asking yourself the question of whether this is good enough for now or whether it’s worth putting in the additional effort to turn this into a complex M system The next point I want to make here is that not all M projects really have the same characteristics and so should be and so you shouldn’t think about planning all M projects in the same way I want to talk about some archetypes of different types of M projects and the implications that they have for the feasibility of the projects and how you might run the projects effectively。
And so the three archetypes I want to talk to are defined by how they interact with real worldor users and so the first archetype is software 2。
0 use cases and so I would define this as。Taking something that software does today。
So an existing part of your product that you have， let’s say。
and doing it better more accurately or more efficiently with M。
It’s taking a part of your product that’s already automated or already partially automated and adding more automation or more efficient automation using machine learning then the next archetype is human in the loop systems。
so this is where you take something that is not currently automated in your system。
but it’s something that humans are doing or humans could be doing and helping them do that job better more efficiently or more accurate accurately by supplementing their judgment with Mbased tools。
preventing them from needing to do the job on every single data point by giving them suggestions of what they can do so they can shortcut their process in a lot of places。
Human the loop systems are about making the humans that are ultimately making the decisions more efficient or more effective。
And then lastly， autonomous systems。 And so these are systems where you take something that humans do today or maybe is just not being done at all today and fully automated with。
To the point where you actually don’t need humans to do the judgment piece of it at all。
And so some examples of software 2。0 are if you have an ID that has code completion。
can we do better code completion by using M Can we take a recommendation system that is initially using some simple rules and make it more customized can we take our video game AI that’s using this rulebased system and make it much better by using machine learning Some examples of human and loop systems would be building a product to turn hand drawnwn sketches into slides you still have human on the other end that’s evaluating the quality of those sketches before they go in front of a customer or a stakeholder So it’s a human in the loop system。
but it’s potentially saving a lot of time for that human email auto completion。
So if you use Gmail you’ve seen these email suggestions where itll suggest sort of short responses to the email that you got I get to decide whether that email actually goes out to the world So it’s not an automation system it’s a human in the loop system or helping a radiologist to their job master。
Of autonomous systems are things like full self-driv Maybe there’s not even a steering wheel in the car。
I can’t interrupt the autonomous system and take over control of the car even if I wanted to。
or maybe it’s not designed for me to do that very often fully automated customer support。
So if I go on a company’s website and I interact with their customer support without even having the option of talking to an agent or with them making it very difficult to talk to an agent。
that’s an autonomous system or for example， like fully automating website design。
so that to the point where people who are not design experts can just click a button and get a website design for them。
And so I think some of the key questions that you need to ask before embarking on these projects are a little bit different depending on which archetype your project falls into。
So if you’re working on a software 2。0 project then I think some of the questions you should be concerned about are how do you know that your models are actually performing improving performance over the baseline that you already have how confident are you that the type of performance improvement that you might be able to get froml is actually going to generate value for your business if it’s just one。
Pcent better I that really worth the cost Then do these performance improvements lead to what’s called a data flywheel。
which I’ll talk a little bit more about with human in the loop systems。
you might ask a different set of questions before you embark on the project。
Like how good does the system actually need to be useful if the system is able to automate 10% of the work of the human that is ultimately making the decisions or producing the end product。
Is that useful to them or does that just slow it slow them down。
How can you collect enough data to make it that good。
I it possible to actually build a data that is able to get you to that useful threshold for your system and for autonomous systems。
the types of questions you might ask are what is an acceptable failure rate for the system。
How many nines in your performance threshold do you need in order for this sort of not to cause harm in the world and how can you guarantee like how can you be really confident that it won’t exceed that failure rate。
And so this is something that in autonomous vehicles， for example。
teams put a ton of effort into building the simulation and testing systems that they need to be confident that they won’t exceed the failure rate that’s except the very very low failure rate that’s acceptable for those systems。
I want to double click on this data flylywheel concepts for software 2。
0 we talked about can we build a data flylywheel that lead to better and better performance of the system And the way to think about data flylywheel is it’s this virtuous cycle where as your model gets better。
you are able to use better that better model to make a better product which allows you to acquire more users and as you have more users。
there’s users generate more data which you can use to build a better model and this creates this virtuous cycle。
And so the connections between each of these steps are also important in order for more users to allow you to collect more data。
You need to have a data loop you need to have a way of automatically collecting data and deciding what data points to label from your users or at least processes for doing these in order for more data to lead to a better model that’s kind of on you as an MLpractition you need to be able to translate more data more granular data more labels into a model that performs better for your users and then in order for the better model to lead to better users。
you need to be sure that better predictions are actually making your product better。
Another point that I want to make on these project archetypes is I would sort of characterize them as having different tradeoffs on this feasibility versus impact two by two that we talked about earlier。
So 2。0 projects since they’re just taking something that you already know you can automate and automating it better。
tend to be more feasible but since you already have an answer to the question that they’re also answering。
they also tend to be lower impact on the other extreme autonomous systems tend to be very difficult to build because the accuracy requirements in general are quite high。
but the impact can be quite high as well。Because you’re replacing something that literally doesn’t exist and human in the loop systems tend to be somewhere in between where you can really like you can use this paradigm of machine learning products to build things that couldn’t exist before but the impact is not quite as high because you still need people in the loop that are helping use their judgment to complement the machine learning model There’s ways that you can move these types of projects on the feasibility impact matrix to make them more likely to succeed。
So if you’re working on the software 2。0 projects you can make these projects have potentially higher impacts by implementing a data loop that allows you to build continual improvement data flywheel that we talked about before and potentially allows you to use the data that you’re collecting from users interacting with this system to automate more tasks in the future So for example。
in the code completion IDde example that we gave before you can if you’re building something like Githubcopilot then think about all the things that the data that you’re collecting from that。
Be useful for building in the future。 you can make human in the loop systems more feasible through good product design and we’ll talk a little bit more about this in a future lecture。
but there’s design paradigms in the product itself that can reduce the accuracy requirement for these types of systems and another way to make these projects more feasible is by adopting sort of a different mindset。
which is let’s just make the system good enough and ship it into the real world so we can start the process of seeing how how real users interact with it and using the feedback that we get from our humans in the loop to make the model better。
And then lastly， autonomous systems can be made more feasible by adding guardrails or in some cases。
adding humans in the loop。 and so this is you can think of this as the approach to autonomous vehicles where have safety drivers in the loop early on in the projects or where you introduce teleoperation so that a human can take control of the system if it looks like something is going wrong。
I think another point that is really important here is despite all of this talk about。
What’s feasible to do with ML， the complexity that ML introduces in your system。
I don’t mean by any of this to say that you should do necessarily a huge amount of planning before you dive into using ML at all。
just make sure that the project that you’re working on is the right project and then just dive in and get started。
and in particular， I think a failure mode that I’m seeing crop up more and more over the past couple of years that you should avoid is falling into the trap of tool fetishization。
So one of the great things that’s happened in ML over the past couple of years is the rise of this MLOs discipline。
And alongside of that has been proliferation of different tools that are available on the market to help with different parts of the ML process。
And one thing that I’ve noticed that this is caused for a lot of folks is this sort of general feeling that you really need to have perfect tools before you get started You don’t need perfect tools to get started and you also don’t need a perfect model And in particular。
just because Google or Uber is doing something like just because they have a feature store as part of their stack or they serve models in the particular way doesn’t mean that you need to have that as well。
And so a lot of what we’ll try to do in this class is talk about what’s the middle ground between doing things in the right way from a production perspective。
but not introducing too much complexity early on into your project。
So that’s one of the reasons why FStL is a class about building ML powered products in a practical way and not an ML opps class that’s focused on what is the state of the art in the best possible infrastructure that you can use and a talk and blog posts and associated set of things on this concept that I really like is this。
Mlops at reasonable scale push by some of the folks from Covio and the sort of central thesis of MLop reasonable scale is you’re not Google。
you probably have a finite compute budget， not entire cloudud you probably have a limited number of folks on your team you probably have not an infinite budget to spend on this and you probably have a limited amount of data as well。
And so those differences between what you have and what Uber has or what Google has have implications for what the right stack is for the problems that you’re solving。
And so it’s worth thinking about these cases separately。
And so if you’re interested in what one company did and recommends for an M stack that isn’t designed to scale to becoming Uber scale that Id recommend checking out this talk。
to summarize what we’ve covered so far， machine learning is an incredibly powerful technology but it does add a lot of complexity And so before you embark on a machine learning project you should make sure that you’re thinking carefully about whether you。
Really need Ml to solve the problem that you’re solving and whether the problem is actually worth solving at all。
given the complexity that this adds。 And so let’s avoid being M teams that have their projects get killed because we’re working on things that don’t really matter to the business that we’re a part of All and the last topic I want to cover today is once you’ve sort made this decision to embark on an M project。
What are the different steps that you’re gonna to go through in order to actually execute on that project and this will also give you an outline for some of the other things you can expect from the class。
So the running case study that we’ll use here is a modified version of a problem that I worked on when I was at openaiI。
which is pose estimation。 Our goal is to build a system that runs on a robot that takes the camera feed from that robot and uses it to estimate the position in 3D space and the orientation。
the rotation of each of the objects in the scene so that we can use those for downstream tasks and in particular so we can use them to feed into a separate model which will be used to tell the robot how it actually can grasp the different objects in the scene machinech learning projects start like any other project。
A planningning and project setup phase。 And so what the types of activities we’d be doing in this phase。
When we’re working on this pose estimation project are things like deciding to work on pose estimation at all。
determining whether how much this is going to cost， what resources we need to allocate to it。
considering the ethical implications and things like this a lot of what we’ve been talking about so far in this lecture。
once we plan the project then we’ll move into a data collection and labeling phase。
And so for pose estimation， what this might look like is collecting the corpus of objects that we’re gonna train our model on。
setting up our sensors like our cameras to capture our information about those objects。
actually capturing those objects and somehow figuring out how to annotate these images that we’re capturing with ground truth。
like the pose of of the objects in those images。 One point I want to make about the lifecycle of MLl projects is that this is not like a straightforward path。
machine learning projects tend to be very iterative。
and each of these phases can feed into any of the phases before。
as you learn more about the problem that you’re working on So for example。
you might realize that actually it’s way too hard for us to get data in order to solve this problem or it’s really difficult for us to label。
Poose of these objects in 3D space。 But what we can do is it’s actually much cheaper for us to annotate like per pixel segmentation。
So can we reformulate the problem in a way that allows us to。
to use what we’ve learned about data collection and labeling to plan a better project。
Once you have some data to work on， then you enter the sort of training and debugging phase。
And so what we might do here is we might implement a baseline for our model。
not using like a complex neural network， but just using some openC functions。
And then once we have that working， we might find a state of the art model and reproduce it。
debuggger implementation and iterate on our model。
runs some hyperparameter sweeps until it performs well on our task。
This can feed back into the data collection and labeling phase because we might realize that we actually need more data in order to solve this problem or we might also realize that there’s something flawed in the process that we’ve been using to label the data that we’re using data labeling process might need to be revisited but we can also loop all the way back to the project planning phase because we might realize that actually this task is a lot harder than we thought or the requirements that we specified at the planning phase trade off with each other so we need to revisit which are most important So for example maybe we thought that we had an accuracy requirement of estimating the pose of these objects to one1th of one centimeter and we also had a latency requirement for inference on our models of 1100th of a second to run on robotic hardware and we might realize that hey we can get this really really tight accuracy requirement or we can have really fast inference。
's very difficult to do both。 So is it possible to relax one of those assumptions Once you’ve trained a model that works pretty well offline for your task。
then your goal is going to be to deploy that model。
test it in the real world and then use that information to figure out where to go next for the purpose of this project that might look like piloting the grasping system in the lab so before we roll it out to actual users can we test it in a realistic scenario and we might also do things like writing tests to prevent regressions and evaluate for bias in the model and then eventually rolling this out into production and monitoring it and continually improving it from there and so we can feedback here into the training and debugging stage because oftentimes what we’ll find is that the model that works really well for our offline data set once it gets into the real world。
it doesn’t actually work as well as we thought whether that’s because the accuracy requirement that we had for the model was wrong。
like we actually needed it to be more accurate than we thought or maybe the metric that we’re looking at the accuracy is not actually the metric that really matters for success at the downstream task that。
tryrying to solve that could cause us to revisit the training phase。
We also could loop back to the data collection and labeling phase because common problem that we might find in the real world is that。
There’s some mismatch between the training data that we collected and the data that we actually saw when we went out and tested this。
We could use what we learned from that to go collect more data or mind for hard cases。
like mind for the failure cases that we found in production。 And then finally。
as I alluded to before we could all the way back to the project planning phase because we realize that the metric that we picked doesn’t really drive the downstream behavior that we desired。
just because the grasp model is accurate doesn’t mean that the robot will actually be able to successfully grasp the object So we might need to use a different metric to really solve this task。
or we might realize that the performance in the real world isn’t that great。
And so we maybe need to add additional requirements to our model as well。
maybe it just needs to be faster in order to run on a real robot。
So these are kind of like what I would think of as the activities that you do in any particular machine learning project that you undertake。
but there’s also some sort of crossproject things that you need in order to be successful。
which we’ll talk about in the class as well。 you need to be able to work on these problems together as a team。
and you need to have the right infrastructure。oling to make these processes more repeatable。
And these are topics that we’ll cover as well。 So this is like a broad conceptual outline of the different topics that we’ll talk about in this class。
And so to wrap up for today， what we covered is machine learning is a complex technology。
And so you should use it because you need it or because you think it’ll generate a lot of value。
but it’s not a cure all。 It doesn’t solve every problem。
It won’t automate every single thing that you wanted to automate。
So let’s pick projects that are gonna be valuable。 But in spite of this。
you don’t need a perfect setup to get started。 And let’s spend the rest of this course walking through the project lifecycle and learning about each of these stages and how we can how we can use them to build great ML powered products。
全栈深度学习：1：课程介绍与何时使用机器学习
在本节课中，我们将学习这门课程的目标与愿景，并探讨一个核心问题：何时应该使用机器学习技术。我们还将概述一个典型机器学习项目的生命周期，这构成了本课程后续内容的概念框架。
🎯 课程愿景：构建由机器学习驱动的产品
全栈深度学习课程旨在成为为构建机器学习驱动产品的人们提供的课程和社区。现在是一个讨论机器学习驱动产品的激动人心的时刻，因为机器学习正迅速成为主流技术。这一点可以从初创公司的融资、招聘信息以及大公司对该技术的持续投入中看出。
与2018年我们刚开始授课时相比，情况发生了巨大变化。当时，许多最令人兴奋的机器学习产品都是由最大的公司构建的。如今，由机器学习驱动的强大产品范围要广泛得多，不再仅限于大型公司。例如，像Descript这样的应用，以及初创公司构建的电子邮件生成工具。
这种快速变化的原因之一是模型训练正开始变得商品化。现在，使用像Hugging Face这样的工具，只需一两行代码就能部署最先进的NLP或计算机视觉模型。此外，许多框架（如Keras和PyTorch Lightning）正在围绕某些标准进行统一，减少了构建这些系统所需的“意大利面条式”代码。
展望未来，机器学习的历史特点是公众对该技术认知的起伏。为了避免重蹈“AI寒冬”的覆辙，我们不仅需要在研究上取得进展，还需要确保这些进展能转化为实际的产品。这就是我们避免重蹈覆辙的方式。
然而，构建机器学习驱动的产品，在许多方面需要与学术界开发模型的过程有根本性的不同。
🔄 从“平地球机器学习”到真实世界循环
在学术环境中开发模型的过程，可以被称为“平地球机器学习”。这个过程可能对许多人来说很熟悉：
-
选择一个问题。
-
收集一些数据。
-
清理和标注数据。
-
迭代开发模型，直到在收集的数据集上表现良好。
-
评估模型，如果指标良好，则撰写报告、论文或幻灯片，然后完成。
但在现实世界中，如果你将该模型部署到生产环境，它不一定能在真实世界中长期表现良好。因此，机器学习驱动的产品需要一个外部循环：
-
将模型部署到生产环境。
-
测量模型与真实用户交互时的表现。
-
使用真实世界的数据构建数据飞轮。
-
将此过程作为外部循环持续进行。
仅仅因为你没有在机器学习系统中看到这个外部循环，并不意味着它不存在。本课程的核心就是关于如何完成构建机器学习驱动产品的这个过程。
本课程的目标与范围
以下是本课程旨在达成的目标：
-
教授通用技能：教授可用于构建机器学习产品的通用技能集，并理解机器学习产品各个部分如何组合在一起。
-
介绍MLOps概念：教授足够的MLOps知识以完成任务，但不会涵盖MLOps主题的全部深度。
-
分享最佳实践：分享我们在现实世界中看到的有效最佳实践，并尝试解释其背后的动机。
-
助力求职：教授一些可能有助于机器学习工程职位面试的内容。
-
构建社区：形成一个社区，供学员向同行学习现实世界中有效和无效的方法。
本课程不打算涵盖以下内容：
-
从零开始教学：本课程不打算从零开始教授机器学习或软件工程。如果你有机器学习学术背景但从未写过生产代码，或者是软件工程师但从未上过机器学习课程，你可以跟上本课程，但我们强烈建议你先学习相关基础知识。
-
覆盖所有技术：我们不会涵盖深度学习或更广泛的机器学习技术的全部广度。我们会讨论许多实践中使用的技术，但很可能不会讨论你特定用例的特定模型。
-
培养领域专家：本课程的目标不是让你成为机器学习任何单一方面的专家。虽然有相关的项目和实验，但重点不在于成为计算机视觉、NLP或其他任何机器学习分支的专家。
-
专注于研究：本课程也不旨在帮助你进行深度学习或任何其他机器学习领域的研究。
-
全面的MLOps：MLOps是一个涉及从基础设施、工具到组织实践的广泛主题，本课程的目标并非在此全面深入。
本课程的目标是向你展示端到端地构建一个机器学习驱动产品需要什么，并为你指出可能需要深入学习的领域，以解决你正在处理的具体问题。
关于实验项目
本课程的实验部分将围绕一个问题展开：创建一个应用程序，允许你拍摄手写文本页面的照片，然后将其转录为实际文本。
你将构建一个网络后端，用于接收网络请求、解码图像并将其发送给一个预测模型（OCR模型），该模型会将图像转录为文本。这些模型将由一个模型训练系统生成。我们将使用最先进的工具来构建这个系统，这些工具能在不增加太多复杂性的前提下，以有原则的方式完成构建。
总结本节：机器学习驱动的产品正在成为主流，这在很大程度上是因为如今构建机器学习模型比四五年前要容易得多。因此，未来的挑战在于：既然我们能够相对容易地创建这些模型，我们如何实际使用这些模型来构建优秀的产品？这正是我们将在本课程中讨论的主要内容。根本的挑战在于，构建优秀产品不仅需要不同的工具，还需要不同的流程和思维方式。
上一节我们介绍了课程的总体愿景和结构。接下来，我们将深入探讨一个在启动任何新机器学习项目时都应该首先问自己的关键问题。
🤔 何时应该（以及不应该）使用机器学习
本节我们将探讨机器学习适用于解决哪些问题。关键要点包括：首先，机器学习会引入很多复杂性，因此在准备好之前不应轻易使用，应考虑在将其引入技术栈之前用尽其他选项。另一方面，这并不意味着你需要完美的基础设施才能开始。我们还将讨论哪些类型的项目通常适合应用机器学习，以及如何判断项目是否可行且会对组织产生影响。
机器学习项目的高失败率
首先，一个关键事实是：机器学习项目的失败率通常高于一般的软件项目。虽然常被引用的“87%的机器学习项目失败”这个精确数字可能值得怀疑，但根据经验观察，失败率可能仍在25%左右，这仍然是一个很高的数字。
为什么会这样？一个值得承认的原因是，对于许多应用来说，机器学习从根本上说仍然是研究。但我们也认为，许多机器学习项目甚至在开始之前就注定要失败。
以下是几种常见的失败模式：
-
技术不可行或范围不当：项目在技术上不可行，或者范围规划不当，导致开发第一个版本模型的工作量过大，项目因耗时过长看不到任何价值而失败。
-
模型开发与部署脱节：一个擅长开发模型的团队可能不是将其部署到生产环境的合适团队。这导致模型在Jupyter笔记本中看起来很有希望，但从未成功跃升到生产环境。
-
组织内部对成功标准未达成共识：组织内部对什么是成功没有达成共识。即使模型运行良好且知道如何部署，但组织的其他部分无法接受它实际运行并向用户提供预测。
-
解决了错误的问题（价值不足）：模型运行良好，解决了设定的问题，但解决的问题不够大，组织认为不值得为引入该技术而增加额外的复杂性。
最后一点值得深入探讨：机器学习项目的门槛应该是，项目的价值必须超过开发成本，以及机器学习系统给软件带来的额外复杂性。
机器学习作为一种技术，往往比其他软件以更高的速度引入技术债务。原因包括：
-
系统间边界模糊：机器学习系统的预测会影响与之交互的其他系统。
-
昂贵的数据依赖：如果机器学习系统依赖于由系统其他部分生成的特征，那么维护这些依赖关系可能非常昂贵。
-
常见的设计反模式：在开发机器学习系统时，设计反模式很常见。
-
受外部世界不稳定性影响：用户行为的变化会显著影响机器学习模型的性能。
因此，在开始一个新的机器学习项目之前，你应该问自己：
-
我们准备好使用机器学习了吗？
-
我们真的需要这项技术来解决这个问题吗？
-
使用机器学习来解决这个问题在伦理上是否合适？
要判断是否准备好使用机器学习，可以问以下问题：
-
我们是否有产品？（是否有东西可以用来收集数据以判断是否有效？）
-
我们是否已经在以相同的方式收集和存储数据？（如果目前没有进行数据收集，那么构建第一个机器学习系统将会很困难。）
-
我们是否有能够完成这项工作的团队？
要知道是否需要机器学习来解决问题，首先应该问：我们真的需要解决这个问题吗？ 还是我们只是出于对技术的兴奋而发明一个使用机器学习的理由？我们尝试过使用规则或简单统计来解决这个问题吗？ 通常，你最终要部署的、将使用机器学习系统的第一个版本，应该是一个简单的基于规则或统计的系统，因为很多时候你可以用一些简单的规则获得复杂机器学习系统80%的收益。当然也有例外，但对于一般规则，如果你没有至少考虑过是否可以使用基于规则的系统来实现相同的结果，那么可能还没有准备好使用机器学习。
如何选择高影响力、低成本的机器学习项目
如果我们觉得组织已经准备好使用机器学习，并且知道正在处理的问题适合用机器学习解决，那么总结来说，就像任何其他项目优先级排序一样，你需要寻找高影响力和低成本的用例。
我们将讨论一些启发式方法，用于判断机器学习应用是否可能具有高影响力和低成本。我们将讨论诸如产品中的摩擦点、流程中的复杂部分、降低预测成本有价值的地方，以及观察行业内其他人正在做什么等启发式方法。后者是选择要处理问题的非常被低估的技术。
我们还将讨论一些启发式方法，用于从成本角度评估机器学习项目是否可行。总体优先级框架是：选择那些可行（低成本）且高影响力的项目。
寻找高影响力项目的思维模型
以下是几种寻找高影响力机器学习项目的思维模型：
-
《AI经济学》：这本书的核心观点是，人工智能从根本上降低了预测的成本。便宜的预测意味着预测将在更多地方发生，甚至发生在过去因成本太高而无法解决的问题中。这个思维模型的项目选择启示是：思考那些廉价预测将产生巨大商业影响的项目，即那些你现在会因为成本高而不愿雇佣大量人员来做预测的地方。
-
思考产品需求：例如，Spotify的“每周发现”功能通过减少用户自己搜寻一切的摩擦，将一切打包呈现，从而极大地改善了产品。思考你的产品中哪些地方可以通过减少摩擦来增加价值。
-
机器学习擅长解决的问题：Andrej Karpathy的“软件2.0”文章提出，机器学习在你可以用机器学习（梯度下降）替换现有软件系统中非常复杂的部分（一堆手写规则）时特别有用。如果你系统中有一个部分是复杂的、手动定义的规则，那么这可能是用机器学习自动化的绝佳候选。
-
观察行业内的实践：查看其他人在你的行业中用机器学习做什么。可以阅读大公司的论文、早期技术公司的博客文章，以及行业报告和案例研究，以获取灵感和了解如何解决问题。
评估机器学习项目成本
我喜欢从三个主要驱动因素来思考机器学习项目的成本：
-
数据可用性（最重要）
-
准确率要求
-
问题本身的内在难度
关于数据可用性，需要问的关键问题包括：
-
我们是否已经拥有这些数据？如果没有，获取（和标注）的难度和成本有多大？
-
我们总共需要多少数据？（这很难预先评估，但如果有办法猜测是5000、10000还是100000个数据点，这是一个重要的输入。）
-
数据的稳定性如何？如果底层数据预计不会随时间发生太大变化，那么项目可行性就高得多。
-
有什么数据安全要求？如果你能从用户那里收集数据并用于重新训练模型，那对项目的总体成本是有利的。反之，如果你甚至无法查看用户生成的数据，那么项目成本会更高，因为调试和构建数据飞轮会更困难。
关于准确率要求，需要问的问题包括：
-
做出错误预测的成本有多高？（从自动驾驶汽车（成本极高）到推荐系统（单次错误影响较小））
-
系统需要多频繁地正确才能有用？（例如，DALL-E 2作为创意辅助工具，生成数千张图像并选择最喜欢的一张，不需要每次都正确。）
-
模型做出错误预测的伦理影响是什么？
关于问题难度，需要问的问题包括：
-
这个问题是否足够明确，可以用机器学习解决？
-
其他人是否在处理类似的事情？（不一定完全相同，但如果是一个从未用机器学习解决过的新问题，会引入很多技术风险。）
-
他们解决这个问题实际需要多少计算资源？（包括训练和推理）
-
人类能解决这个问题吗？如果能，这通常表明机器学习系统可能能够解决它，但并非完美指标。
准确率要求为何是重要成本驱动因素？ 根本原因是，根据观察，项目成本往往随着准确率要求的提高而超线性增长。作为一个非常粗略的经验法则，每次你在所需准确率上增加一个“9”（例如从99.9%到99.99%），可能会导致项目成本增加10倍，因为你可能需要至少10倍（甚至更多）的数据，并且可能需要大量额外的基础设施、监控和支持来确保模型实际达到该准确率。
关于问题难度，需要指出的是，预测机器学习项目的可行性是出了名的困难。这个领域发展如此之快，如果你不跟上最先进的技术，你对可行性的理解很快就会过时。
一个常听到但并非完美的启发式方法是“Andrew Ng启发式”：一个正常人在一秒钟内能做的任何事情，我们都可以用AI自动化。虽然有一些例子支持这个说法（如图像内容识别、语音理解），但也有明显的反例（如理解人类幽默或讽刺、复杂的手部物体操作、泛化到从未见过的新场景）。因此，这不是一个推荐用于严肃评估项目可行性的启发式方法。
目前，机器学习中仍然困难的领域包括：
-
强化学习问题（尽管在某些用例中，拥有大量数据和计算资源时，强化学习可以解决现实世界问题）。
-
监督学习中的特定问题，如问答、文本摘要、视频预测、3D建模、真实世界语音识别、抵抗对抗样本、做数学题、解决文字问题等。
我们可以尝试对仍然困难的问题类型进行推理：
-
输出复杂、高维或模糊的问题：例如3D重建（高维）、视频预测（高维且模糊）、对话系统（开放性强、模糊）、开放式推荐系统。
-
需要高可靠性/鲁棒性的问题：机器学习系统往往会以各种意想不到且难以推理的方式失败。任何需要高精度或鲁棒性的地方都会更困难，例如安全地处理分布外数据、抵抗对抗攻击、高精度要求下的3D物体位姿估计。
-
需要良好泛化到新数据的问题：这包括需要系统进行类似推理、规划或理解因果关系的问题。例如，自动驾驶汽车中的边缘情况处理、控制问题（尽管现在越来越多地融入机器学习），以及数据量小的场景。
如何评估项目可行性并开始行动
总结一下，你应该如何尝试评估机器学习项目是否可行？
-
首先问：我们真的需要用机器学习来解决这个问题吗？
-
预先投入工作，与所有最终需要签署项目的人一起定义成功标准，避免机器学习团队孤立工作，导致项目因无人真正需要解决该问题或其解决方案的价值不值得给产品增加的复杂性而被取消。
-
考虑使用机器学习解决此问题的伦理问题（本课程后期会有专门讲座讨论）。
-
进行文献综述，确保有类似问题的研究案例。
-
尝试快速构建一个带标签的基准数据集，以便开始了解模型表现如何。
-
仅在此之后，构建一个最小可行模型（可能只是手动规则或简单线性回归）。
-
如果可行，将其部署到生产环境，或者至少在你现有的问题上运行它，建立一个基线。
-
最后，重新审视：一旦构建了这个可能甚至不使用机器学习的最小可行模型，认真问自己，这对于现在来说是否足够好，或者是否值得投入额外的努力将其转变为复杂的机器学习系统。
机器学习项目的三种原型及其启示
并非所有机器学习项目都具有相同的特征，因此不应以相同的方式规划所有项目。我想讨论三种不同类型的机器学习项目原型，以及它们对项目可行性和有效运行项目的影响。这三种原型根据它们与真实世界用户的交互方式来定义：
-
软件2.0用例：定义为利用机器学习，将当前软件已经做的事情（产品中已有的部分）做得更好、更准确或更高效。它是将产品中已经自动化或部分自动化的部分，通过机器学习增加更多自动化或更高效的自动化。
-
人在回路系统：将当前系统中尚未自动化、但人类正在做或可以做的事情，通过基于机器学习的工具来帮助人类更高效、更准确地完成工作，或者通过提供建议来防止他们需要对每个数据点都进行处理，从而简化他们的流程。人在回路系统旨在让最终做出决策的人类更高效或更有效。
-
自主系统：将人类今天做的事情（或者今天根本没有做的事情）完全自动化，达到实际上不需要人类进行判断的程度。
每种原型在启动项目前需要问的关键问题有所不同：
-
对于软件2.0项目：你如何知道你的模型实际上比现有基线表现更好？你有多大把握机器学习可能带来的性能改进会为你的业务创造价值？这些性能改进是否会导致数据飞轮？
-
对于人在回路系统：系统需要多好才能有用？如果系统只能自动化最终决策或生产最终产品的人类10%的工作，这对他们有用吗？还是会拖慢他们？你如何收集足够的数据使其达到那个有用的阈值？
-
对于自主系统：系统的可接受故障率是多少？你需要性能阈值达到多少个“9”才能避免对世界造成伤害？你如何保证它不会超过那个故障率？（例如，自动驾驶汽车团队投入大量精力构建模拟和测试系统。）
关于数据飞轮：对于软件2.0，我们讨论了能否构建一个数据飞轮，导致系统性能越来越好。数据飞轮是一个良性循环：随着模型变得更好，你能用更好的模型打造更好的产品，从而吸引更多用户；随着用户增多，用户生成更多数据，你可以用这些数据构建更好的模型。这个循环中的每个连接都很重要：要让更多用户允许你收集更多数据，你需要有一个数据循环（自动收集数据并决定标注哪些数据点的方法）；要让更多数据带来更好的模型，这取决于你作为机器学习从业者将更多、更细粒度的数据和标签转化为对用户表现更好的模型的能力；要让更好的模型带来更好的用户，你需要确保更好的预测确实让你的产品变得更好。
这三种项目原型在可行性-影响力矩阵上具有不同的权衡：
-
软件2.0项目：由于只是将已知可以自动化的事情做得更好，往往更可行，但由于它们已经在回答已知问题，影响力也往往较低。
-
自主系统：由于准确率要求通常很高，往往非常难以构建，但影响力也可能非常高，因为你正在替换一个根本不存在的东西。
-
人在回路系统：往往介于两者之间，你可以用这种机器学习产品范式构建以前不存在的东西，但由于仍然需要人类利用其判断来补充机器学习模型，影响力没有那么高。
有方法可以移动这些类型项目在可行性-影响力矩阵上的位置，使它们更有可能成功：
-
对于软件2.0项目：通过实施数据循环，构建持续改进的数据飞轮，并可能利用从用户与系统交互中收集的数据来未来自动化更多任务，从而使其具有更高的潜在影响力。
-
对于人在回路系统：通过良好的产品设计（产品本身的设计范式可以降低这些系统的准确率要求）以及采用不同的心态（让系统足够好并发布到真实世界，以便开始观察真实用户如何与之交互，并利用从“人在回路”中获得的反馈来改进模型），可以使其更可行。
-
对于自主系统：通过添加安全护栏，或在某些情况下添加人在回路，可以使其更可行（例如，自动驾驶汽车项目早期配备安全驾驶员，或引入远程操作以便在出现问题时人类可以接管系统）。
避免“工具迷恋”陷阱，以合理规模开始
尽管讨论了机器学习的复杂性、可行性等，我绝不是说在开始使用机器学习之前必须进行大量的规划。关键是确保你正在处理的项目是正确的项目，然后就直接开始投入。
特别是，我认为过去几年中出现的一种应该避免的失败模式是陷入工具迷恋的陷阱。随着MLOps学科的兴起，市场上出现了大量帮助机器学习过程不同部分的工具。我注意到，这导致很多人普遍认为在开始之前真的需要完美的工具。你不需要完美的工具来开始，也不需要完美的模型。仅仅因为Google或Uber在他们的技术栈中包含了特征存储或以特定方式服务模型，并不意味着你也需要那样做。
本课程的很多内容将尝试讨论在从生产角度以正确方式做事和不在项目早期引入过多复杂性之间的中间地带。这就是为什么全栈深度学习是一门关于以实用方式构建机器学习驱动产品的课程，而不是一门专注于最先进、最佳可能基础设施的MLOps课程。
一个与此概念相关的我很喜欢的演讲和博客文章是Covariant的一些人提出的“合理规模的MLOps”。其核心论点是：你不是Google，你可能拥有有限的计算预算，而不是整个云；你的团队可能人数有限；你可能没有无限的预算；你的数据量也可能有限。这些你与Uber或Google之间的差异，对于你解决问题的正确技术栈有影响。因此，值得分开考虑这些情况。
总结本节：机器学习是一项非常强大的技术，但它确实增加了很多复杂性。因此，在开始一个机器学习项目之前，你应该仔细思考是否真的需要机器学习来解决你正在解决的问题，以及考虑到它增加的复杂性，这个问题是否真的值得解决。让我们避免成为那些因为处理对业务无关紧要的事情而导致项目被取消的机器学习团队。
上一节我们探讨了何时应该使用机器学习，以及如何选择有价值的项目。现在，让我们来看看一旦决定启动一个机器学习项目，你将经历哪些步骤来实际执行它。这也将为你概述本课程后续可以期待的其他内容。
📊 机器学习项目生命周期概述
我们将使用一个修改版的问题作为贯穿始终的案例研究：姿态估计。我们的目标是构建一个在机器人上运行的系统，它接收机器人的摄像头数据流，并利用它来估计场景中每个物体在3D空间中的位置和方向（旋转），以便将其用于下游任务，特别是输入到一个单独的模型中，该模型将告诉机器人如何实际抓取场景中的不同物体。
机器学习项目像任何其他项目一样，始于规划和项目设置阶段。在这个阶段，我们可能进行的活动包括：决定研究姿态估计问题、确定成本、分配所需资源、考虑伦理影响等。这很大程度上是我们本节课到目前为止讨论的内容。
规划好项目后，我们将进入数据收集和标注阶段。对于姿态估计，这可能包括：收集我们将要训练模型的物体样本、设置传感器（如摄像头）来捕获这些物体的信息、实际捕获这些物体，并找出如何用真实值（如这些图像中物体的姿态）来标注这些捕获的图像。
关于机器学习项目生命周期的一个重要点是：这不是一条直线路径。机器学习项目往往非常迭代，每个阶段都可以反馈到之前的任何阶段，因为你对你正在处理的问题有了更多了解。例如，你可能会意识到获取数据来解决这个问题实在太难了，或者很难在3D空间中标注这些物体的姿态。但也许我们可以更便宜地标注像素级分割。那么，我们能否重新表述问题，利用我们从数据收集和标注中学到的东西来规划一个更好的项目？
一旦有了一些数据可以处理，你就进入了训练和调试阶段。在这里，我们可能会为模型实现一个基线（可能不使用复杂的神经网络，而只用一些OpenCV函数）。然后，找到一个最先进的模型并复现它，调试实现，迭代我们的模型，运行一些超参数扫描，直到它在我们的任务上表现良好。
这可以反馈到数据收集和标注阶段，因为我们可能意识到实际上需要更多数据来解决这个问题，或者可能发现我们用于标注数据的过程存在缺陷，需要重新审视数据标注过程。我们也可以一路循环回项目规划阶段，因为我们可能意识到这个任务比想象的要难得多，或者我们在规划阶段指定的需求相互冲突，需要重新审视哪些最重要。例如，我们可能认为需要将物体姿态估计的准确率要求设定为十分之一厘米，同时模型推理的延迟要求设定为百分之一秒以在机器人硬件上运行。然后我们可能意识到，我们无法同时满足极高的准确率要求和极快的推理速度。那么，是否可以放宽其中一个假设？
一旦你训练了一个在离线任务上表现相当好的模型，你的目标就是部署该模型，在真实世界中测试它，然后利用这些信息来确定下一步的方向。对于这个项目，这可能看起来像是在实验室中试点抓取系统（在推广给实际用户之前，在真实场景中测试），以及编写测试以防止回归和评估模型中的偏见，最终将其投入生产、进行监控并持续改进。
我们可以从这里反馈到训练和调试阶段，因为通常我们会发现，在离线数据集上表现良好的模型一旦进入真实世界，实际表现并不如我们想象的好。这可能是因为我们对模型的准确率要求是错误的（实际上需要更高的准确率），或者我们关注的准确率指标实际上并不是下游任务成功真正重要的指标。这可能导致我们重新审视训练阶段。
我们也可以循环回数据收集和标注阶段，因为我们在真实世界中可能发现的一个常见问题是：我们收集的训练数据与实际测试时看到的数据之间存在不匹配。我们可以利用从中学到的东西去收集更多数据，或者挖掘困难案例（挖掘我们在生产中发现的失败案例）。
最后，正如之前提到的，我们甚至可以一路循环回项目规划阶段，因为我们意识到我们选择的指标并不能真正驱动我们期望的下游行为。仅仅因为抓取模型准确，并不意味着机器人实际上能够成功抓取物体。因此，我们可能需要使用不同的指标来真正解决这个任务，或者我们可能意识到模型在真实世界中的表现并不那么好，可能需要为模型添加额外的要求（例如，它需要更快才能在真实的机器人上运行）。
这些是你在承担任何特定机器学习项目时可能进行的活动。但为了成功，你还需要一些跨项目的东西，我们也会在课程中讨论：你需要能够作为一个团队在这些问题上协作，并且需要拥有正确的基础设施和工具来使这些流程更具可重复性。这些也是我们将涵盖的主题。
总结本节课：机器学习是一项复杂的技术，因此你应该因为需要它或者认为它会创造巨大价值而使用它，但它不是万能的。它不能解决所有问题，也不会自动化你想自动化的每一件事。因此，让我们选择那些有价值的项目。但尽管如此，你不需要完美的设置就可以开始。让我们在本课程的剩余时间里，逐步讲解项目生命周期，并了解每个阶段以及我们如何利用它们来构建优秀的机器学习驱动产品。
Full Stack 深度学习 ｜ Full Stack Deep Learning 2022 p02 P02_Lab_Intro_and_Overview-实验室介绍与概览 -BV1k4YXznEjw_p2-
Al right，Welcome to the lab portion of the full stack deep learning 2022 horse In this video。
we’re going to go over at a high level the text recognizer application that we’ll be building through the rest of these labs。
We’ll also see how to get started with the labs on Google coabb。
and we’ll get comfortable with this Jupiter notebook format that we be using for the labs。
So we’ll start off at the lab Github repository， you can find a link for this in the description of this video。
This repository is where all of the material for the labs is saved。
And if we just scroll down a little bit to the readme section。
can find a description of the labs and all of these badges that say open in coabab。
So these are what we’re gonna want to click to start working on the labs。
So let’s go ahead and click this button to try out this first lab that just as an overview of the application architecture for what we’re gonna to be building this time。
Now， here I am in a Jupiter notebook environment that’s being hosted by Google completely free。
All you need is a Google account。 So you might be asked to sign in。😊。
Create a Google account if you don’t have one。 But once that’s done。
you’ve got a Jupyter notebook that can run Python for you。 So let’s go ahead and start running code。
If I click there on that cell and then hit either this play button here or hit the shift enter。
Give it a second to connect and our code is executed and pulled in our application and shown it inside of the notebook。
These Jupyter notebooks are composed of mixed different cell。
So you can see that as I navigate up and down using the arrow keys。
you can see there’s a highlight coming up that’s moving between the different cells of the notebook。
each cell is either a markdown cell that’s got text and media in it that’s describing what’s going on in the code。
or it’s got just plain code in it。 And then we can see the outputs underneath the cell。
This cell brought in this iframe class and then used it to embed this application that we’re gonna build the web page for it。
embed it directly in this notebook。 So notebooks are really great for mixing explanation。😊。
And code and interactive components may be built by that code It’s really ideal for these kinds of educational settings or it makes for really awesome documentation and some folks even do all their development in notebooks so that they can have this kind of rich interactive experience at their fingertips at all time the first thing that we’ve done here is embed the application that we’re gonna to build inside of the notebook as an iframe but just to underscore that we’re building a web application here for our text recognition。
I’m just going to go ahead and open that link This is the web page we’re gonna to build a web page that looks like this one and what we see on the left is where we can put images for our text recognition system go ahead and grab one of these example images down here and submit it and see what happens So if we wait a bit for our tensors to flow。
We get out the output。 So on the left， I read we believe that a comprehensive medical service free to the patient at the point of need and with one standard for all sick people。
etc ceter。 let’s take a look at this output。 So this is a Python string that came from our text recognition system and it says we believe that a comprehensive medical service free to the patient at the point of need。
and with one standard。 there’s an E there。 So it’s not perfect。's making some mistakes。
but you know not bad on this input， it does a pretty good job。 There’s some other samples there。
And you can， if you like upload your own inputs or edit the inputs that we have So what would happen maybe if we zoomed in on that section where we had the mistake previously。
we can submit that and take a look looks like the error goes away。
There’s some effective context ti that’s causing the network to make the mistake there。
These interactive applications with models embedded in them are helpful even during model development hopping back to that Jupiter notebook let’s continue through it and see how。
😊。
This app gets built There’s lots of text in this notebook that goes through all the stuff that I’m gonna be talking about at a high level in this video in much greater detail and talks about other choices for how we might have built this application I’m just going hit the highlights here and just strongly recommend that you check the notebook out for yourself and read through the text check out some of the links What we were just looking at above was the frontend or the user facingcing component of the application we’re gonna build that entirely in Python in order to keep things as simple as possible in order to make it possible for one person to be able to build this entire application and understand the whole thing themselves but that frontends not the whole thing。
The frontend is just the thing that we render that we show to people so that they can interact with our application The model doesn’t have to be in the same place The thing that takes in images and spits out text that machine learned algorithm that model doesn’t have to be in the same place as the front end for our application we put our backend in Amazon web services。
Whi is a tool for building serverless applications。
We’ll talk in detail about what that means and why you would choose that for your machine learning models。
But the most important thing about this frontend backend distinction is that our frontend our backend are independent of each other。
So just because we’ built this frontend that looks like what we just saw that was built in Python that doesn’t mean we can only use our model in that context。
So this cell here just runs a quick ping to that backend model sends it in image and then waits for it to get back with the predicted text。
So we use some python libraries here that are designed for working with web services。
you can check out the comments on that code for details on how it works we sent a URL to an image and we get back the text that was in that image So I read and since this is election year in West Germany Dr。
Adower is in a tough spot。 Joyce Egenton cables and that’s what the Python string above reads So our model is able to get the text there have。
Raw string and we can interact with it with our python codes。 we can put it somewhere else。
We can interact from the command line as well if we want do our frontend in javascript。
We’ve got these two separate pieces to see how these pieces fit together and how they relate to the rest of the application。
Let’s pull up a little diagram here。 This is an interactive diagram built with Miro which a nice visualization tool that let’s us see how all these pieces fit together。
nicely embedded in this Jupyter notebook here even have to go to another web page。
So we were starting up here at this user session So that’s the person interacting with our model。
and they’re talking to that frontend server that’s in Python using the library radio and that frontend servers doing a couple things。
It’s sending requests to that backend that does our predictions and AWs Lada And then you might have may have noticed of those buttons for flagging that allows users to give feedback on how the model is doing and we’ll use a tool called Gantry to。
😊，Take in that feedback， analyze it and use it to improve our models。
The information for how to run our server and how to run our backend。
those live in a container registry on Amazon web services containers are built using Docker So this is the handoff point between the way that we build models and the way that they end up in production interacting with users So I think it’s helpful at this point to actually stop and zoom back out and start over at the beginning to go from the training of models which if you’ve taken a class that covered deep learning and talked about building models with pietorch training on GPus that’ll be familiar to you and then through the process of iteratively building a model until we’re ready to put it in production so we’ll approach the handout from the other direction So to start off we need to determine what kind of computers we’re going to use to build our neural networks。
and because neural networks operate by doing sequences of really large matrix multiplications in other array。
Oper that are much faster on GPUus than on CPUus。 We’re gonna need GPUus to do our computing。
That’s one reason why we’re using Google coabab as one of the ways to deliver the labs。
It’s because Google coab comes with free GPUus to set that up in any coabab is to go to runtime change runtime type and pick a hardware accelerator and pick GPU So it should already be the case for this lab for you。
And to check that we actually have that GPU。 we can run this command and videodia Smi。
that’s kind of like a top command or a check of what processes are running but just for the GPU。
so we can see we’ve got a GPU。 It happens to be a Tesla P100。
you might get something different and nothing’s running on it yet because we haven’t done anything for building our full application。
we use the Lambda Las GPU cloud You should check them out if you’re interested in having access to more compute than just maybe your home machine or what coab provides for free because we’re doing all of our really heavy work on the GPU。
😊，We actually don’t need to write the majority of our model development code in some fast language In the end。
we’re going be running stuff on a GPU with lower levelve libraries anyway so it makes a lot of sense to use a language that’s easier to write in even if it’s a little bit slower because the performance bottleneck is the stuff that’s happening in C+ on a GPU the language that people use in deep learning is Python there’s really great libraries through developing neural networks in Python libraries that do GPU acceleration from Python and also have the things that we need to train neural networks like automatic differentiation so that we can run gradient descent easily so the library Pytorrch that kind of bridges the gap between Python and C+ gives us that GPU accelerated array math that we need along with a bunch of neural network primitives and architectures So the cell just demonstrates the basic way to interact with torch creating tensors or arrays manipulating the mathematically and then asking for gradients even Pytorrch is a little bit too low。
Level for when you’re developing neural networks。 It doesn’t include a highlevel framework for training neural networks or any of the other kinds of engineering type tasks you need like saving your work as your model is training So we use pytorch lightning as our highleve training engineering framework They’ve got really great documentation for pytorch lightning including a bunch of videos like this one embedded in the notebook。
We’ll talk a lot about how to use pytorch and how to use pytorch lightning in a future lab So we’ve got the libraries that we need to build our models What we’re missing is the kind of developer tools around model building creating a machine learning model is kind of like writing code。
we’re trying to create a computer program that can do something take some inputs return some outputs It’s just that this computer program happens to be a giant pile of numbers inside of tensors and so because things are similar to general software development we want developer tools but because they’re so different we need slightly different developer tools One of the most important things that。
😊，Need is the ability to sort of track what’s going on as we’re making changes and just baseline gi for version control doesn’t do a great job for that。
So the tool that we use to solve those problems to track our experiments as we’re trying out different configuration values and to track the artifacts or large binary files that we generate as we’re doing those experiments we choose weights and biases so let’s pull down a page from weights and biases and embedded in the notebook like we did with our application So you can see there’s lots of information here there’s charts that tell how our model was doing over time and across epos the inputs and outputs of our model。
the ground truth labels and then there’s all kinds of other information like system metrics included as well Lots and lots of rich information that gets logged for us using the weights and biases tool we can also take that information and turn it into nice dashboards for communicating results to people toss these on pull requests use them as blog posts to share our。
For internal communication， use them for checking on long running， training jobs。
all kinds of things。So we incorporate that information and add annotations and additional information to make it easier to draw insights。
Once we’ve trained and developed our model， we’ve run our experiments。
we need to turn that into something that we can use in production Pytorch lightning will be throughout training。
saving the current values of the weights of our model to disk。
we store those on weights and biases cloud。 And then when they’re ready to be deployed to production。
compile them down to an artifact that’s a little bit more independent that doesn’t require all of our development code because we’ve stored that artifact on weights and biases。
we can take a look at it。 And so our model file is saved alongside a bunch of really useful metadata and other information So we can see the lineage of this artifact what other artifacts were generated along the way to creating it and which jobs were run in order to create those We’ve now made it through the entirety of the application from training a model all the way up to putting it in production and interacting with。
😊，As a user so let’s take a look at that diagram again。
we’ve looked at all of these pieces in this short video and over the course of the labs。
we’ll see how all of these pieces are put together to create a working。
continually improved machine learning application I’m looking forward to。
全栈深度学习：第2章：实验室介绍与概览 🧪
在本节课中，我们将学习全栈深度学习2022课程的实验部分。我们将从高层次了解将要构建的文本识别应用程序，学习如何在Google Colab上开始实验，并熟悉我们将用于实验的Jupyter Notebook格式。
开始实验
首先，我们需要访问实验的GitHub仓库。您可以在本视频的描述中找到链接。这个仓库保存了所有实验材料。
在README部分，您可以找到实验的描述以及一系列标有“在Colab中打开”的徽章。点击这些按钮即可开始实验。
现在，我们点击第一个实验的按钮，该实验概述了我们将要构建的应用程序架构。
使用Jupyter Notebook
现在，我们进入了一个由Google免费托管的Jupyter Notebook环境。您只需要一个Google账户即可使用。
Jupyter Notebook由不同类型的单元格混合组成。您可以使用箭头键上下导航，高亮显示会在不同的单元格之间移动。
每个单元格要么是Markdown单元格，包含描述代码的文本和媒体；要么是纯代码单元格。我们可以在单元格下方看到输出结果。
这种格式非常适合混合解释、代码和交互式组件，非常适合教育环境或作为出色的文档。
应用程序概览
我们首先在Notebook中以内嵌iframe的形式嵌入了将要构建的应用程序。为了强调我们正在构建一个用于文本识别的Web应用程序，我们直接打开了该链接。
这是一个我们将要构建的网页。左侧是我们可以放置图像以供文本识别系统处理的地方。我们可以从示例图像中选择一张并提交，看看会发生什么。
等待片刻后，我们得到了输出结果。左侧显示了我们相信为患者提供全面医疗服务，在需要时免费，并为所有病人设定统一标准等。让我们仔细看看这个输出。
这是一个来自我们文本识别系统的Python字符串。它说我们相信为患者提供全面医疗服务，在需要时免费，并为所有病人设定统一标准。这里有一个“E”字母，所以它并不完美，会犯一些错误，但在这个输入上表现相当不错。
还有其他样本可供尝试。您也可以上传自己的输入或编辑我们已有的输入。例如，如果我们放大之前出错的部分并提交，错误可能会消失。这表明存在一些上下文效应导致网络在那里犯错。
这些嵌入了模型的交互式应用程序即使在模型开发过程中也很有帮助。
前端与后端
我们刚才看到的是应用程序的前端，即面向用户的组件。为了尽可能简化，并使一个人能够构建整个应用程序并理解所有内容，我们将完全使用Python构建前端。
但前端并不是全部。前端只是我们渲染给用户看，以便他们与应用程序交互的部分。模型不必放在同一个地方。接收图像并输出文本的机器学习算法（模型）不必与应用程序的前端放在同一个地方。
我们将后端放在亚马逊网络服务（AWS）中，这是一个用于构建无服务器应用程序的工具。我们将在后续详细讨论这意味着什么以及为什么为机器学习模型选择它。
但关于前端和后端区分最重要的是，我们的前端和后端是相互独立的。仅仅因为我们构建了这个用Python编写的前端，并不意味着我们只能在该上下文中使用我们的模型。
以下单元格快速向该后端模型发送一个ping请求，发送一张图像，然后等待它返回预测的文本。我们使用了一些专为处理Web服务设计的Python库。我们发送了一个图像的URL，并返回了该图像中的文本。
因此，我们的模型能够获取原始字符串形式的文本，我们可以用Python代码与之交互。我们可以把它放在其他地方，也可以从命令行与之交互，或者用JavaScript构建前端。
应用程序架构图
为了了解这些部分如何组合在一起以及它们与应用程序其他部分的关系，我们来看一个图表。这是一个用Miro构建的交互式图表，Miro是一个很好的可视化工具，让我们可以看到所有这些部分如何组合在一起，甚至可以直接嵌入到这个Jupyter Notebook中，无需跳转到其他网页。
我们从顶部的用户会话开始。这是与我们的模型交互的人。他们使用Gradio库与Python中的前端服务器通信。该前端服务器做两件事：向在AWS Lambda上执行预测的后端发送请求；以及处理用户通过标记按钮提供的反馈，我们将使用一个名为Gantry的工具来接收、分析这些反馈，并用其改进我们的模型。
关于如何运行我们的服务器和后端的信息，存储在亚马逊网络服务的容器注册表中。容器是使用Docker构建的。这是我们构建模型的方式与它们最终投入生产、与用户交互的方式之间的交接点。
从模型训练到生产部署
我认为在这一点上停下来，从头开始回顾是有帮助的：从模型训练开始（如果您上过涵盖深度学习的课程，并讨论过使用PyTorch在GPU上训练模型，这应该很熟悉），然后通过迭代构建模型的过程，直到我们准备好将其投入生产。所以我们将从另一个方向来处理这个交接。
首先，我们需要确定将使用哪种计算机来构建我们的神经网络。因为神经网络通过执行一系列非常大的矩阵乘法和其他数组操作来工作，这些操作在GPU上比在CPU上快得多。我们需要GPU来进行计算。这就是我们使用Google Colab作为提供实验的方式之一的原因，因为它提供免费的GPU。
要在Colab中设置GPU，请转到“运行时”->“更改运行时类型”，选择硬件加速器并选择GPU。对于本实验，应该已经为您设置好了。为了检查我们是否确实拥有GPU，我们可以运行命令 nvidia-smi。这类似于top命令或检查正在运行的进程，但只针对GPU。我们可以看到我们有一个GPU（例如Tesla P100），目前还没有任何程序在上面运行，因为我们还没有为构建完整应用程序做任何事情。
我们使用Lambda Labs GPU云。如果您有兴趣获得比家用机器或Colab免费提供的更多计算资源，可以查看它们。
因为我们在GPU上完成所有繁重的工作，实际上我们不需要用某种快速语言编写大部分模型开发代码。最终，我们将在GPU上使用较低级别的库运行程序，所以使用一种即使稍慢但更容易编写的语言是很有意义的，因为性能瓶颈发生在GPU上用C++进行的操作上。
深度学习领域使用的语言是Python。有非常棒的库用于在Python中开发神经网络，这些库可以从Python进行GPU加速，并且拥有训练神经网络所需的功能，如自动微分，以便我们可以轻松运行梯度下降。
PyTorch库弥合了Python和C++之间的差距，为我们提供了所需的GPU加速数组数学运算以及一系列神经网络基元和架构。以下单元格演示了与Torch交互的基本方式：创建张量（或数组）、进行数学操作，然后请求梯度。
然而，PyTorch在开发神经网络时仍然有点底层。它不包含用于训练神经网络的高级框架或任何其他类型的工程任务，例如在模型训练时保存工作。因此，我们使用PyTorch Lightning作为我们的高级训练工程框架。PyTorch Lightning有非常棒的文档，包括嵌入在Notebook中的视频。我们将在未来的实验中详细讨论如何使用PyTorch和PyTorch Lightning。
开发工具
我们拥有了构建模型所需的库。我们缺少的是围绕模型构建的开发工具。创建机器学习模型有点像编写代码：我们试图创建一个可以执行某些操作（接收一些输入，返回一些输出）的计算机程序。只是这个计算机程序恰好是张量中的一堆巨大数字。
因为这与通用软件开发相似，我们需要开发工具；但又因为它们如此不同，我们需要略有不同的开发工具。最重要的事情之一是能够在我们进行更改时跟踪进展情况，而基线Git版本控制在这方面做得并不好。
因此，我们使用Weights & Biases工具来解决这些问题：在我们尝试不同配置值时跟踪实验，并跟踪在进行这些实验时生成的工件（或大型二进制文件）。
让我们在Notebook中下拉一个Weights & Biases页面，就像我们对应用程序所做的那样。您可以看到这里有很多信息：图表显示我们的模型随时间推移和跨周期的表现；模型的输入和输出、真实标签；以及各种其他信息，如系统指标。使用Weights & Biases工具为我们记录了大量丰富的信息。
我们还可以利用这些信息创建漂亮的仪表板，用于向人们传达结果：将它们放在拉取请求上、用作博客文章分享、用于内部沟通、检查长时间运行的训练作业等。我们整合这些信息并添加注释和附加信息，以便更容易得出见解。
从训练到生产
一旦我们训练和开发了模型，运行了实验，就需要将其转化为可以在生产中使用的东西。PyTorch Lightning会在整个训练过程中将模型权重的当前值保存到磁盘。我们将这些存储在Weights & Biases云上。然后，当它们准备好部署到生产环境时，我们将它们编译成一个更独立的工件，该工件不需要我们所有的开发代码。
因为我们将该工件存储在Weights & Biases上，我们可以查看它。因此，我们的模型文件与一堆非常有用的元数据和其他信息一起保存。我们可以看到该工件的谱系：在创建它过程中生成了哪些其他工件，以及为了创建那些工件运行了哪些作业。
总结
在本节课中，我们一起学习了全栈深度学习2022课程的实验部分。我们从高层次了解了将要构建的文本识别应用程序，学习了如何在Google Colab上开始实验，并熟悉了Jupyter Notebook格式。
我们探讨了应用程序的前端与后端架构，了解了它们如何通过Web服务进行通信。我们还回顾了从模型训练（使用GPU、PyTorch和PyTorch Lightning）到使用Weights & Biases进行实验跟踪和工件管理，再到最终通过容器化部署到生产环境（如AWS Lambda）并与用户交互的完整流程。
通过图表，我们看到了所有这些部分（用户会话、前端服务器、后端预测服务、反馈收集与分析工具、容器注册表）如何组合在一起，形成一个持续改进的机器学习应用程序。
在接下来的实验中，我们将深入探讨如何将这些部分组合在一起，构建一个可工作的、持续改进的机器学习应用程序。
Full Stack 深度学习 ｜ Full Stack Deep Learning 2022 p03 P03_Lab_01-PyTorch中的神经网络 -BV1k4YXznEjw_p3-
Hey， friends， welcomee to the first official lab of the Fullt Deep learning 2022 edition。
I am Charles， and I helped revamp the labs for this year and I’m really excited to be sharing it with you。
So let’s dive in。 This is the repository for the labs this year on Gitthub。
This is where all the information for the labs lives。 And so you can see there’s folders for labs 1。
2 and 3 here already。 we’re going to keep adding more lab folders as we go through the labs。
we’re gonna iteratively build a deep learning code base for training are text recognition system。
If we take a look at what that looks like in lab1。 it’s pretty simple。
there’s just this one library text recognizer and the text recognizer has basically two pieces for handling data and for handling models。
If we look at the directory for lab 2。 We’ll see now we’ve got an extra library for training。
And we still have that text recognizer。😊，Library， but it’s gotten more complicated。
we’ve added more pieces。 We’ve added lightning models， for example。
if you’re following along with local development， then what you’ll do is clone this repository and every week as we release new labs you can clone it again if you’re watching this after the course is ended you’ll just clone it once and you’ll have all the labs and to do a lab you’ll go into each lab folder look at the notebooks and open this notebook in Jupiter from the command line So for more details on how to get set up with local development you can check out the video on getting local development set up doing that requires that you have a Linux machine with a GPU available locally or set one up in the cloud the easiest way to get started is just use Google coab So jumping back to the main page to the readme and scrolling down we can find this table which has all the labs in it and then these badges that you can click that will open the notebooks up in coab So I’m gonna go ahead and click this one for that first lab。
Deep neural networks and pytorch。 So in these videos。
I’m not gonna go line by line through these labs。 there’s lots of text and content in there that explains everything that’s going on in them。
So what I’m going to do in these videos is give you an overview of how to use each lab and hit on some of the highlights to start off。
we got to get our bearings here in these notebooks。
So the first thing that I wanted to walk through was this setup cell here。
this will be the thing you got to run at the beginning of each lab。
So it’s go ahead and run this on coabab it clones the repository it sets up the environment。
if you’re running things locally you don’t need to clone。
you don’t need to set up the environment just gets us in the right directory so that we’re ready to go once it’s done it prints out the directory that we’re in lab ones directory and the contents there So it’s got that text recognizer library。
So from there we can just keep going and go through the lab if you want But heads up that these notebooks are not static。
you can edit them。😊，And change them as you need so for example。
I might be curious what’s going on inside of that folder so I can add an additional cell and in that cell。
write my own python code， write my own shell commands。
for example I can issue a shell command find and look at just what’s inside that text recognizer directory so it shows me the contents of that directory the models folder the data folder and all those python files I can also run python code so I could import textrer data u as u and then one thing will do a lot in the labs is use this double question mark to examine an object let’s run that and this will pull up inside the notebook the source code and the documentation for some python objects so you can also print objects if you want it’s maybe more familiar or display objects but this is a nice way for taking a look at the actual code here we can see that this utilities library contains this base data set class。
😊，The rest of this lab， what we’re going to do is we’re going to work up to understanding what that base dataset class is doing and what the components of the models part of the text recognizer library are doing and we’re going do that while learning about the ptorrch library as a whole So the way that this lab notebook works is that we first build the full training of neural network with just the very fundamental parts of ptorrch So torch do tensors and math operations on them and then we iteratively replace bits of that with higher level abstractions inside torch and then eventually from the text recognizer library and we eventually end up with a much cleaner neater way for fitting neural networks and for trying out different datas。
different models and then in following labs will continue to elaborate on top of that So just two last things that are want to cover at a meta level about these labs and notebooks the first one is that you’re going to want to run these notebooks linearly。
😊，Top to bottom。 if I’m scrolling along here and reading and not executing the cells and then I come across one that’s interesting。
and let’s say I want to run this one。 and I run it。 I’ll get an error says Xtrain is not defined。
And the reason why is because it’s expected that you’ve run the cells up to this point when you get to this cell。
So Xtrain is defined up here when we’re building our data So in this happens what you can do is run all the cells before the one that you’re currently running again the way to do that in coab is runtime run before there should be a way to do that in other types of notebooks if we then execute this cell。
we’ll see that it runs all the variables in it been defined all the libraries have been imported and things have been set up if you get into a situation where that doesn’t help then what you should do is start the notebook over so thats in coab run time restart run time So this will get you back to square one And then the last thing that I wanted to touch on was the exercises at the end of the lab。
😊，Notbooks these exercises are here in case you want to dive deeper on this particular component of the stack。
They’re not mandatory in any way。 They’re there for your learning benefit。
Maybe you don’t feel like learning more about ptorch this time around。 That’s fine。
Maybe later you’re gonna care to learn more about AWS lambda and the exercises are marked with these little stars。
So the stars are meant to indicate how difficult really how much effort it’s going take to complete an exercise。
So the exercise just has one star， you’re probably just changing around arguments for a function that’s already implemented for you and seeing what they do or taking a component of a library that we’ve already talked about and using it in a slightly different way。
Other exercises are marked with up to three stars3 star exercise is going require you to read the documentation of one of the libraries that we’re using and extend what we did。
So add new pieces I new functionality。 And then you’ll also find some two star exercises that are somewhere in between That’s all I have to say about this lab。
So go ahead， dive right in。😊，Learning about Pytorrch and Torch。nn。
and I will see you in the next lab on Pytorrch Lightning and convolutional networks。
全栈深度学习：Lab 01：PyTorch中的神经网络 🧠
欢迎来到2022版全栈深度学习的第一个正式实验课。我是Charles，今年我协助更新了实验内容，非常高兴能与大家分享。让我们开始吧。
概述
在本节课中，我们将学习如何使用PyTorch构建和训练神经网络。我们将从最基础的PyTorch张量操作开始，逐步引入更高级的抽象，最终构建一个简洁的文本识别系统代码库。通过这个过程，你将掌握PyTorch的核心组件。
实验环境与结构
今年的所有实验资料都存放在GitHub仓库中。你可以看到已经存在Lab 1、2、3的文件夹，随着课程进行，我们会添加更多内容。我们将迭代式地构建一个用于训练文本识别系统的深度学习代码库。
查看Lab 1的目录结构，它目前非常简单，只有一个名为text_recognizer的库，该库主要包含处理数据和模型的两个部分。
当我们查看Lab 2的目录时，会发现新增了一个用于训练的库，同时text_recognizer库也变得更为复杂，例如我们添加了Lightning模型等组件。
如果你在本地进行开发，需要克隆此仓库。每周我们发布新实验时，你可以再次克隆更新。如果你在课程结束后观看，只需克隆一次即可获得所有实验内容。进行实验时，进入每个实验文件夹，在Jupyter中打开相应的笔记本即可。
关于如何设置本地开发环境的更多细节，你可以查看关于设置本地开发的视频。这要求你拥有一台带GPU的Linux本地机器或在云端设置一台。最简单的方式是使用Google Colab。
回到主README页面并向下滚动，你可以找到一个包含所有实验的表格，以及可以点击的徽章，这些徽章将在Colab中打开相应的笔记本。我将点击第一个实验的徽章。
实验内容：深度神经网络与PyTorch
在这些视频中，我不会逐行讲解实验内容。笔记本中已有大量文字和内容来解释其中发生的一切。因此，在这些视频中，我将概述如何使用每个实验，并重点介绍一些关键内容。
首先，我们需要熟悉这些笔记本。我想介绍的第一个内容是这里的设置单元格。这是在每个实验开始时都需要运行的部分。在Colab上运行它会克隆仓库并设置环境。如果你在本地运行，则不需要克隆或设置环境，它只是将我们带到正确的目录，以便我们准备就绪。
运行完成后，它会打印出我们所在的目录（Lab 1的目录）及其内容。我们可以看到text_recognizer库。从那里，我们可以继续并完成实验。
请注意，这些笔记本不是静态的，你可以根据需要编辑和更改它们。例如，我可能好奇那个文件夹里有什么，因此我可以添加一个额外的单元格，并在该单元格中编写自己的Python代码或Shell命令。
例如，我可以发出一个Shell命令来查看text_recognizer目录内的内容：
find text_recognizer -type f -name "*.py"
我也可以运行Python代码，例如导入text_recognizer.data.utils，并在实验中使用双问号??来检查一个对象。运行这个命令会在笔记本内显示该Python对象的源代码和文档。你也可以使用更熟悉的print或display来查看对象。
在这里，我们可以看到这个实用程序库包含一个BaseDataset类。
在本实验的其余部分，我们将逐步理解这个BaseDataset类的作用以及text_recognizer库模型部分的组件。我们将在学习整个PyTorch库的过程中完成这些。
这个实验笔记本的工作方式是，我们首先仅使用PyTorch最基础的部分（即torch张量及其数学运算）来构建一个完整的神经网络训练流程。然后，我们逐步用torch内部更高级的抽象，最终用text_recognizer库中的组件替换其中的部分。最终，我们得到一种更简洁、更清晰的方法来拟合神经网络，并尝试不同的数据和模型。在后续的实验中，我们将在此基础上继续扩展。
使用笔记本的注意事项
关于这些实验和笔记本，最后还有两点需要说明。
第一，你需要按顺序从上到下线性地运行这些笔记本。如果我在这里滚动阅读但没有执行单元格，然后遇到一个感兴趣的单元格并运行它，可能会遇到错误，例如“Xtrain未定义”。这是因为运行到该单元格时，期望你已经运行了之前的所有单元格。如果发生这种情况，你可以重新运行当前单元格之前的所有单元格。在Colab中，可以通过“运行时”->“运行之前”来实现。其他类型的笔记本也有类似功能。
如果我们执行这个单元格，会看到它运行了，所有变量都已定义，所有库都已导入，一切已设置就绪。如果遇到这种方法也无法解决的问题，那么你应该重新启动笔记本。在Colab中，选择“运行时”->“重启运行时”，这将使你回到起点。
第二，我想提一下实验笔记本末尾的练习题。这些练习题是为了让你在想要深入了解技术栈的某个特定组件时使用。它们不是强制性的，只是为了你的学习受益。也许这次你不想深入学习PyTorch，那也没关系。练习题标有小星星，星星的数量旨在表示完成该练习的难度或所需的工作量。
-
一星练习：可能只是更改已实现函数的参数，观察它们的作用，或者以稍微不同的方式使用我们已经讨论过的库组件。
-
三星练习：需要你阅读我们使用的某个库的文档，并扩展我们所做的工作，添加新的功能或组件。
-
你还会发现一些介于两者之间的二星练习。
关于本实验我要说的就是这些。请开始深入学习PyTorch、torch.nn吧。我将在下一个关于PyTorch Lightning和卷积网络的实验中与大家再见。
总结
本节课我们一起学习了如何设置和使用全栈深度学习的实验环境，了解了实验代码库的迭代式结构，并明确了第一个实验的目标：从PyTorch基础开始，逐步构建一个文本识别系统。我们还掌握了有效使用Jupyter笔记本进行实验的注意事项。现在，你可以开始动手探索PyTorch的世界了。
Full Stack 深度学习 ｜ Full Stack Deep Learning 2022 p04 P04_Lab_02-PyTorch_Lightning与卷积神经网络 -BV1k4YXznEjw_p4-
Hey there folks， welcome to the second lab of FSDL 2022。
today we’ll be covering Pytorch lightning and convolutional neural networks。So once again。
we’re gonna start from the Github repo for the labs in that repo。
whether you’re on Gitthub or if you’ve cloned it locally， if you navigate to lab2 folder。
you’ll see in the notebooks folder， we’ve got two lab notebooks this time one covering ptorrch lightning and one covering convolutional neural networks I’m gonna use the notebooks via coab again So going back to that main page scrolling down to the bottom and clicking these badges to open the notebooks in coLab So the first lab 2 a introduces ptorrch lightning。
which is our training framework on top of ptorrch in lab1 we wrote a training loop using just components of torch starting with torch tensors and then adding features of torch do NN and other libraries inside pytorrch and we ended up with something like this pseudocode here that worked well for the problems we were trying to solve in that notebook fixed data set fairly simple models as we started to try and add more features like GPU acceleration。
We found out that it was insufficiently flexible to handle all the problems that we wanted to handle And in general。
it’s also hard to reuse a pytorch training loop that you write yourself unless you’ve written enough of them that you know all the things that you’re gonna want to do in the future Also this is something that lots and lots of people want to do if you’re using pytorch you’re probably using it to train neural networks so you’re probably gonna want to do a lot of these things with data loaders and moving to and from GPUus and applying gradients。
this is really kind of like shared boilerplate and it’s got lots of sharp edges， difficult things。
surprising behaviors this is a classic case where you want to use a library or a framework rather than rolling your own and pytorrch lightingning is one of the most popular frameworks for training in pytorch Pytorch lightingning is a really rich library with tons of features and components we’re going to focus on some of the most important ones which are lightning modules that go on top of our torch modules and connect them to training Lightning data modules which organize our torch data loaders and data。
😊，The Pytorch Lightning trainer， which takes those things。
puts them together and does our actual training loop and our validation loop and our testing loop and then pytorch lightning callbacks that allow us to add features to training turn on model checkpointing and turn off model checkpointing without having to rewrite code So the notebook walks you through Pytorch lightning and its components I’m not gonna focus on that bit of it and I’m instead going focus on how lightning gets used in the full stack deep learning code base that we’re building up to make this text recognition system So with each lab we’re going add additional components to this code base slowly iteratively building up to a code basease that’s capable of training models and deploying them and monitoring them The previous lab all we had was models and data Now the textre our library has a new component lit models that has our pytor lightning modules in it right now there’s just a baseline version will add more throughout the course of the labs bigger。
Is now we have a new library training on top of our text recognizers。
this contains the things that we need to train our models。 So most importantly。
this run experiment do pi script。 you’ll be using this script to train models。
you can run it from inside a Jupiter notebook。 There’s some example commands here that show you how to get the help documentation from the script。
but you don’t necessarily have to just use this experiment running framework as a script。
It’s also an import module inside training。 So you can bring it into the Jupyter notebook and take a look at its components。
if you want to play around with them dynamically。 So where this training script really gets used is in the second lab to train convolutional neural networks。
So let’s jump over to that。 So the second lab lab to be covers convolutional neural networks。
There’s a lot of material on what convolutions are and why we use them and also how to design convolutional neural networks。
So read through the notebook to get that information In this video I wanted to focus on using the run experiment。
😊，That’s this section here。 So I’ve already run through the notebook up to this point。
remember that you want to run the notebooks top to bottom。
otherwise the bottom cells won won’t run So this cell here will run training it uses GP if they’re available I’m going run this and walk you through the outputs that appear So first we see just some information from p towards lightning about what hardware we have available some logging messages then we see a summary of our model including number of trainable parameters and what layers we have and then we see this progress bar pop up progress bar is showing our progress through and epoch of training you can see as batches go through this progress bar is filling up there’s also some metric reporting that’s happening behind my head once we get towards the end of the epoC validation starts you can see the validation started there and once validation finishes we run the model on the test set we get some reported metrics there performance on training validation and test and we also see that the model got saved as。
Checkpoint this collection of cells here walks you through how to reload models from a checkpoint and then run them and play around with them in the notebook。
I think it’s really important to always interact with your models and your data in order to understand your problem better It’s really easy to just get obsessed with metrics and charts and watching numbers go down or go up and lose connection with the actual problem that you’re trying to solve。
And the end result of this is always going to be a model that looks good on paper false flat on its face as an actual component of an m powered product。
I always like to as quickly as possible get my models back into an interactive context where I can play with them。
So that’s what this cell is doing here sending different inputs through our model One of the neat things about this is you’ll pretty quickly discover that there are some ambiguous inputs in this dataset。
So the model says that this particular input is probably a number0 and that’s a good guess。
but it might be a capital O or a lowercase It might even be like a kind of slanted D So there’s some。
😊，Examp in this dataset set， some classes that are really easy to confuse with each other。
and that actually means that doing character recognition at a single character level is probably a bad idea because normally you would use context to disambiguate is this a zero or a capital or a lowercase that’s going depend on what other letters are around it but our model because it just sees one character at a time can’t really disambiguate between these cases。
That’s something that you could find out by looking at the data but you wouldn’t see it necessarily in the metrics。
So then the last thing that we do in this lab is try and resolve that issue with the ambiguity of single characters really in the end we don’t care about recognizing individual characters we care about handling text that people are submitting to us and no one’s going want to sit around and submit one character at a time to our model The first step that we want to do is work on lines of handwritten text and our data only has individual characters so you might think we have to go back and get new data in order to keep going one of most。
😊，Important tricks for avoiding a really expensive and complex data collection process is to use data synthesis to bootstrap the data that you have so you can train a basic model that can get out there in the world and then start collecting the data that you really want to train on。
We build a kind of fake lines of handwritten text data set by using a data set of sentences the brown corpus and then just using those handwritten text and digits to create these lines of text。
So let’s see what that looks like。 We’ve just taken those individual images of handwritten characters and put them next to each other to create this line of text。
So this synthetic line data is not perfect。 Let’s see a few more of them。😊，Yeah。
these don’t look exactly like text that you might encounter in the real world。
The handwriting style is inconsistent， it kind of maybe looks like a ransom note of like pasted together text。
but this is a start It’s really great for codebased development and idea generation to at least have something that looks closer to the actual data that will bring up problems that are actually gonna come around when you have the real data to try out more complex modeling approaches。
data handling approaches and you can incorporate it into training alongside real data to improve your model performance and get it over the line so that you’re able to put it out there。
全栈深度学习：Lab 02：PyTorch Lightning与卷积神经网络
在本教程中，我们将学习如何使用PyTorch Lightning框架来简化训练流程，并探索卷积神经网络（CNN）在图像识别任务中的应用。我们将从基础概念开始，逐步构建一个文本识别系统。
概述
欢迎来到FSDL 2022的第二个实验。本节课我们将涵盖两个核心主题：PyTorch Lightning训练框架和卷积神经网络。我们将从GitHub仓库的Lab2文件夹开始，使用两个Jupyter Notebook进行学习。第一个Notebook介绍PyTorch Lightning，第二个则深入讲解卷积神经网络。我们将使用Colab来运行这些Notebook。
PyTorch Lightning简介
在第一个实验中，我们使用纯PyTorch组件编写了训练循环。虽然这对于简单数据集和模型是有效的，但随着我们尝试添加更多功能（如GPU加速），我们发现其灵活性不足。此外，自定义的训练循环通常难以复用，且涉及大量重复的样板代码。PyTorch Lightning是一个流行的框架，旨在解决这些问题，它封装了训练、验证和测试的通用逻辑。
PyTorch Lightning功能丰富，我们将重点关注几个核心组件：
-
LightningModule： 在
torch.nn.Module基础上构建，将模型与训练流程连接起来。 -
LightningDataModule： 用于组织
torch的数据加载器和数据。 -
Trainer： 接收上述组件，并执行实际的训练、验证和测试循环。
-
Callbacks： 允许我们添加功能（如模型检查点），而无需重写代码。
Notebook将引导你了解这些组件。在本教程中，我们将重点展示Lightning如何被集成到我们正在构建的全栈深度学习代码库中，以实现文本识别系统。
构建代码库
随着每个实验的进行，我们将逐步向代码库添加组件，最终构建一个能够训练、部署和监控模型的系统。
在上一个实验中，我们只有模型和数据。现在，我们的text_recognizer库新增了一个lit_models组件，其中包含了PyTorch Lightning模块。目前只有一个基线版本，我们将在后续实验中添加更多。
更重要的是，我们现在在text_recognizer之上有了一个新的training库，它包含了训练模型所需的一切。其中最关键的是run_experiment.py脚本，你将使用这个脚本来训练模型。你可以从Jupyter Notebook内部运行它，也可以将其作为模块导入并动态查看其组件。
这个训练脚本的真正用武之地是在第二个实验（Lab 2b）中，用于训练卷积神经网络。
卷积神经网络
第二个实验（Lab 2b）涵盖了卷积神经网络。Notebook中包含了大量关于卷积是什么、为何使用卷积以及如何设计CNN的材料。请仔细阅读以获取这些信息。
在本视频/教程中，我想重点介绍如何使用run_experiment脚本。以下是相关步骤。
我已经按顺序运行了Notebook直到这个部分。请记住，你需要从上到下运行Notebook，否则底部的单元格将无法执行。
这个单元格将启动训练过程，如果可用，它会使用GPU。我将运行它并带你查看出现的输出。
首先，我们看到一些来自PyTorch Lightning的关于可用硬件和日志信息。接着，我们看到模型的摘要，包括可训练参数的数量和各层信息。然后，一个进度条弹出，显示我们在一个训练周期中的进度。你可以看到随着批处理的进行，进度条在填充。同时，后台也在报告一些指标。
接近周期结束时，验证开始。验证完成后，我们在测试集上运行模型，得到一些报告指标，包括在训练集、验证集和测试集上的性能。我们还会看到模型被保存为一个检查点文件。
接下来的几个单元格将引导你如何从检查点重新加载模型，然后在Notebook中运行并与它们交互。
我认为，为了更好地理解你的问题，始终与你的模型和数据进行交互是非常重要的。人们很容易只关注指标和图表，看着数字上升或下降，却与实际要解决的问题脱节。这样做的最终结果往往是，一个在纸面上看起来不错的模型，在实际作为AI驱动产品的组件时却表现不佳。
我总是喜欢尽快将我的模型放回一个可以交互的环境中，以便进行探索。这就是这个单元格所做的事情：向我们的模型发送不同的输入。这样做的一个好处是，你很快就会发现数据集中存在一些模糊的输入。
例如，模型可能认为某个特定输入是数字“0”，这是一个合理的猜测，但它也可能是大写字母“O”或小写字母“o”，甚至可能是一个倾斜的“D”。因此，在这个数据集中，有些类别确实很容易相互混淆。
这实际上意味着，在单个字符级别进行字符识别可能不是一个好主意，因为通常你会使用上下文来消除歧义（例如，这是“0”还是“O”取决于周围的字母）。但由于我们的模型一次只能看到一个字符，它无法真正区分这些情况。通过查看数据你可能会发现这一点，但在指标中不一定能直接看到。
解决单字符歧义问题
我们在本实验最后要做的事情，就是尝试解决单字符歧义的问题。归根结底，我们并不关心识别单个字符，而是关心处理人们提交给我们的文本。没有人会愿意一次只向我们的模型提交一个字符。
我们想要的第一步是处理手写文本行。但我们的数据只有单个字符，你可能会认为我们必须回去收集新数据才能继续前进。避免昂贵且复杂的数据收集过程最重要的技巧之一，就是使用数据合成来引导你已有的数据。这样你可以训练一个基础模型，将其部署到实际环境中，然后开始收集你真正想要训练的数据。
我们通过使用一个句子数据集（布朗语料库），然后利用那些手写字符和数字来创建这些文本行，从而构建了一种“伪造”的手写文本行数据集。让我们看看它是什么样子。
我们只是将那些单个的手写字符图像并排放置，以创建这行文本。这种合成的行数据并不完美。让我们多看几个例子。
是的，这些看起来并不完全像你在现实世界中可能遇到的文本。笔迹风格不一致，看起来可能有点像拼贴在一起的勒索信文本。但这只是一个开始。对于基于代码的开发和想法生成来说，拥有一些看起来更接近实际数据的东西是非常棒的。这可以让你在获得真实数据之前，就能尝试更复杂的建模方法和数据处理方法，并发现实际会遇到的问题。你还可以将其与真实数据一起纳入训练，以提高模型性能，使其达到可以部署的水平。
总结
在本节课中，我们一起学习了如何使用PyTorch Lightning框架来简化和组织神经网络的训练流程。我们探索了卷积神经网络的基本原理及其在图像识别中的应用。通过动手实验，我们训练了一个CNN模型用于字符识别，并发现了单字符识别任务的局限性。最后，我们引入了数据合成的概念，通过创建合成文本行数据，为处理更真实的文本识别任务迈出了第一步。记住，与模型和数据保持交互是深入理解问题和构建有效AI产品的关键。
Full Stack 深度学习 ｜ Full Stack Deep Learning 2022 p05 P05_Lab_03-Transformers与段落 -BV1k4YXznEjw_p5-
Hey folks， welcome back to the FullSt deep Learning 2022 edition Labs， we’re on Lab 3。
and in this lab we’re going to introduce the transformformer architecture and the ResNe transformformer model that we’ll be using in our text recognition system and we’re going to apply it to paragraphs of handwritten text。
So as always， we’re starting from the Github repo for the labs here。
If you’re running the labs on coabab， just scroll down to the bottom here and click this opening coabab link。
but I also wanted to show what this looks like if you’re running the labs on your own machine I’ve got a shell here on my Linux machine that’s got a GPU and I’ve activated the Conda environment for the course FSDL text Rer 2022 and make sure that you have gone through the local setup instructions so that you’re ready to go and then let’s run our Jupiter server if you have a machine with an attached screen。
it should open up automatically otherwise again see the setting up local development instructions for how to get Jupyter running So from the root directory of the repository navigate to lab3 notebooks and the transformers notebook。
the lab 3 notebook that’s new this time and make sure to run that first setup cell and to in general。
run the cells from top to bottom as you’re reading through the notebook。
Lb covers some of the core ideas of the transformer architecture for sequence modeling。
including why transformers have largely replaced recurrent neural networks for sequence modeling over the past couple of years and as with the other overview videos。
I’m going to focus on the components of this lab that touch on the FSDL text Rer codebase and especially the components that we’ll be using in future labs。
So one of the most important pieces that’s introduced in this lab is a new model。
the Resnet transformer model He uses a small Resnet to encode our input image and then a transformer to decode that image into a sequence So let’s take a look at the forward method for that resnet transformer So this is what gets called if you just call the model directly So if we look in the forward method it takes images as input and returns a sequence of class labels that we can interpret as a sequence of characters in the input image focusing in this code on the most important bits。
we encode the inputs here that’s done with the resnet so you can check out the encode method if you want to see the details of how that works and then we start a loop we loop over the length of the sequence and we take the current outputs and feed them in alongside those encoded inputs into our transformer decoder here in this self do decocode call and the rest of this is just setting。
upp so that we append the outputs as we’re going and then we stop once all of the sequences in the batch have reached the end。
This is the way that our transformer model is going run during inference。
But one of the most important things about transformers is that they run differently during inference and during training。
we can see that difference， if we look at the transformer lit model。
So this is the lightning module that wraps around our basic pytorch module and allows us to train it。
So if we look at the training step method And remember this training step method is what getss called in pytorch lightning during each step of training。
we scroll down to the actual source code for the training step。
we can see that takes that batch of both input images and ground truth labels and passes both of those things to this method teacher forward So that’s not the normal forward method。
And if we look at that teacher forward method and see what’s different about it。
the most important difference is this teacher forward method has both inputs and。😊。
Ground truth labels as part of its signature。 So this is obviously something we can’t do during inference when we don’t have ground truth labels。
but when we do have them， it’s a much simpler process to run our transformer。
we just encode the inputs and then decode them using the ground truth labels instead of the model’s outputs in that decoding step。
This is the sort of secret sauce that makes the transformer architecture a lot more scalable。
or at least one of the most important components of what makes the transformer architecture really scalable。
this decocode call here because it’s all happening at once and not rolled out into a for loop is really amenable to parallelization。
So that’s parallelization across the length of a sequence。
not just parallelization across a batch or parallelization of model computations。
So there’s lots more information about the transformer architecture and intuitions about how it works and its relation to re currentrent architectures in this lab。
I want to focus on how we use this architecture to。😊，To train our text recognition system。
The main reason we’re using this architecture here is because we want to have variable length outputs for fixed size input。
So we want to decouple how big input is and how big output is。
And the multilayer perceptron s fully connected architectures we’ve looked at and the convolutional architectures we’ve looked at aren’t able to do that。
So you see why we need that， let’s take a look at some of our data。
So this cell here draws random elements from a batch and shows it to us。
So looking at the text there。 Some thought the result would be all sorts of horrible illnesses brought on by the confined atmosphere。
the shareholders who travel by it will be so heartily sick， etc ce。
We have this paragraph here in this image。 And if we run this cell again and get a new input。
We can see it’s the same size image。 But the length of the paragraph is different。
Let’s run it again。The lengths of these paragraphs is changing each time that we execute this cell。
even though the size of the input image is the same。
And so our transformer decoder can produce variable length sequences from that encoded input。
We’ve now got all the pieces that we need to train a model to take in images of multiple lines of text and output the content of that image。
This model is quite a bit bigger than the previous models that we’ve been working with。
we’re using a resnet a lot bigger than the previous convolutional networks we’ve been using and now we’ve tacked on this transformer。
We’re definitely going need GPU acceleration from this point on。
and we’re going to need some tricks to make sure that we can do our training and our experimentation quickly enough。
So this is the cell that runs training here and has got some flags past to our run experiment script to try and make that training quicker。
So I’m going to start it running and then start describing some of these flags here。've got。😊。
A data class in a model class， theyre new data classes and model classes， the I am paragraphs。
dataset set of paragraphs of handwritten text and this resnet transformer model class。
we also use the loss flag to say which lightning model we want to use which training wrapper we want to use we want to use the transformer one we’re using GPU acceleration as we have in the past but now we’re doing a couple of different things to make this loop run faster first we’re reducing the batch size if you run into out of memory errors you might need to reduce the batch size further to run on your machine。
if you hit a batch size of one and you still can’t run it on your machine。
then your GPU is not big enough to run this model。
One trick that we use to try and be able to fit the model on smaller GPUus is this precision flag which sets the size of the floating point numbers。
the precision of the floating point numbers that we’re using typical in Python is 64 bit floating point numbers common in deep learning for years to use single。
Precision or 32 B floating point numbers that’s built into Pytorch from the very beginning with Pytorch lighting。
it’s really easy to reduce the precision and size of these floating point numbers even further down to 16 bit。
but you’ll notice that first line doesn’t say 16 bit precision but 16 bit automatic mixed precision。
So we do sometimes need higher precision single precision floating points rather than half precision floating points and luckily there’s nice tools built into pytorch lightingning and pytorch now for changing that just with a little flag rather than having to write code And then the other thing that we do is we limit the number of training and test and validation batches that we draw。
So instead of drawing all batches from the training set we only draw 10 we only test on one batch we only validate on two batches。
So this is to allow us to kind of play around with this model play around with some of these flags and get used to it before we launch these full long training jobs in future labs。
So as you can see we’ve hit the validation step that actually takes a pretty long time even though it’s just two。
😊，Examples and you can see the testing data load has started running during validation and testing。
we run the model the same way we would during inference in production。
We don’t use the teacher forward method。 We use the regular forward method。
So validation in testing run more slowly。 Once that’s done。
we get out some metrics that tell us how our model is doing a loss and a character error rate。
the character error rate is a kind of edit distance。
how many characters as a fraction of the number of characters in the ground true sequence。
how many characters we have to remove and add back in to get the correct value we want to character error rate down in the low 5 to 15% and we’re at almost 200%。
So a lot more training needed。😊，The exercise in section don’t direct you to actually try and train this model that’s going to take some pretty serious time。
Instead， we’re gonna start learning things about training models with pytorch lightning and using some of these pieces here。
Try out some of these training tricks built into the pytorch lightning trainer via command line flags and then also trying out this really important trick or workflow in Ml engineering overfitting a single batch So getting the model to predict perfectly the outputs for just one batch。
So we’ll talk a lot more about why we do overfitting of a single batch and how it fits into your overall Ml engineering workflows and your model development practice。
But to start， let’s just try and get it working。 There’s some suggestions here on how to go about bat。
how to get the model to start fitting a single batch within just a few minutes of training and then tune it so that you can end up getting a model that fits very well has a very low character error rate and low loss for an individual batch at the end。
😊，OfAbout 1000 forward passive。 So in future labs， we’ll start training models on this more realistic handwriting data sets and adding more instrumentation around our training so that it’s easier to iterate。
We can keep track of what’s happening。 We can visualize what’s happening。
and then we can hand off models that we’ve built through this training process to our production deployment process。
So we’ll see all of that in future labs。😊。
全栈深度学习：Lab 3：Transformers与段落识别 🧠
在本教程中，我们将学习Transformer架构，并将其应用于手写文本段落的识别任务。我们将介绍ResNet-Transformer模型，并了解其训练与推理过程的差异。
概述
本节将介绍Transformer架构的核心概念，以及为何它在序列建模任务中逐渐取代了循环神经网络。我们将重点讲解在文本识别系统中使用的ResNet-Transformer模型。
环境与代码准备
首先，我们需要准备好开发环境。你可以通过以下两种方式之一运行本实验：
-
在Google Colab上运行：访问GitHub仓库，点击底部的“Open in Colab”链接。
-
在本地机器上运行：确保已激活课程专用的Conda环境（
fsdl-text-rec-2022），并按照本地设置说明完成配置。然后，从仓库根目录导航至lab3/notebooks/，打开transformers.ipynb笔记本文件。
重要提示：请务必按顺序从上到下运行笔记本中的单元格，并首先运行初始的设置单元格。
Transformer架构简介
上一节我们准备好了环境，本节中我们来看看Transformer架构。Transformer是一种用于序列建模的神经网络架构，近年来因其强大的并行计算能力和优异的性能，在自然语言处理等领域广泛应用。
Transformer的核心优势在于其能够并行处理整个序列，这与需要顺序处理的循环神经网络（RNN）有本质区别。这种并行性使得Transformer在训练时效率更高。
ResNet-Transformer模型
了解了Transformer的基本思想后，我们来看看本实验将使用的具体模型——ResNet-Transformer。该模型结合了卷积神经网络（CNN）和Transformer的优点。
模型结构如下：
-
编码器：使用一个小型ResNet对输入图像进行编码，提取视觉特征。
-
解码器：使用一个Transformer解码器，将编码后的特征解码为字符序列。
以下是模型前向传播（推理时）的核心逻辑伪代码：
def forward(images):
encoded_inputs = resnet.encode(images) # 使用ResNet编码图像
outputs = []
for step in range(max_sequence_length):
# 将当前输出和编码输入送入Transformer解码器
step_output = transformer.decode(encoded_inputs, current_outputs)
outputs.append(step_output)
# 判断是否所有序列都已生成结束符
if all_sequences_finished:
break
return outputs
训练与推理的差异
Transformer模型在训练和推理时运行方式不同，这是其高效训练的关键。
在训练时，我们使用“教师强制”策略。这意味着在解码的每一步，我们都将真实的上一时刻标签（而非模型自己预测的上一时刻输出）作为解码器的输入。这可以防止错误在训练早期累积，并允许并行计算整个输出序列。
在推理（或验证、测试）时，我们没有真实标签。因此，模型必须使用自己在前一步预测的输出，作为下一步解码的输入，形成一个自回归的循环过程。
这种差异体现在代码中：
-
训练：调用
teacher_forward(inputs, ground_truth_labels)方法。 -
推理：调用常规的
forward(inputs)方法。
处理变长输出
我们选择Transformer架构的一个主要原因是它能够处理固定大小输入，变长输出的问题。
以下是查看数据批次中段落长度变化的示例代码思路：
# 从数据加载器中获取一个批次
batch = next(iter(data_loader))
images, labels = batch
# 图像尺寸是固定的（例如，H x W），但每个标签（段落）的字符长度各不相同
print(f"图像尺寸: {images.shape}")
print(f"各段落长度: {[len(label) for label in labels]}")
尽管所有输入图像被调整为相同尺寸，但其中的手写段落包含的字符数量（即输出序列长度）却各不相同。Transformer解码器能够根据编码后的输入，动态生成不同长度的序列，完美适应这一需求。
模型训练配置与技巧
由于ResNet-Transformer模型比之前介绍的模型更大、更复杂，我们需要使用GPU进行加速，并采用一些技巧来提升训练效率。
以下是启动训练时可能用到的一些关键配置（以PyTorch Lightning为例）：
trainer = pl.Trainer(
gpus=1, # 使用GPU加速
max_epochs=10,
precision=16, # 使用16位混合精度训练，节省显存并加速
limit_train_batches=10, # 仅使用10个批次进行训练（快速调试）
limit_val_batches=2, # 仅使用2个批次进行验证
limit_test_batches=1, # 仅使用1个批次进行测试
)
参数说明：
-
precision=16：启用自动混合精度训练。大部分计算使用16位浮点数以提升速度和减少显存占用，但某些操作会自动保持32位精度以保证数值稳定性。 -
limit_*_batches：限制每个周期使用的批次数量。这在模型调试和超参数快速尝试时非常有用，可以避免漫长的完整数据遍历。
如果遇到显存不足（OOM）错误，可以尝试进一步减小 batch_size。
过拟合单个批次
在正式开始大规模训练之前，一个重要的模型调试步骤是尝试过拟合单个批次。
这个练习的目的是：确保模型有足够的能力（容量）来学习训练数据。如果能快速过拟合一个很小的数据集（如单个批次），则说明模型的前向传播、反向传播和优化器配置基本正确，没有严重的bug。
以下是进行此练习的建议步骤：
-
将训练和验证数据都限制为仅一个批次。
-
使用较高的学习率。
-
训练几百到一千个步骤。
-
观察损失是否迅速下降，字符错误率是否趋近于0。
如果模型无法过拟合单个批次，则可能需要检查模型架构、数据预处理流程或损失函数是否正确。
总结
本节课中我们一起学习了：
-
Transformer架构的基本原理及其并行计算的优势。
-
ResNet-Transformer模型如何结合视觉编码和序列解码来完成文本识别任务。
-
模型在训练（教师强制）和推理（自回归）时的关键区别。
-
如何使用 PyTorch Lightning 的便捷配置（如混合精度、限制批次）来加速模型调试和训练。
-
过拟合单个批次作为模型开发工作流中重要的调试和验证步骤。
在接下来的实验中，我们将在此基础上，使用更完整的数据集进行长时间训练，并引入更多的工具来监控、可视化和优化训练过程，最终将训练好的模型部署到生产环境中。
Full Stack 深度学习 ｜ Full Stack Deep Learning 2022 p06 P06_Lecture_02-开发基础设施与工具 -BV1k4YXznEjw_p6-
Hi， everyone。 Welcome to week 2 of Full Sta deep Learning 2022。 Today。
we have a lecture on development infrastructure and tooling。 My name is Sergey。
and I have my assistant， Miishka right here。😊。
So just diving right in， the dream of machine learning development is that you provide a project spec。
identify birds， maybe some sample data， here’s what the birds look like， here’s what I want to see。
And then you get a continually improving prediction system， and it’s deployed at scale。
But the reality is that it’s not just some sample data。
you really have to find the data aggregate it， process it， clean it， label it。
then you have to find the model architecture， potentially the pre-trained weights。
then you still have to look at the model code and probably edit it， debug it， run。
training experiments， review the results that’s going to feed back into maybe trying a new architecture。
debugging some more code， and then when that’s done， you can actually deploy the model。
and then after you deploy it， you have to monitor the predictions and then you close the data flywheel loop。
basically your user is generating fresh data for you that you then have to add your data。
So this reality has roughly kind of three components。 and we divided into data in red。
There’s development in yellow and deployment in green。
And there are a lot of tools like the infrastructurefraural landscape is pretty large。
So we have three lectures to cover all of it。 and today we’re going to concentrate on the development part。
the middle part， which is probably what you’re familiar with from previous courses。
most of what you do is model development。We actually want to start even a little bit before that and talk about software engineering。
You know， it starts with maybe the programming language。And for machine learning， it’s pretty clear。
It has to be Python。 And the reason is because of all the libraries that have been developed for it。
It’s just the winner in scientific and data computing。There have been some contenders。
so Julia is actually the JU in Jupiter Jupiter notebooks to write Python code， you need an editor。
You can be old school and use Vim or Ems， a lot of people just write in Jupiter notebooks or Jupiter Lab。
which also gives you a code editor window VS code is a very popular text editor。
Python specific code editor Py Charm is really good as well。At FSD， we recommend V S code。
It has a lot of nice stuff。It hast built， you know， in addition to the nice editing features。
it has built in git version control so you can see your commit， you can actually stage line by line。
you can look at documentation as you write your code。
You can open projects remotely so like the window I’m showing here is actually on a remote machine that I have SS into。
you can Li code as you write， and if you haven’t seen Ls before。
it’s basically this idea that if there are code style rules that you want to follow like certain number of spaces for indentation。
whatever you decide you want to do， gotta you should just codify it so that you don’t ever have to think about it or manually put that in。
your tools just do it for you。And if you’ve run something that just looks at your code all the time。
you can do a little bit of static analysis。So， for example。
there’s two commas in a row It’s not going to run in this file。
Or potentially you’re using a variable that never got defined。And in addition。
Python now has type hints， so you can actually say。
you know this variable is supposed to be an integer。
and then if you use it as an argument to a function that expects a float。
a static type checker can catch and tell you about it before you actually run it。
So we set that all up in the lab， by the way， and you will see how that works。
It’s a very nice part of the lab。A lot of people develop in Jupyter notebooks。
And they’re really fundamental to data science。 And I think for good reason。
I think it’s a great kind of first draft of a project。
You just open up this notebook and you start coding。
There’s very little thought that you have to put in before you start coding and start seeing immediate output。
So that kind of like。Fast feedback cycle。That’s really great。
And Jeremy Howard is a great practitioner， so if you watch the fast AI course videos。
you’ll see him use them to their full extent。They do have problems though， for example。
the editor that you use in the notebook is pretty primitive right there’s no refactoruring support。
there’s no maybe peeking of the documentation， there’s no coppilot which I have now got used to in VS code。
There’s out of order execution artifacts， so if you’ve run the cells in a different order。
you might not get the same result as if you ran them all in line。It’s hard to version them。
You either strip out the output of each cell， in which case。
You lose some of the benefit because sometimes you want to save the artifact that you produced in the notebook。
Or the file is pretty large and keeps changing。And it’s hard to test， because。
It’s just not very amenable to the unit testing frameworks and best practices that people have built up。
Counterpoint to everything I just said is that you can kind of fix all of that。
And that’s what Jeremy Howard’s trying to do with Nb dev。
which is this package that lets you write documentation。
your code and tests for the code all in a notebook。
The full slide deep learning recommendation is go ahead and use notebooks。
actually use the VS code built in notebook support， So I actually don’t I’m not in the browser ever。
I’m just in my VS code but I’m coding in a notebook style。
But also I usually write code in a module that then gets imported into a notebook and with this live reload extension。
it’s quite nice because when you change code in the module and rerun the notebook that it gets the updated code。
And also you have nice things like you have a terminal， you can look at files and so on。
And by the way， it enables really awesome debugging。 So if you want to debug some code。
you can put a break point here on the right。 you see the little red dot。
and then I’m about to launch the cell with the debug cell command。
and that’ll drop me in into the debugger at that break point。
And so this is just really nice without leaving the editor。 I’m able to to do a lot。😊。
No books are great， sometimes you want something a little more interactive。
maybe some you can share with the world and streamreamlet has come along and let you just decorate Python codes。
you write a Python script， you decorate it with widgets and data loaders and stuff。
And you can get interactive applets where people can。
Let’s say a variable can be controlled by a slider and everything just gets reverun very efficiently and then when you’re happy with your applet。
you can publish it to the web and just share that streamlit address with your audience。
It’s really quite great。😊，For setting up the Python environment。It can actually be pretty tricky。
so for deep learning usually you have a GPU and the GPU needs Kuda libraries。
And Python has a version， and then。Each of the requirements that you use like pi torch or nupy have their own specific version。
also， some requirements are for production， like torch， but somewhere are only for development。
For example， Black is a code styling tool or my Pi is a static analysis tool。
It be nice to just separate the two。So we can achieve all these desired things by specifying Python and kuda versions in environment that Yaml file and use Conda to install the Python and the kuda version that we specified。
but then all the other requirements we specify in with basically just very minimal constraints。
so we say like torch version greater than 1。7， or maybe no constraints like nuy any version。
And then we use this tool called PIP tools that will analyze the constraints we gave and the constraints they might have for each other and find a mutually compatible version of all the requirements。
And then locks it so that when you come back to the project。
you have exactly the versions of everything you used。
And we can also just use a make file to simplify this。 Now， we do this in lab。
So you’ll see this in lab。And on that note， please go through labs one through three。
they’re already out and。Starts with an overview of what the labs are going to be about。
then pytorch lightning and Pytorch。 And then we go through CNNs， Transers。
And we see a lot of this structure that I’ve been talking about。
So that is it for software engineering。And the next thing I want to talk about are specifically deep learning frameworks and distributed training。
So why do we need frameworks。Well， deep learning is actually not a lot of code if you have a matrix math library like Nmpyine。
Now， fast that AI course does this pretty brilliantly。
they basically have you build your own deep learning library and you see how very little code it is。
But when you have to deploy stuff onto KUuda for GPU power deep learning。
and when you have to consider that you might be writing weird layers that have to。
you have to figure out the differentiation of the layers that you write。
That can get to be just a lot to maintain And so and then also there’s all the layer types that have been published in the literature。
like like convolutional layers， there’s all the different optimizers。 So there’s just a lot of code。
And for that you really need a framework So which framework should you use right Well。
I think Josh answered this， you know， pretty concisely about a year ago and you said ja is for researchers Pytorrchches for engineers and Tensorflows for boomers。
😊，So Pytorch is the full stack deep learning choice， but seriously， though。You know。
both Pytorch and Tensorflowlow and Jax。They all are similar。
you define a deep learning model by running Python code。Writing and running Python code。
And then what you get is an optimized execution graph that can target CPUUs， GPUs， TUs。
mobile deployments。Now， the reason you might prefer pytororch is because it just basically is absolutely dominant right so if you look at the number of models。
trained models that are shared on Huging face， which is like the largest model zoo。
we’ll talk about it in a few minutes。You know， there’s models that are both Pytorch and Tensorflow。
There’s some models on jacks。 There’s some models that are Tensorflow only。
There’s a lot of models that are just for pytorch。
If you track paper submissions to academic conferences。
it’s about 75 plus percent Pytorch implementations of these research papers。
and my face is blocking the stat， but it’s something like 75% of machine learning competition winners use PyTch in 2022。
Now Tensorflowlow is kind of cool。 Tensorflow that JS in particular lets you run deep learning models in your browser。
and Pythtor doesn’t have that。 And then Karis as a development experience， is I think。
pretty unmatched for just stacking together layers， easily training the model。😊。
And mean then there’s Jacks， which you might have heard about。 So Jacks， you know。
the main thing is you need a meta framework for deep learning。 We’ll talk about in a second。
But Pytorch， that’s the pick， excellent dev experience。Its people used to say， well。
maybe it’s a little slow， but it really is production ready， even as is。
but you can make it even faster by compiling your model with torch script。
There’s a great distributor training ecosystem， there’s libraries provision， audio， 3D data。
you know， et cetera， there’s mobile deployment targets。And with Pythtororch lightning。
which is what we use in labs。Have a nice structure for how to kind of where do you put your actual model code。
where do you put your optimizer code， where do you put your training code， your evaluation code。
How should the data loaders look like？And then what you get is if you just kind of structure your code as Pythorch Lightning expects it。
you can run your code on CPUU or GPU or any number of GPUs or TUs with just you know。
a few characters change in your code， there’s a performance profiler， there’s model checkpointing。
there’s 16 bit precision， there’s distributor training libraries， it’s just all very nice to use。
Now， another possibility is fast AI software， which is developed alongside the fast AI core。
And it provides a lot of advanced tricks like data augmentations， better weight initializations。
learning rates schedulechedries。It has this kind of modular structure where there’s data blocks and learners and then even vision text tabular applications。
The main problem with that I see is the code style is quite different。 and in general。It’s。
It’s a little bit different than mainstream Pytorch。
it can be very powerful if you go in on it at FSDL， we recommend PyTtorch lightning。
TensorF is not just for boomers， right， FSDL prefers Pytorch because we think it’s a stronger ecosystem。
But Tensorflow is still perfectly good。 And if you have a specific reason to prefer it。
such as that’s what your employer uses， You’re gonna have a good time。 It still makes sense。
It’s not bad。 Jackx is a recent， a more recent project from Google。😊。
Which is really not specifically deep learning， it’s about just general vectorization of all kinds of code and also auto differentiation of all kinds of code。
Including physics simulations， stuff like that。And then whatever you can express in JaX gets compiled to GPU or TU code and super fast。
For deep learning， there are separate frameworks like Fl or Haiku。And， you know， here at FSDL。
we say， use it if you have a specific need， Maybe you’re doing research on something kind of weird。
That’s fine。 Or， you know， potentially， you’re working at Google。 you’re not allowed to use Ptorch。
That could make it a pretty good reason to use Js。
There’s also this notion of meta frameworks and model zoos that I want to cover。
So model zoos is the idea that。
Sure， you can just start with blank pytorage。 But most of the time you’re going to start with at least a model architecture that someone’s developed and published。
And a lot of the time， you’re going to start with actually a pre trained model。
meaning someone trained the architecture。On specific data。
they got weights that they then saved and uploaded to a hub and you can download and actually start not from scratch。
but from a pretrain model。 Onyx is this idea that deep learning models are all about the same right Like we know what an MLP type of layer is。
We know what a CNN type of layer is。 and it doesn’t matter if it’s written in pytorage or Tensorflow or cafe。
whatever it’s written and we should be able to actually ported between the different code bases。
because the real thing that that we care about are the weights and the weights are just numbers。
right。So Onyx is this format that lets you convert from Pythtorch to Tensorflowlow and vice versa。
And it can work super well。 It can also not work super well。 You can run into some edge cases。
So if it’s something that you need to do， then definitely worth a try。
But it’s not necessarily going to work for all types of models。Hugging face has become an absolutely。
Sttellar repository of models， starting with NLP， but have since expanded to all kinds of tasks。
audio classification， image classification， object detection， there’s 60。
000 pretrain models for all these tasks。There is a specific library of transformers that works with Pytorrch。
Tensorflow， jacks。Alsol is 7。5，000 data sets that people have uploaded。
There’s also a lot more to it。 It’s worth checking out。 You can host your model for inference。
and there’s， there’s community aspects to it。So it’s a great resource。
Another great resource specifically for Vi is it’s called Tim。
a state of the art computer vision models can be found on Tim to search Tim GitHub。Next up。
let’s talk about distributed training。So the scenarios are we have multiple machines represented by little squares here with multiple GPUs on each machine。
And you are sending batches of data。To be processed by a model that has parameters right。
and the data batch can fit on a single GPU or potentially not fit on a single GPU。
and the model parameters can fit on a single GPU or potentially not fit in a single GPU。
So let’s say the best case， the easiest case is your batch of data fits on a single GPU。
your model parameters fit on a single GPU， and that’s really called trivial parallelism。
You can launch independent experiments on other GPUs， so maybe do a hyperparameter search。
Or potentially， you increase your batch size until it can no longer fit on one GPU。
And then you have to figure something else out。And what you have to then figure out is， okay， well。
my model still fits on a single GPU， but my data no longer fits on a single GPU。
so now I have to go and do something different。And what that different thing is。
usually is data parallelism。It lets you distribute a single batch of data across GPUus。
And then average gradients that are computed by the model across all the GPUs。
So it’s the same model on each GPU， but different batches of data because a lot of this work is across GPU。
we have to make sure that the GPUs have fast interconnect so GPU is connected usually through a PCI interface to the computer。
and so if there’s no other connection， then all the data has to flow through the PCI bus all the time。
It’s possible that there is a faster interconnect like NV link between the GPUs。
and then the data can leave the PCI bus alone and just go straight across the fast interconnect。
And the speedup you can expect is if you are using server cards like A 100s， A600s， V100s。
It’s basically a linear speed up for data parallelism， which is really cool。
If you’re using consumer cards like 2080s or 3080s， we’ll talk about it a little further down。
Then unfortunately， it’s going to be a sub linear speed up。 So maybe if you have four GPUs。
it’ll be like a 3 x speed up。 if you have 8 GPs， maybe a 5 x speed up。
And that’s due to the fact that the consumer cards don’t have as fast as an interconnect。
So data parallelism is implemented in Pytorch in the distributed data parallel library。
There’s also a thirdpart library called Hraoid and you can use either one super simply using Pytororch Lightning you basically say what’s your strategy if you don’t say anything then it’s single GPU。
but if your strategy is GDPP， then it uses the PyTtorch distributed data parallel。
if you use a strategy Ho， then it uses Hood。It seems like the speedups basically about the same。
there’s no real reason to use H over distributed data parallel。
But it might make it easier for a specific case that you might have。 So it’s good to know about。
But the first thing to try is just distributed data parallel。Now we come to a more advanced area。
which is now we can’t even fit our model。 Our model is so large。 It has billions of parameters。
It doesn’t actually fit on a single GPU。 So we have to spread the model。
Not just the data over multiple GPUs。And there’s three solutions to this。
So sharded data parallelism starts with the question。What exactly。Is in the GPU memory。
What has taken up the GPU memory。So okay， we have the model parameters。
The floats that make up are actual layers。We have the gradients， we need to know about the gradients。
because that’s what we average to do our back。But we also have optimizer states and that’s actually a lot of data for the atom optimizer that’s probably the most often used optimizer today。
it has to be statistics about the gradients basically and in addition。
if you’re doing kind float 16 training， then your model parameters and gradients might be float 16 but the optimizer will keep a copy of them as F 32 as well。
so it can be a lot more data。And then， plus， of course， you send your batch of data。
So all of this has to fit on a GPU。But does it actually have to fit on every GPU is the question。
So the baseline that we have is， yeah let’s send all of this stuff to each GPU。
and that might take up like 129 gigabytes of data in this in this example。
this is from the paper called zeroro optimizations towards training trillion parameter models。Okay。
so what if we shard the optimizer states， sharding is a。
Concept is from databases where if you have one source of data。
you actually break it up into shards of data such that across your distributed system。
part of your each node only sees a shard， a single shard of the data。So here。
the first thing we can try is we can shard the optimizer states。 Each GPU doesn’t have to have。
All the optimizer state， it just has to have its little shard of it。
And we can do the same for gradients。And that’s called zero2。And then。Pretty crazily。
we can also do it for the model parameters themselves。 And that’s called 0 of3。
DAMO开发者矩阵，由阿里巴巴达摩院和中国互联网协会联合发起，致力于探讨最前沿的技术趋势与应用成果，搭建高质量的交流与分享平台，推动技术创新与产业应用链接，围绕“人工智能与新型计算”构建开放共享的开发者生态。
更多推荐
0
0- 0
已为社区贡献266条内容
所有评论(0)