2-17. Model Testing and Evaluation

So let’s go back to our two examples: a medical model and spam e-mail model. Recall that for the medical model, we have four possibilities: a true positive which is when the patient is sick and we correctly diagnosed him as sick; a true negative which is when the patient is not sick and we … Read more

Now, accuracy may not always be the best metric to use. Let’s look at this example. It consists of detecting credit card fraud. So, we have a bunch of data in the form of credit card transactions, and some of them are good like the ones in the left, and some of them are fraudulent … Read more

Well, let’s remember that accuracy is a ratio between the correctly classified points and the total number of points. We notice that there are six points correctly classified as positive and five correctly classified as negative. These add to 11 correctly classified points. Since the total number of points is 14, we conclude that the … Read more

Now, we’ll study one of the ways to measure how good a model is, called that the accuracy. The accuracy is the answer to the question, “Out of all the patients, how many did we classify correctly?” The answer is, the ratio between the number of correctly classified points and the number of total points. … Read more

So let’s see. The true positives are the points that are positive and the model correctly labels as positive. These are the six blue points over here. The true negatives are the points that are negative and the model correctly labels as negative. These are the five negative red points over here. The false positives … Read more

So after we develop a model, we want to find out how good it is. This is a difficult question. But in this section, we’ll learn a few different metrics that will tell us how good our model is. So we’re going to look at two main examples. The first example is a model that … Read more

Here is the outline for the section. In machine learning we have a problem to solve. Normally, the problem corresponds to evaluating some data and making predictions. That problem is represented by this broken car over here. Now, in order to solve it, we have a few tools. These tools are the algorithms. So things … Read more

In this section we’ll learn a way to tell overfitting, underfitting, and a good model. So here we have our data three times. Our data seems to be well split by a quadratic equation of degree two. So we’re going to try and fit three models. The first one a linear or degree one model … Read more

As the last thing, we’ll learn a very useful method to recycle our data called, K-Fold Cross Validation. As we have learned, testing is done by separating our data into a training and a testing set, but this is not always ideal as we seem to be throwing away some data that could be useful … Read more

Now, let’s look at the second metric, which is recall. Recall is the answer to the following question, out of the points that are labeled positive, how many of them were correctly predicted as positive? So in the medical model, a recall is the answer to the following question, out of the sick patients, how … Read more

So let’s define the precision metric as follows. Here’s the confusion matrix of the medical model, and we’ve added a red X in the spot that we really want to avoid, which is the false negatives. So precision will be the answer to the question, out of all the points predicted to be positive, how … Read more

So for the medical model, it seems that we’d much rather misdiagnose a healthy person as sick and send them for more tests than misdiagnose a sick person as healthy and send them home without treatment. In this case, a false positive is much worse than a false negative. So the rule of this model … Read more

Hi, and welcome to the small evaluation and validation section. My name is Luis Serrano and I’m an instructor here at Udacity. In this section, we’ll focus on two questions. The first question is, how well is my model doing? So, let’s say we’ve trained our machine learning model and the question is, is this … Read more