How effective are predictions?

You may have come across some nerdy technical explanations of prediction effectiveness, such as mean squared error, accuracy, precision and recall. We won’t get into those in detail. I want to review a couple of powerful concepts that are important to understand, namely false positives and false negatives.

Some predictions will be right, and some predictions will be wrong. There are two ways you can be right: Predict no when it is actually a no, and predict yes when it is actually a yes.  For example:

  • we predicted the customer would not defect,  and they actually did not defect.
  • we predicted the patient would get the flu, and they did get the flu.
  •  There are two ways to be wrong. Predict yes, when it is actually a no, and predict no, when it is actually a yes. These are known as false positives and false negatives respectively. Some examples:
    • we predicted the customer will not defect to Beta, but they do –> False Negative
    • we predict a patient will get the flu and they do get the flu –> False Positive.

    The influenza example is intuitive to understand. Every year there are people who don’t get a flu shot who needed it because they got sick (false negatives), and people who got a flu shot who didn’t need it.

    Which is worse: Being wrong with False Positive or False Negative

    For certain models, effectiveness needs to consider which has a worse impact when your prediction is wrong. Let’s start with the flu example.

    A strong case could be made that being wrong with false negatives is a more severe outcome than being wrong with false positives. In other words, not giving people a flu shot who need it, is more severe than giving people a flu shot who did not need it.

    In the customer churn example, let’s look at the risks:

    It could be argued that being wrong by not offering a loyalty offer and losing the customer is worse than giving a loyalty offer to a customer who did not need it but keeping the customer.

    It is important to understand these risk/reward tradeoffs when looking at model effectiveness described in the false positive and true positive rates.