How Models Work
ML model: A mathematical representation of a real-world process that has been trained to make predictions or decisions based on data.
Example: If you train a model with photos of cats and dogs, it learns features (like ears, tails, fur texture) that distinguish the two. Later, when you show it a new photo, it can predict whether it’s a cat or dog.
Let’s start with a decision tree model which is a simple ML model that’s also a building block for some of the best models in data science
Decision Tree
A decision tree is a popular machine learning algorithm used for both classification and regression tasks. It models decisions as a tree-like structure of questions (nodes) that split data based on certain features.
Example: Suppose you’re predicting if someone will play tennis
Basic Data Exploration
The first step to any machine learning project is to understand the data. And you can do this by using the Exploratory Data Analysis approach
import pandas as pd
# save filepath to variable for easier access
melbourne_file_path = '../input/melbourne-housing-snapshot/melb_data.csv'
# read the data and store data in DataFrame titled melbourne_data
melbourne_data = pd.read_csv(melbourne_file_path)
# print a summary of the data in Melbourne data
melbourne_data.describe()
Dataframe: Holds data in a table like structure
Interpreting Data Description
The results show 8 numbers for each column in your original dataset. The first number, the count, shows how many rows have non-missing values.
Missing values arise for many reasons. For example, the size of the 2nd bedroom wouldn’t be collected when surveying a 1 bedroom house. We’ll come back to the topic of missing data.
The second value is the mean, which is the average. Under that, std is the standard deviation, which measures how numerically spread out the values are.
To interpret the min, 25%, 50%, 75% and max values, imagine sorting each column from lowest to highest value. The first (smallest) value is the min. If you go a quarter way through the list, you’ll find a number that is bigger than 25% of the values and smaller than 75% of the values. That is the 25% value (pronounced “25th percentile”). The 50th and 75th percentiles are defined analogously, and the max is the largest number.
Your First Machine Learning Model
Most datasets would have numerous variables with a mix of both useful and not for your model. How can you pare down this overwhelming amount of data to something you can understand? Start by picking a few variables using your own intuition
To choose variables/columns, we’ll need to see a list of all columns in the dataset. That is done with the columns property of the DataFrame
import pandas as pd
melbourne_file_path = '../input/melbourne-housing-snapshot/melb_data.csv'
melbourne_data = pd.read_csv(melbourne_file_path)
melbourne_data.columns
# The Melbourne data has some missing values (some houses for which some variables weren't recorded.)
# We'll learn to handle missing values in a later tutorial.
# Your Iowa data doesn't have missing values in the columns you use.
# So we will take the simplest option for now, and drop houses from our data.
# Don't worry about this much for now, though the code is:
# dropna drops missing values (think of na as "not available")
melbourne_data = melbourne_data.dropna(axis=0)
# Selecting the prediction target
# This will pull out the values in the specified variable "Price"
y = melbourne_data.PriceFeatures
The columns that are inputted into our model (and later used to make predictions) are called “features.”
melbourne_features = ['Rooms', 'Bathroom', 'Landsize', 'Lattitude', 'Longtitude']
# By convention this data is called "X"
X = melbourne_data[melbourne_features]Building the model
Library used: scikit-learn aka sklearn
Steps to building and using a ml model:
- Define: What type of model will it be? A decision tree or something else?
- Fit: Capture patterns from provided data
- Predict: Predict
- Evaluate: Determine how accurate the predictions compare to the actual solution
from sklearn.tree import DecisionTreeRegressor
# Define model. Specify a number for random_state to ensure same results each run
# Many machine learning models allow some randomness in model training. Specifying a number for `random_state` ensures you get the same results in each run. This is considered a good practice. You use any number, and model quality won't depend meaningfully on exactly what value you choose.
melbourne_model = DecisionTreeRegressor(random_state=1)
# Fit model
melbourne_model.fit(X, y)
print("Making predictions for the following 5 houses:")
print(X.head())
print("The predictions are")
print(melbourne_model.predict(X.head()))Result
Making predictions for the following 5 houses:
Rooms Bathroom Landsize Lattitude Longtitude
1 2 1.0 156.0 -37.8079 144.9934
2 3 2.0 134.0 -37.8093 144.9944
4 4 1.0 120.0 -37.8072 144.9941
6 3 2.0 245.0 -37.8024 144.9993
7 2 1.0 256.0 -37.8060 144.9954
The predictions are
[1035000. 1465000. 1600000. 1876000. 1636000.]Model Validation
Now that you’ve learned to create a model, you need to measure how good it measures up to its purpose.
First, you need to summarize the model quality into an understandable way. If you compare predicted and actual home values for 10,000 houses, you’ll likely find mix of good and bad predictions. Looking through a list of 10,000 predicted and actual values would be pointless. We need to summarize this into a single metric.
In comes Mean Absolute Error (MAE): The prediction error is error = actual - predicted then take the absolute value of it to guarantee a positive result
- So, if a house cost $150,000 and you predicted it would cost $100,000 the error is $50,000.
from sklearn.metrics import mean_absolute_error
predicted_home_prices = melbourne_model.predict(X)
mean_absolute_error(y, predicted_home_prices)
# Result: 434.71594577146544
# The estimated error ranges around $434 off from the actual home priceProblem: In-sample scores
- The code above measured is considered an in-sample score since we used a single sample of houses for both building and the model evaluating it.
- This is bad, because the model can find outlying patterns that are unrelated to calculating the actual prediction
Imagine that, in the large real estate market, door color is unrelated to home price.
However, in the sample of data you used to build the model, all homes with green doors were very expensive. The model’s job is to find patterns that predict home prices, so it will see this pattern, and it will always predict high prices for homes with green doors.
Since this pattern was derived from the training data, the model will appear accurate in the training data.
But if this pattern doesn’t hold when the model sees new data, the model would be very inaccurate when used in practice.
Since models’ practical value come from making predictions on new data, we measure performance on data that wasn’t used to build the model. The most straightforward way to do this is to exclude some data from the model-building process, and then use those to test the model’s accuracy on data it hasn’t seen before. This data is called validation data.
Coding it
Use train_test_split to break up the data into two pieces.
from sklearn.model_selection import train_test_split
# split data into training and validation data, for both features and target
# The split is based on a random number generator. Supplying a numeric value to
# the random_state argument guarantees we get the same split every time we
# run this script.
train_X, val_X, train_y, val_y = train_test_split(X, y, random_state = 0)
# Define model
melbourne_model = DecisionTreeRegressor()
# Fit model
melbourne_model.fit(train_X, train_y)
# get predicted prices on validation data
val_predictions = melbourne_model.predict(val_X)
print(mean_absolute_error(val_y, val_predictions))
# Result: 265806.91478373145
# WOW the out-of-sample data has an error of over 250k dollarsThis is the difference between a model that is almost exactly right, and one that is unusable for most practical purposes.
There are many ways to improve this model, such as experimenting to find better features or different model types.
Underfitting and Overfitting
Fine-tune your model for better performance
Underfitting and overfitting are two common problems that can hinder a model’s ability to perform well, especially on new, unseen data.
Analogy: Underfitting is like a student who doesn’t study enough and performs poorly on both practice tests (training data) and the real test (new data). While, overfitting is like a student who studies so much that they only memorize the patterns from the practice tests, so they struggle to apply that to new problems in the real test
- Underfitting: The model is too simple and doesn’t capture the underlying patterns.
- Overfitting: The model is too complex and memorizes the training data, including noise
So in our model, we can try to control the Decision tree depth and find the sweet spot that lies between under and overfitting issues.
from sklearn.metrics import mean_absolute_error
from sklearn.tree import DecisionTreeRegressor
def get_mae(max_leaf_nodes, train_X, val_X, train_y, val_y):
# Get MAE based on the max_leaf_nodes argument
model = DecisionTreeRegressor(max_leaf_nodes=max_leaf_nodes, random_state=0)
model.fit(train_X, train_y)
preds_val = model.predict(val_X)
mae = mean_absolute_error(val_y, preds_val)
return(mae)
# compare MAE with differing values of max_leaf_nodes
for max_leaf_nodes in [5, 50, 500, 5000]:
my_mae = get_mae(max_leaf_nodes, train_X, val_X, train_y, val_y)
print("Max leaf nodes: %d \t\t Mean Absolute Error: %d" %(max_leaf_nodes, my_mae))
"""
Max leaf nodes: 5 Mean Absolute Error: 347380
Max leaf nodes: 50 Mean Absolute Error: 258171
Max leaf nodes: 500 Mean Absolute Error: 243495
Max leaf nodes: 5000 Mean Absolute Error: 254983
"""
# Based on this result, it looks like 500 is the optimal number of leavesRandom Forests
Decisions trees have a problem. A deep tree with lots of leaves will overfit because each prediction is coming from historical data from only the few houses at its leaf. But a shallow tree with few leaves will perform poorly because it fails to capture as many distinctions in the raw data.
Introducing random forests, which uses multiple trees and makes a prediction by averaging the predictions of each component tree. It generally has better predictive accuracy than a single decision tree and works well with default parameters.
If you keep modeling, you can learn more models with even better performance, but they may need very specific parameters
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_absolute_error
forest_model = RandomForestRegressor(random_state=1)
forest_model.fit(train_X, train_y)
melb_preds = forest_model.predict(val_X)
print(mean_absolute_error(val_y, melb_preds))
# Result: 191669.7536453626Random forest decision tree still has room with improvement, but it’s a big improvement over the error of 250k+. There are parameters that allow you to change the performance of the Random Forest