Tech Tutorials

Introduction

Last week, we learned about Random Forest Algorithm. Now we know it helps us reduce a model's variance by building models on resampled data and thereby increases its generalization capability. Good!

Now, you might be wondering, what to do next for increasing a model's prediction accuracy ? After all, an ideal model is one which is good at both generalization and prediction accuracy. This brings us to Boosting Algorithms.

Developed in 1989, the family of boosting algorithms has been improved over the years. In this article, we'll learn about XGBoost algorithm.

XGBoost is the most popular machine learning algorithm these days. Regardless of the data type (regression or classification), it is well known to provide better solutions than other ML algorithms. In fact, since its inception (early 2014), it has become the "true love" of kaggle users to deal with structured data. So, if you are planning to compete on Kaggle, xgboost is one algorithm you need to master.

In this article, you'll learn about core concepts of the XGBoost algorithm. In addition, we'll look into its practical side, i.e., improving the xgboost model using parameter tuning in R.

On 5th March 2017: How to win Machine Learning Competitions ?

Table of Contents

1. What is XGBoost? Why is it so good?
2. How does XGBoost work?
3. Understanding XGBoost Tuning Parameters
4. Practical - Tuning XGBoost using R

What is XGBoost ? Why is it so good ?

XGBoost (Extreme Gradient Boosting) is an optimized distributed gradient boosting library. Yes, it uses gradient boosting (GBM) framework at core. Yet, does better than GBM framework alone. XGBoost was created by Tianqi Chen, PhD Student, University of Washington. It is used for supervised ML problems. Let's look at what makes it so good:

1. Parallel Computing: It is enabled with parallel processing (using OpenMP); i.e., when you run xgboost, by default, it would use all the cores of your laptop/machine.
2. Regularization: I believe this is the biggest advantage of xgboost. GBM has no provision for regularization. Regularization is a technique used to avoid overfitting in linear and tree-based models.
3. Enabled Cross Validation: In R, we usually use external packages such as caret and mlr to obtain CV results. But, xgboost is enabled with internal CV function (we'll see below).
4. Missing Values: XGBoost is designed to handle missing values internally. The missing values are treated in such a manner that if there exists any trend in missing values, it is captured by the model.
5. Flexibility: In addition to regression, classification, and ranking problems, it supports user-defined objective functions also. An objective function is used to measure the performance of the model given a certain set of parameters. Furthermore, it supports user defined evaluation metrics as well.
6. Availability: Currently, it is available for programming languages such as R, Python, Java, Julia, and Scala.
7. Save and Reload: XGBoost gives us a feature to save our data matrix and model and reload it later. Suppose, we have a large data set, we can simply save the model and use it in future instead of wasting time redoing the computation.
8. Tree Pruning: Unlike GBM, where tree pruning stops once a negative loss is encountered, XGBoost grows the tree upto max_depth and then prune backward until the improvement in loss function is below a threshold.

I'm sure now you are excited to master this algorithm. But remember, with great power comes great difficulties too. You might learn to use this algorithm in a few minutes, but optimizing it is a challenge. Don't worry, we shall look into it in following sections.

How does XGBoost work ?

XGBoost belongs to a family of boosting algorithms that convert weak learners into strong learners. A weak learner is one which is slightly better than random guessing. Let's understand boosting first (in general).

Boosting is a sequential process; i.e., trees are grown using the information from a previously grown tree one after the other. This process slowly learns from data and tries to improve its prediction in subsequent iterations. Let's look at a classic classification example:

Four classifiers (in 4 boxes), shown above, are trying hard to classify + and - classes as homogeneously as possible. Let's understand this picture well.

1. Box 1: The first classifier creates a vertical line (split) at D1. It says anything to the left of D1 is + and anything to the right of D1 is -. However, this classifier misclassifies three + points.
2. Box 2: The next classifier says don't worry I will correct your mistakes. Therefore, it gives more weight to the three + misclassified points (see bigger size of +) and creates a vertical line at D2. Again it says, anything to right of D2 is - and left is +. Still, it makes mistakes by incorrectly classifying three - points.
3. Box 3: The next classifier continues to bestow support. Again, it gives more weight to the three - misclassified points and creates a horizontal line at D3. Still, this classifier fails to classify the points (in circle) correctly.
4. Remember that each of these classifiers has a misclassification error associated with them.
5. Boxes 1,2, and 3 are weak classifiers. These classifiers will now be used to create a strong classifier Box 4.
6. Box 4: It is a weighted combination of the weak classifiers. As you can see, it does good job at classifying all the points correctly.

That's the basic idea behind boosting algorithms. The very next model capitalizes on the misclassification/error of previous model and tries to reduce it. Now, let's come to XGBoost.

As we know, XGBoost can used to solve both regression and classification problems. It is enabled with separate methods to solve respective problems. Let's see:

Classification Problems: To solve such problems, it uses booster = gbtree parameter; i.e., a tree is grown one after other and attempts to reduce misclassification rate in subsequent iterations. In this, the next tree is built by giving a higher weight to misclassified points by the previous tree (as explained above).

Regression Problems: To solve such problems, we have two methods: booster = gbtree and booster = gblinear. You already know gbtree. In gblinear, it builds generalized linear model and optimizes it using regularization (L1,L2) and gradient descent. In this, the subsequent models are built on residuals (actual - predicted) generated by previous iterations. Are you wondering what is gradient descent? Understanding gradient descent requires math, however, let me try and explain it in simple words:

• Gradient Descent: It is a method which comprises a vector of weights (or coefficients) where we calculate their partial derivative with respective to zero. The motive behind calculating their partial derivative is to find the local minima of the loss function (RSS), which is convex in nature. In simple words, gradient descent tries to optimize the loss function by tuning different values of coefficients to minimize the error.

Hopefully, up till now, you have developed a basic intuition around how boosting and xgboost works. Let's proceed to understand its parameters. After all, using xgboost without parameter tuning is like driving a car without changing its gears; you can never up your speed.

Note: In R, xgboost package uses a matrix of input data instead of a data frame.

Understanding XGBoost Tuning Parameters

Every parameter has a significant role to play in the model's performance. Before hypertuning, let's first understand about these parameters and their importance. In this article, I've only explained the most frequently used and tunable parameters. To look at all the parameters, you can refer to its official documentation.

• General Parameters: Controls the booster type in the model which eventually drives overall functioning
• Booster Parameters: Controls the performance of the selected booster
• Learning Task Parameters: Sets and evaluates the learning process of the booster from the given data

General Parameters

1. Booster[default=gbtree]
   • Sets the booster type (gbtree, gblinear or dart) to use. For classification problems, you can use gbtree, dart. For regression, you can use any.
2. nthread[default=maximum cores available]
   • Activates parallel computation. Generally, people don't change it as using maximum cores leads to the fastest computation.
3. silent[default=0]
   • If you set it to 1, your R console will get flooded with running messages. Better not to change it.

Booster Parameters

As mentioned above, parameters for tree and linear boosters are different. Let's understand each one of them:

Parameters for Tree Booster

1. nrounds[default=100]
   • It controls the maximum number of iterations. For classification, it is similar to the number of trees to grow.
   • Should be tuned using CV
2. eta[default=0.3][range: (0,1)]
   • It controls the learning rate, i.e., the rate at which our model learns patterns in data. After every round, it shrinks the feature weights to reach the best optimum.
   • Lower eta leads to slower computation. It must be supported by increase in nrounds.
   • Typically, it lies between 0.01 - 0.3
3. gamma[default=0][range: (0,Inf)]
   • It controls regularization (or prevents overfitting). The optimal value of gamma depends on the data set and other parameter values.
   • Higher the value, higher the regularization. Regularization means penalizing large coefficients which don't improve the model's performance. default = 0 means no regularization.
   • Tune trick: Start with 0 and check CV error rate. If you see train error >>> test error, bring gamma into action. Higher the gamma, lower the difference in train and test CV. If you have no clue what value to use, use gamma=5 and see the performance. Remember that gamma brings improvement when you want to use shallow (low max_depth) trees.
4. max_depth[default=6][range: (0,Inf)]
   • It controls the depth of the tree.
   • Larger the depth, more complex the model; higher chances of overfitting. There is no standard value for max_depth. Larger data sets require deep trees to learn the rules from data.
   • Should be tuned using CV
5. min_child_weight[default=1][range:(0,Inf)]
   • In regression, it refers to the minimum number of instances required in a child node. In classification, if the leaf node has a minimum sum of instance weight (calculated by second order partial derivative) lower than min_child_weight, the tree splitting stops.
   • In simple words, it blocks the potential feature interactions to prevent overfitting. Should be tuned using CV.
6. subsample[default=1][range: (0,1)]
   • It controls the number of samples (observations) supplied to a tree.
   • Typically, its values lie between (0.5-0.8)
7. colsample_bytree[default=1][range: (0,1)]
   • It control the number of features (variables) supplied to a tree
   • Typically, its values lie between (0.5,0.9)
8. lambda[default=0]
   • It controls L2 regularization (equivalent to Ridge regression) on weights. It is used to avoid overfitting.
9. alpha[default=1]
   • It controls L1 regularization (equivalent to Lasso regression) on weights. In addition to shrinkage, enabling alpha also results in feature selection. Hence, it's more useful on high dimensional data sets.

Parameters for Linear Booster

Using linear booster has relatively lesser parameters to tune, hence it computes much faster than gbtree booster.
1. nrounds[default=100]
   • It controls the maximum number of iterations (steps) required for gradient descent to converge.
   • Should be tuned using CV
2. lambda[default=0]
   • It enables Ridge Regression. Same as above
3. alpha[default=1]
   • It enables Lasso Regression. Same as above

Learning Task Parameters

These parameters specify methods for the loss function and model evaluation. In addition to the parameters listed below, you are free to use a customized objective / evaluation function.

1. Objective[default=reg:linear]
   • reg:linear - for linear regression
   • binary:logistic - logistic regression for binary classification. It returns class probabilities
   • multi:softmax - multiclassification using softmax objective. It returns predicted class labels. It requires setting num_class parameter denoting number of unique prediction classes.
   • multi:softprob - multiclassification using softmax objective. It returns predicted class probabilities.
2. eval_metric [no default, depends on objective selected]
   • These metrics are used to evaluate a model's accuracy on validation data. For regression, default metric is RMSE. For classification, default metric is error.
   • Available error functions are as follows:
     • mae - Mean Absolute Error (used in regression)
     • Logloss - Negative loglikelihood (used in classification)
     • AUC - Area under curve (used in classification)
     • RMSE - Root mean square error (used in regression)
     • error - Binary classification error rate [#wrong cases/#all cases]
     • mlogloss - multiclass logloss (used in classification)

We've looked at how xgboost works, the significance of each of its tuning parameter, and how it affects the model's performance. Let's bolster our newly acquired knowledge by solving a practical problem in R.

Practical - Tuning XGBoost in R

In this practical section, we'll learn to tune xgboost in two ways: using the xgboost package and MLR package. I don't see the xgboost R package having any inbuilt feature for doing grid/random search. To overcome this bottleneck, we'll use MLR to perform the extensive parametric search and try to obtain optimal accuracy.

I'll use the adult data set from my previous random forest tutorial. This data set poses a classification problem where our job is to predict if the given user will have a salary <=50K or >50K.

Using random forest, we achieved an accuracy of 85.8%. Theoretically, xgboost should be able to surpass random forest's accuracy. Let's see if we can do it. I'll follow the most common but effective steps in parameter tuning:

1. First, you build the xgboost model using default parameters. You might be surprised to see that default parameters sometimes give impressive accuracy.
2. If you get a depressing model accuracy, do this: fix eta = 0.1, leave the rest of the parameters at default value, using xgb.cv function get best n_rounds. Now, build a model with these parameters and check the accuracy.
3. Otherwise, you can perform a grid search on rest of the parameters (max_depth, gamma, subsample, colsample_bytree etc) by fixing eta and nrounds. Note: If using gbtree, don't introduce gamma until you see a significant difference in your train and test error.
4. Using the best parameters from grid search, tune the regularization parameters(alpha,lambda) if required.
5. At last, increase/decrease eta and follow the procedure. But remember, excessively lower eta values would allow the model to learn deep interactions in the data and in this process, it might capture noise. So be careful!

This process might sound a bit complicated, but it's quite easy to code in R. Don't worry, I've demonstrated all the steps below. Let's get into actions now and quickly prepare our data for modeling (if you don't understand any line of code, ask me in comments):

# set working directory
path <- "~/December 2016/XGBoost_Tutorial"
setwd(path)

# load libraries
library(data.table)
library(mlr)

# set variable names
setcol <- c("age",
            "workclass",
            "fnlwgt",
            "education",
            "education-num",
            "marital-status",
            "occupation",
            "relationship",
            "race",
            "sex",
            "capital-gain",
            "capital-loss",
            "hours-per-week",
            "native-country",
            "target")

# load data
train <- read.table("adultdata.txt", header = FALSE, sep = ",",
                    col.names = setcol, na.strings = c(" ?"),
                    stringsAsFactors = FALSE)
test <- read.table("adulttest.txt", header = FALSE, sep = ",",
                   col.names = setcol, skip = 1,
                   na.strings = c(" ?"), stringsAsFactors = FALSE)

# convert data frame to data table
setDT(train)
setDT(test)

# check missing values
table(is.na(train))
sapply(train, function(x) sum(is.na(x)) / length(x)) * 100
table(is.na(test))
sapply(test, function(x) sum(is.na(x)) / length(x)) * 100

# quick data cleaning
# remove extra character from target variable
library(stringr)
test[, target := substr(target, start = 1, stop = nchar(target) - 1)]

# remove leading whitespaces
char_col <- colnames(train)[sapply(test, is.character)]
for (i in char_col) set(train, j = i, value = str_trim(train[[i]], side = "left"))
for (i in char_col) set(test, j = i, value = str_trim(test[[i]], side = "left"))

# set all missing value as "Missing"
train[is.na(train)] <- "Missing"
test[is.na(test)] <- "Missing"

Up to this point, we dealt with basic data cleaning and data inconsistencies. To use xgboost package, keep these things in mind:

1. Convert the categorical variables into numeric using one hot encoding
2. For classification, if the dependent variable belongs to class factor, convert it to numeric

R's base function model.matrix is quick enough to implement one hot encoding. In the code below, ~.+0 leads to encoding of all categorical variables without producing an intercept. Alternatively, you can use the dummies package to accomplish the same task. Since xgboost package accepts target variable separately, we'll do the encoding keeping this in mind:

# using one hot encoding
>labels <- train$target
>ts_label <- test$target
>new_tr <- model.matrix(~.+0, data = train[,-c("target"), with = FALSE])
>new_ts <- model.matrix(~.+0, data = test[,-c("target"), with = FALSE])

# convert factor to numeric
>labels <- as.numeric(labels) - 1
>ts_label <- as.numeric(ts_label) - 1

For xgboost, we'll use xgb.DMatrix to convert data table into a matrix (most recommended):

# preparing matrix
>dtrain <- xgb.DMatrix(data = new_tr, label = labels)
&t;dtest <- xgb.DMatrix(data = new_ts, label = ts_label)

As mentioned above, we'll first build our model using default parameters, keeping random forest's accuracy 85.8% in mind. I'll capture the default parameters from above (written against every parameter):

# default parameters
params <- list(
    booster = "gbtree",
    objective = "binary:logistic",
    eta = 0.3,
    gamma = 0,
    max_depth = 6,
    min_child_weight = 1,
    subsample = 1,
    colsample_bytree = 1
)

Using the inbuilt xgb.cv function, let's calculate the best nround for this model. In addition, this function also returns CV error, which is an estimate of test error.

xgbcv <- xgb.cv(
    params = params,
    data = dtrain,
    nrounds = 100,
    nfold = 5,
    showsd = TRUE,
    stratified = TRUE,
    print.every.n = 10,
    early.stop.round = 20,
    maximize = FALSE
)
# best iteration = 79

The model returned lowest error at the 79th (nround) iteration. Also, if you noticed the running messages in your console, you would have understood that train and test error are following each other. We'll use this insight in the following code. Now, we'll see our CV error:

min(xgbcv$test.error.mean)
# 0.1263

As compared to my previous random forest model, this CV accuracy (100-12.63)=87.37% looks better already. However, I believe cross-validation accuracy is usually more optimistic than true test accuracy. Let's calculate our test set accuracy and determine if this default model makes sense:

# first default - model training
xgb1 <- xgb.train(
    params = params,
    data = dtrain,
    nrounds = 79,
    watchlist = list(val = dtest, train = dtrain),
    print.every.n = 10,
    early.stop.round = 10,
    maximize = FALSE,
    eval_metric = "error"
)

# model prediction
xgbpred <- predict(xgb1, dtest)
xgbpred <- ifelse(xgbpred > 0.5, 1, 0)

The objective function binary:logistic returns output predictions rather than labels. To convert it, we need to manually use a cutoff value. As seen above, I've used 0.5 as my cutoff value for predictions. We can calculate our model's accuracy using confusionMatrix() function from caret package.

# confusion matrix
library(caret)
confusionMatrix(xgbpred, ts_label)
# Accuracy - 86.54%

# view variable importance plot
mat <- xgb.importance(feature_names = colnames(new_tr), model = xgb1)
xgb.plot.importance(importance_matrix = mat[1:20])  # first 20 variables

As you can see, we've achieved better accuracy than a random forest model using default parameters in xgboost. Can we still improve it? Let's proceed to the random / grid search procedure and attempt to find better accuracy. From here on, we'll be using the MLR package for model building. A quick reminder, the MLR package creates its own frame of data, learner as shown below. Also, keep in mind that task functions in mlr doesn't accept character variables. Hence, we need to convert them to factors before creating task:

# convert characters to factors
fact_col <- colnames(train)[sapply(train, is.character)]
for (i in fact_col) set(train, j = i, value = factor(train[[i]]))
for (i in fact_col) set(test, j = i, value = factor(test[[i]]))

# create tasks
traintask <- makeClassifTask(data = train, target = "target")
testtask <- makeClassifTask(data = test, target = "target")

# do one hot encoding
traintask <- createDummyFeatures(obj = traintask, target = "target")
testtask <- createDummyFeatures(obj = testtask, target = "target")

Now, we'll set the learner and fix the number of rounds and eta as discussed above.

# create learner
lrn <- makeLearner("classif.xgboost", predict.type = "response")
lrn$par.vals <- list(
    objective = "binary:logistic",
    eval_metric = "error",
    nrounds = 100L,
    eta = 0.1
)

# set parameter space
params <- makeParamSet(
    makeDiscreteParam("booster", values = c("gbtree", "gblinear")),
    makeIntegerParam("max_depth", lower = 3L, upper = 10L),
    makeNumericParam("min_child_weight", lower = 1L, upper = 10L),
    makeNumericParam("subsample", lower = 0.5, upper = 1),
    makeNumericParam("colsample_bytree", lower = 0.5, upper = 1)
)

# set resampling strategy
rdesc <- makeResampleDesc("CV", stratify = TRUE, iters = 5L)

With stratify=T, we'll ensure that distribution of target class is maintained in the resampled data sets. If you've noticed above, in the parameter set, I didn't consider gamma for tuning. Simply because during cross validation, we saw that train and test error are in sync with each other. Had either one of them been dragging or rushing, we could have brought this parameter into action.

Now, we'll set the search optimization strategy. Though, xgboost is fast, instead of grid search, we'll use random search to find the best parameters.

On March 11, 2011, the most intense earthquake in Japanese history hit its north east coast — magnitude 9.0. The earthquake was so powerful that it severely damaged buildings, road, and rail infrastructure in Tokyo, which was 373 km away from the epicenter. This was an undersea megathrust earthquake.

Fifty minutes after the earthquake, tsunami waves as high as 13 meters hit the eastern part of Japan, leaving more than 15,500 dead, 6,000 injured, and 2,500 people missing. The tsunami struck so hard that it resulted in a Level-7 nuclear disaster in Fukushima (in Japanese, Fukushima means Fortune Island); it is the second nuclear disaster in the history of mankind which was given level-7 criticality after the Chernobyl disaster in 1986.

Fukushima had six separate boiling water reactors maintained by Tokyo Electric Power Company (TEPCO). Due to the powerful earthquake and tsunami, the nuclear power plant in Fukushima was subjected to serious structural damage. This resulted in hydrogen-air explosion on top of each nuclear reactor, releasing huge amounts of radioactive particles including Iodine-131, Caesium-134/137, Tellurium-129m, and Strontium 90 into the atmosphere. These radioactive particles contaminated air, soil, and water in and around Fukushima.

Unfortunately, Fukushima was one of the most agriculturally important regions in Japan. There are close to two million farmers in Japan, of which 70,000 are in Fukushima. The farmers of Fukushima had to deal with radioactive contamination of their soil, crops, livestock, and the marine ecosystem in addition to the tragedy caused by the tsunami. Due to radioactive contamination everywhere, the government of Japan imposed restrictions on the growing and selling of crops, dairy products, and seafood.

Now, Japan needed to increase its production to make up for the agricultural loss in Fukushima and, very importantly, it needed food products without radioactive contaminants. If this is not achieved, Japan would head toward a food crisis.

In Kameoka (a satellite town to the west of the Japanese city of Kyoto), the Japanese agriculture technology company SPREAD had started working on a smart farming system on barren land. SPREAD built a vegetable factory in a 2868.22 m²-area.

Inside the doors of this warehouse-like agriculture complex, SPREAD uses vertical farming system.

SPREAD vegetable factory grew lettuce in a soil-less and sunless ecosystem.

In an interview with CNN on September 19, 2016, Shinji Inada, the president of SPREAD, said, "The turning point was the incident in 2011 at the Fukushima nuclear facility. After what happened there, people became more aware of the importance of safe food and it kind of turned the tables for us."

Here a sophisticated form of hydroculture known as the Hydroponics method is used, where plants are grown in an aqueous solution of mineral nutrients with water as a solvent. In this method, terrestrial plants are grown with roots exposed to a mineral nutrient solution.

Since this is an indoor farm, LED lights on top of each shelf provide adequate light for photosynthesis and the growth of lettuce.

At this vegetable factory, human involvement is restricted to sowing the seeds; later, everything till the harvest is taken care of by robots and sensors.

With the help of modern sensor technology, the following factors are measured:

• Lighting
• Level of nutrients in the mineral solvent
• Humidity level inside the setup
• Temperature level inside the setup
• Level of CO₂

To maintain these factors at the optimal level, robots are installed which take corrective measures.

The lettuce is grown in a highly controlled environment at this facility. As a result of not using any fertilizer, the lettuce grown at this agriculture facility is richer in beta-carotene (an antioxidant) than farm-grown lettuce.

The setup has stacker cranes which carry the stacks to the robots for harvesting after they have reached maturity. It takes 40 days for a lettuce head to be ready for harvest, whereas it takes two months for farm-grown lettuce. After the harvest, the lettuces are moved to the packaging section without any outside intervention. There are either retail or packaging boxes wholesale used for that purpose. The SPREAD’s facility at Kameoka has an output capacity of 21,000 lettuces per day.

This innovative move by SPREAD is not only providing vegetables free from radioactive contamination but also contributing to the ecology to a great extent. Here is how:

Water Recycling

Water used for cultivation of lettuce in the vegetable factory is reused after filtration and purification. At this facility, 98% of water is recycled, and it significantly reduces the water consumption—a mere 0.83 liters is used to grow one head of lettuce.

Zero damage due to pesticides

Since this vegetable factory uses the hydroponics method to cultivate lettuce, pesticides are not required during the cultivation lifecycle. Hence, soil and water contamination is prevented and the balance of microbes and insects in ecosystem is preserved.

Reduction in energy consumption

SPREAD has developed specialized LED lights and an air conditioning system to be used at their vegetable factory, reducing the energy consumption by 30%.

SPREAD is currently working on a second vegetable factory at Keihanna, with an output capacity of 30,000 heads of lettuce per day with 86.7% reduction in water usage.

sudo apt-get update

sudo apt-get install build-essential

sudo apt-get install tcl8.5

wget http://download.redis.io/releases/redis-stable.tar.gz

tar xzf redis-stable.tar.gz

cd redis-stable

make 

make test

make install

?  utils sudo ./install_server.sh

Welcome to the redis service installer

This script will help you easily set up a running redis server.





Please select the redis port for this instance: [6379]

Selecting default: 6379

Please select the redis config file name [/etc/redis/6379.conf]

Selected default - /etc/redis/6379.conf

Please select the redis log file name [/var/log/redis_6379.log]

Selected default - /var/log/redis_6379.log

Please select the data directory for this instance [/var/lib/redis/6379]

Selected default - /var/lib/redis/6379

Please select the redis executable path [/usr/local/bin/redis-server]

Selected config:

Port           : 6379

Config file    : /etc/redis/6379.conf

Log file       : /var/log/redis_6379.log

Data dir       : /var/lib/redis/6379

Executable     : /usr/local/bin/redis-server

Cli Executable : /usr/local/bin/redis-cli

Is this ok? Then press ENTER to go on or Ctrl-C to abort.

Copied /tmp/6379.conf => /etc/init.d/redis_6379

Installing service...

 Adding system startup for /etc/init.d/redis_6379 ...

   /etc/rc0.d/K20redis_6379 -> ../init.d/redis_6379

   /etc/rc1.d/K20redis_6379 -> ../init.d/redis_6379

   /etc/rc6.d/K20redis_6379 -> ../init.d/redis_6379

   /etc/rc2.d/S20redis_6379 -> ../init.d/redis_6379

   /etc/rc3.d/S20redis_6379 -> ../init.d/redis_6379

   /etc/rc4.d/S20redis_6379 -> ../init.d/redis_6379

   /etc/rc5.d/S20redis_6379 -> ../init.d/redis_6379

Success!

Starting Redis server...

Installation successful!

?  utils redis-cli

127.0.0.1:6379> exit

127.0.0.1:6379> SET users:GeorgeWashington "lang: python, born:1990"

OK

127.0.0.1:6379> GET users:GeorgeWashington

"lang: python, born:1990"

127.0.0.1:6379> exit

?  redis_tutorial virtualenv venv -p python3.5

Running virtualenv with interpreter /usr/bin/python3.5

Using base prefix '/usr'

New python executable in venv/bin/python3.5

Also creating executable in venv/bin/python

Installing setuptools, pip...done.

?  redis_tutorial source venv/bin/activate

(venv)?  redis_tutorial

(venv)?  redis_tutorial

(venv)?  redis_tutorial pip install redis

Downloading/unpacking redis

  Downloading redis-2.10.5-py2.py3-none-any.whl (60kB): 60kB downloaded

Installing collected packages: redis

Successfully installed redis

Cleaning up...

(venv)?  redis_tutorial python

Python 3.5.2 (default, Jul 17 2016, 00:00:00)

[GCC 4.8.4] on linux

Type "help", "copyright", "credits" or "license" for more information.

>>> import redis

>>> r = redis.StrictRedis()

>>> r.get("mykey")

>>> r.get("mykey")

>>> r.get("users:GeorgeWashington")

b'lang: python, born:1990'

This post will help you get started with Python decorators through some real life examples. Some familiarity with the Python programming language is expected.

Before directly jumping into decorators, let’s take a step back and start with Python functions. This will help you understand the concepts better.

Functions

def introduce(name):

    return 'My name is %s' % name

• def is the keyword used to define a function.
• introduce is the name of the function.
• the variable inside parentheses (name) is the required argument for the function.
• next line is the body or definition of the function.

Function Properties

In Python, functions are treated as first-class objects. This means that Python treats functions as values. We can assign a function to a variable, pass it as an argument to another function, or return it as a value from another function.

def print_hello_world():

    print('Hello World!')

>>> modified_world = print_hello_world

>>> modified_world()

Hello World!

def execute(func):

    print('Before execution')

    func()

    print('After execution')

>>> execute(print_hello_world)

Before execution

Hello World!

After execution

Python also supports the nesting of functions. It means we can define another function in the body or definition of some other function. Example:

def foo(x):

    def bar(y):

        return x+y

    return bar

In the example above, we have used two concepts described earlier.

1. Returning a function (bar) as a return value of the function foo

2. Nesting function bar in the definition of the function foo

Let’s see this code in action.

>>> v1 = foo(2)

Here v1 stores the return value of the function foo,which is another function bar. Now what will happen if we call v1 with some parameter?

>>>print(v1(5))

7

When a function is handled as data (in our case, return as a value from another function), it implicitly carries information required to execute the function. This is called closures in Python. We can check the closure of the function using __closure__ attribute of the function. This will return a tuple containing all the closures of the function. If we want to see any content of the closure, we can do something like v1.__closure__[0].cell_contents.

>>> v1.__closure__

(<cell at 0x7f4368e6c590: int object at 0xa41140>,)

>>> v1.__closure__[0].cell_contents

2

So, now that we looked at both function properties, let's see how we can use these properties in real scenarios.

Going Ahead

Suppose we want to perform some generic functionality before or/and after function execution. It can be like printing the execution time of the function.

One way to do this is by writing whatever we want to do before and after execution as initial and final statements, respectively. Example:

def print_hello_world():

    print('Before Execution')

    print('Hello World!')

    print('After Execution')

Is this a good way. I leave it to you. What will happen if we have several functions and need to perform the same task for all other functions too?

Another way could be to write a function that will take any other function as an argument and return the function along with performing the task before and after function execution. Example:

def print_hello_world():

    print('Hello World')



def dec(func):

    def nest_func(*args, **kwargs):

        print('Before Execution')

        r =  func(*args, **kwargs)

        print('After Execution')

        return r

    return nest_func

The function print_hello_world just prints ‘Hello World’. Function dec takes a function as an argument and creates another function nest_func in its definition. nest_func prints some statements before and after the execution of the function is passed as an argument to function dec.

Let’s pass the function print_hello_world to dec.

>>> new_print_hello_world = dec(print_hello_world)

new_print_function is another function returned by the function dec. What will be the output on calling new_print_hello_world function? Let’s check it.

>>> new_print_hello_world()

Before Execution

Hello World

After Execution

>>> print_hello_world = dec(print_hello_world)

>>> print_hello_world()

Before Execution

Hello World!

After Execution

We have changed the functionality of the function print_hello_world without changing the source code of the function itself.

So what next? If everything is clear till this point, then we have already learned about decorators. Let me explain.

Decorators

A decorator is a function which gives us the freedom to enhance or change the functionality of any function dynamically, without making changes in the code of the function.

In our case, function dec provides us with this functionality (as it changes the functionality of the function print_hello_world). So dec is called decorator. Instead of passing print_hello_world explicitly to function dec, we can use its shorthand syntax:

@dec

def print_hello_world():

    print('Hello World')

I hope by now you understand what decorators are. You might be wondering why we need to return a function from the dec function? Just call the function in dec itself in which we can print statements along with executing the function passed as an argument. Example:

def dec1(func):

    print('Before Execution')

    func()

    print('After Execution')

>>> print_hello_world = dec1(print_hello_world)

1. What value does print_hello_world store right now? Can you call it now? (It is storing None which is the return value of the function dec1. So you can’t call print_hello_world now)
2. What if we want to enhance a function having some arguments? One suggestion could be like this:

def dec2(func, arg1, arg2):

    print('Before Execution')

    func(arg1, arg2)

    print('After Execution')

>>> print_hello_world = dec2(print_hello_world, arg1, arg2)

Here, we will not be able to get the value of arg1 and arg2.

I hope these two points clearly explain why decorators are required to return a function.

Decorator Examples

• It can be used to compute the execution time of any function.

  def compute_execution_time(func):
  
      def nest_func(*args, **kwargs):
  
          start = time.time()
  
          response = func(*args, **kwargs)
  
          end = time.time()
  
          print(end-start)
  
          return response
  
      return nest_func

  
  
• In web applications, it can be used to check if the user is logged in or not.

  def login_required(func):
  
      def nest_function(request, *args, **kwargs):
  
          if request.user.is_authenticated():
  
              return func(request, *args, **kwargs)
  
          else
  
              return redirect('/login')
  
      return nest_function

  
  

I hope this article gives you a basic idea about Python decorators and some of their use cases. If you have any queries or feedback, you can reach me at udr.ritesh@gmail.com.

Introduction

"The road to machine learning starts with Regression. Are you ready?"

If you are aspiring to become a data scientist, regression is the first algorithm you need to learn master. Not just to clear job interviews, but to solve real-world problems. Many consultancy firms continue to use regression techniques on a large scale to help their clients. While it's one of the easiest algorithms to learn, it requires persistent effort to master.

Running a regression model is simple. A basic model <- y ~ x does the job. But optimizing it for better accuracy is the real challenge. If your model gives adjusted R² = 0.678, how will you improve it?

This article introduces key regression concepts with practical applications in R. You’ll learn to build, improve, and optimize regression models. The focus here is on linear and multiple regression.

Note: Best suited for readers new to machine learning who have some knowledge of statistics. R should be installed.

Table of Contents

1. What is Regression? How does it work?
2. What are the assumptions made in Regression?
3. How do I know if these assumptions are violated in my data?
4. How can I improve the accuracy of a Regression Model?
5. How can I assess the fit of a Regression Model?
6. Practice Time – Solving a Regression Problem

@jeffjarvis Drop non-responsive To:/CC:/BCC:, hash both sets, then subtract those that match. Old laptops could do it in minutes-to-hours.

— Edward Snowden (@Snowden) November 7, 2016

{"menu":

    {"id": "file",

    "value": "File",

    "popup": {

        "menuitem": [

        {"value": "New", "onclick": "CreateNewDoc()"},

        {"value": "Open", "onclick": "OpenDoc()"},

        {"value": "Close", "onclick": "CloseDoc()"}

    ]}

}}

some_dict = {}

some_dict["menu"]["popup"]["value"] = "New"

import collections

tree = lambda: collections.defaultdict(tree)

some_dict = tree()

# below will create non existent keys

some_dict["menu"]["popup"]["value"] = "New"

ice_cream = collections.defaultdict(lambda: 'Vanilla')

ice_cream['Sarah'] = 'Chunky Monkey'

ice_cream['Abdul'] = 'Butter Pecan'

print(ice_cream['Sarah']) # out: 'Chunky Monkey'

print(ice_cream['Joe']) # out: 'Vanilla

import itertools

def constant_factory(value):

    return itertools.repeat(value).next

d = collections.defaultdict(constant_factory(''))

d.update(name='John', action='ran')

print('%(name)s %(action)s to %(object)s' % d)

from collections import defaultdict



try:

    t=unichr(100)

except NameError:

    unichr=chr



def f1(li):

    '''defaultdict'''

    d = defaultdict(list)

    for k, v in li:

        d[k].append(v)

    return d.items()



def f2(li):

    '''setdefault'''

    d={}

    for k, v in li:

        d.setdefault(k, []).append(v)

    return d.items()





if __name__ == '__main__':

    import timeit

    import sys

    print(sys.version)

    few=[('yellow', 1), ('blue', 2), ('yellow', 3), ('blue', 4), ('red', 1)]

    fmt='{:>12}: {:10.2f} micro sec/call ({:,} elements, {:,} keys)'

    for tag, m, n in [('small',5,10000), ('medium',20,1000), ('bigger',1000,100), ('large',5000,10)]:

        for f in [f1,f2]:

            s = few*m

            res=timeit.timeit("{}(s)".format(f.__name__), setup="from __main__ import {}, s".format(f.__name__), number=n)

            st=fmt.format(f.__doc__, res/n*1000000, len(s), len(f(s)))

            print(st)

            s = [(unichr(i%0x10000),i) for i in range(1,len(s)+1)]

            res=timeit.timeit("{}(s)".format(f.__name__), setup="from __main__ import {}, s".format(f.__name__), number=n)

            st=fmt.format(f.__doc__, res/n*1000000, len(s), len(f(s)))

            print(st)

        print()

3.5.2 |Anaconda 4.1.1 (32-bit)| (default, Jul  5 2016, 11:45:57) [MSC v.1900 32 bit (Intel)]

 defaultdict:       5.48 micro sec/call (25 elements, 3 keys)

 defaultdict:      11.20 micro sec/call (25 elements, 25 keys)

  setdefault:       7.80 micro sec/call (25 elements, 3 keys)

  setdefault:       8.97 micro sec/call (25 elements, 25 keys)



 defaultdict:      14.66 micro sec/call (100 elements, 3 keys)

 defaultdict:      42.19 micro sec/call (100 elements, 100 keys)

  setdefault:      26.71 micro sec/call (100 elements, 3 keys)

  setdefault:      34.78 micro sec/call (100 elements, 100 keys)



 defaultdict:     623.21 micro sec/call (5,000 elements, 3 keys)

 defaultdict:    2207.91 micro sec/call (5,000 elements, 5,000 keys)

  setdefault:    1329.99 micro sec/call (5,000 elements, 3 keys)

  setdefault:    3076.57 micro sec/call (5,000 elements, 5,000 keys)



 defaultdict:    4625.00 micro sec/call (25,000 elements, 3 keys)

 defaultdict:   15950.98 micro sec/call (25,000 elements, 25,000 keys)

  setdefault:    6907.47 micro sec/call (25,000 elements, 3 keys)

  setdefault:   17605.08 micro sec/call (25,000 elements, 25,000 keys)