Developer Insights

Times are changing and have been for a while now. In the world of STEM, women are no longer considered a “bad fit,” which is easily proved by the amazing number of brilliant women in the field today. Women are just as interested in finding out how things work, extracting insight from data, problem-solving, and helping businesses make the right decisions. Staggering amounts of data and knowing that it will only grow has paved the way for boundless opportunities for jobs related to data science, across both genders.

Disappointing Statistics in Favor

Despite being named the sexiest job of the century, data science seems to have few takers among the women folk. Following are some interesting stats about women in the technology domain:

• The American Association of University Women found that the percentage of women in math and computational jobs fell from 35% in 1990 to 26% in 2013.
• BetterBuys’ collated report shows that women make up about 26% of data professionals, with 39% researcher roles, 28% in creative roles, 18% in business management roles, and 13% in developer roles.
• In 2014, women held only 13% of the Chief Information Officer and 25% of Chief Data Officer positions. What is worse is that research found “women were two times more likely than men to quit high-tech positions.”

So, what’s stopping more women from getting into data science and analytics?

We don’t want people talking about gender gap in the world of technology and analytics anymore. Seriously, there is no conspiracy to keep women out of this typically male-dominated sphere. We need diversity in the boardroom, like now.

Why are women not "gung-ho" about such an exciting field?

Women often face challenges in the form of stereotypes and condescension, especially in developing countries like India, when trying to prove their worth. Cultural perception affects their self-confidence and chances of growth. You find women struggling to find work-life balance. Battling these undercurrents and the lack of adequate support and encouragement at home and workplace are sources of stress many talented women are choosing to do without.

Not an ideal situation…

But take heart, aspiring women data scientists. This is what Evenbrite’s Senior Data Scientist, Vesela Gateva, says:

Once you have a very genuine curiosity in a quantitative field or anything science-related, let your curiosity be your main guidance. You shouldn’t think that you’re a woman. I never aspired to be a data scientist. It’s a very recent term. I just ended up being one. All I knew was that I wanted to apply my quantitative skills, solving interesting problems. Women in general tend to give themselves less credit than they deserve. What women should know is that once they have the curiosity, and the basic fundamentals of probability and statistics, computer science, and machine learning, they can figure out the rest on their own.

Gender shouldn’t limit accomplishments, and it certainly shouldn’t define a person’s identity.

14 Women who've hit the stereotype out of the park

What aspiring women data scientists need are to look to bright women who have defied odds to rise to leadership positions in the field of analytics. No point in whining about lack of female representation if you are going to contribute, is there?

Let's appreciate these women for their work and incessant dedication which has helped millions of people around the world to inspire, learn and rise in their respective careers.

Corinna Cortes, Google Research

She needs no introduction to people who in the world of Machine Learning. Corinna Cortes is the head of Google Research (NY), prior to which she was a distinguished researcher for a decade at AT&T Bell Labs. Her development of the algorithm, Support Vector Machines, fetched her the Paris Kanellakis Theory and Practice Award in 2008. She received her PhD in Computer Science in 1993 from the University of Rochester (NY) and has an MS in Physics from the University of Copenhagen. This amazing mother of two is a competitive runner as well. Read her latest tweets here.

Daphne Koller, Co-founder, Coursera

Israeli-American Daphne Koller is a leading expert in the field of machine learning, with special focus on probabilistic graphical models. She is the Chief Computing Officer at Calico Labs. Daphne is also the co-founder of the popular online education platform Coursera. She was a Stanford University professor of Computer Science for nearly two decades. Daphne Koller earned her PhD from Stanford, BS and MS from Hebrew University of Jerusalem, and has done her Post-doctoral research at UCLA. To view her many achievements, go here. When she’s not immersed in her work, you can find her spending time with her daughter or unwinding to music.

Adele Cutler, Random Forest Algorithm Co-Developer

Random Forests (a trademarked statistical classifier) co-developer Adele Cutler has a PhD from University of California, Berkeley, and a math degree from the University of Auckland. She’s been a statistics professor at Utah State University for almost three decades and continues her research in data mining and decision trees. She says, “As statisticians, what we’re really trying to do is think of better ways to get information out of data.” Adele Cutler has varied interests apart from math and stats, including spending time with her family in Taupo and Edinburgh, taking holidays, beading, and knitting. You can find more about her here.

Jenn Wortman Vaughan, Microsoft Research

Jennifer Vaughan is a Senior Researcher at NYC-based Microsoft Research. She is interested in learning models and algorithms related to data aggregation. She received her PhD in 2009 in Computer and Information Science from the University of Pennsylvania, a Masters from Stanford in Computer Science, and a Bachelors in Computer Science from Boston University. She previously worked as an Assistant Professor (CS) in UCLA and was a Harvard University Computing Innovation Fellow. She has a handful of prestigious awards to her name, including a National Science Foundation CAREER award and a Presidential Early Career Award for Scientists and Engineers. In 2006, Jenn co-founded the Annual Workshop for Women in Machine Learning. If you want to know about this rising star, go to here website.

Erin LeDell, Machine Learning Scientist, H2O.ai

California-based H2O.ai Machine Learning scientist, Erin LeDell has a doctorate in “Biostatistics and the Designated Emphasis in Computational Science and Engineering” from the University of California, Berkeley. She has a B.S. and M.A. in Mathematics. Her earlier work history includes working as the Principal Data Scientist at Wise.io and Marvin Mobile Security. Erin is also the founder of DataScientific, Inc. She has co-authored Subsemble: An Ensemble Method for Combining Subset-Specific Algorithm Fits. She is a co-founder of R-Ladies Global, an organization to encourage gender diversity in the R stats community. You can find Erin LeDell here.

Jennifer Bryan, Associate Professor Statistics, UBC

Jennifer Bryan is an Associate Professor, Statistics & Michael Smith Labs, at the University of British Columbia. She's a biostatistician specializing in genomics, and she enjoys statistical computing and data analysis. She has a BA in Economics from Yale and a doctoral degree from the University of California, Berkeley. She takes a popular introductory course in R. Look at her Twitter feed here.

Hilary Mason, Founder, Fast Forward Labs

In her own words, “I love data and cheeseburgers!” Based in New York, Hilary Mason is the founder of Fast Forward Labs, a machine intelligence research company, and the Data Scientist in Residence at Accel. Her magic doesn’t end there. She co-hosts DataGotham, is a member of NYCResistor, and co-founded of HackNY. Apart from being featured in top publications like the Scientific American, she has received the TechFellows Engineering Leadership award and was on the Forbes 40 under 40 Ones to Watch list. She has co-authored Data Driven: Creating a Data Culture. For inspiration, you should look at her LinkedIn profile.

Radhika Kulkarni, Vice President, Advanced Analytics R&D, SAS

Based in Durham, NC, Radhika Kulkarni is the Vice President, Advanced Analytics R&D, at SAS Institute Inc. She has a Masters in Mathematics from IIT-Delhi and a PhD in Operations Research from Cornell University. In her 30-year career with SAS, one of the foremost optimization software vendors, she has received many accolades—she is a SAS CEO Award of Excellence winner and chosen as one of the 100 Diverse Corporate Leaders in STEM by STEMconnector. She loves spending time with her three kids, and is very social. In her own words, “I'm well known to be the party animal.” Check out here tweets here.

Alice Zheng, Senior Manager, Amazon

Alice Zheng is a Senior Manager of Applied Science at Amazon. She heads the optimization team on Amazon's Ad Platform. She was a Microsoft researcher for six years before her stint as the Director of Data Science at Dato. Her focus is on building scalable models in Machine Learning. She has undergraduate degrees in Computer Science and Math and a doctoral degree in electrical engineering from the University of California, Berkeley. Alice Zheng has written two books in the field of data science. She says, “My research focuses on easing the dependence on expertise by making learning algorithms more automated, their outputs more interpretable, and the labeling tasks simpler.” Look at her LinkedIn profile to read more interesting things about her.

Charlotte Wickham, Assistant Professor Statistics, OSU

Charlotte Wickham works as an Assistant Professor of Statistics at the Oregon State University. An R specialist, she creates courseware for Data Camp. She has an Undergraduate degree in Statistics from the University of Auckland and a PhD in Statistics from the University of California, Berkeley. You can visit her website for more information.

Monica Rogati, Former Senior Data Scientist, LinkedIn

Former VP of Data at Jawbone and LinkedIn senior data scientist, Monica Rogati is now an independent data science advisor. Her description on Medium is quite apt: Turning data into products and stories. Based in Sunnyvale, California, she has a PhD in Computer Science from the Carnegie Mellon University and a B.S. in computer science from the University of New Mexico. Her expertise lies in applied machine learning, text mining, and recommender systems. From wearable computing to developing a system to match a job to a candidate, she is an ace at it all. Her LinkedIn profile is chock-full of achievements. You can also follow her at @mrogati.

Alice Daish, Data Scientist, British Museum

Alice Daish is a Data Scientist at the British Museum and a co-Founder of R-Ladies Global. She says, “I love data, R, science and innovation.” Her interests include data analysis, data visualization, predictive modelling, data communication, mentoring, and gender diversity in STEM.S he has a BSc. in Conservation Biology & Ecology from the University of Exeter and an MSc. in Quantitative Biology from Imperial College London. For a more detailed record of her projects and publications, go here. Follow Alice!

Amy O'Connor, Big Data Evangelist, Cloudera

Amy O'Connor is a Big Data evangelist at Cloudera. Prior to this, she was the Senior Director of the Big Data group at Nokia, and prior to that she was Senior Director of Strategy at Sun Microsystems. She describes herself as “a geek in high heels.” Amy O'Connor was on the Information Management’s “10 Big Data Experts to Know” in 2015. She has a BS in Electrical Engineering from the University of Connecticut and an MBA from Northeastern University. Follow her here.

Julia Evans, Machine Learning Engineer, Stripe

Montreal-based Julia Evans says “I love using serious systems in silly ways.” She has undergraduate and graduate degrees in Mathematics and Computer Science from McGill University. She works as a Machine Learning engineer at Stripe. She is passionate about programming and puts events together for women with similar interests. You can read for yourself here. Follow her interesting tweets here.

Women have great communication skills—a necessary skill when you need to tell decision makers what the results of the data analysis are. They are collaborative by nature—a key skill when people from different fields work together. They can think differently and tackle assumptions—vital skills when coupled with business acumen, stats, math, computer science, modeling, and analytical expertise. Admittedly men and women think differently. But that is what analysis is about, isn’t it? Different perspectives?

Like machine learning expert Claudia Perlich, Chief Scientist at Dstillery, said,

“Ultimately, data science is another technical field where women remain statistically a minority, but I do not believe that we need to force the issue or “fight” for a higher female quota. I want to come to work and do what I love and be recognized for what I bring to the table and not waste even one thought on the fact that I am female.”

So there really is no excuse for women to not enter this fascinating world of Data Science is there? Women just need to recognize that they have so much to bring to the table.

Introduction

Treat "forests" well. Not for the sake of nature, but for solving problems too!

Random Forest is one of the most versatile machine learning algorithms available today. With its built-in ensembling capacity, the task of building a decent generalized model (on any dataset) gets much easier. However, I've seen people using random forest as a black box model; i.e., they don't understand what's happening beneath the code. They just code.

In fact, the easiest part of machine learning is coding. If you are new to machine learning, the random forest algorithm should be on your tips. Its ability to solve—both regression and classification problems along with robustness to correlated features and variable importance plot gives us enough head start to solve various problems.

Most often, I've seen people getting confused in bagging and random forest. Do you know the difference?

In this article, I'll explain the complete concept of random forest and bagging. For ease of understanding, I've kept the explanation simple yet enriching. I've used MLR, data.table packages to implement bagging, and random forest with parameter tuning in R. Also, you'll learn the techniques I've used to improve model accuracy from ~82% to 86%.

Table of Contents

1. What is the Random Forest algorithm?
2. How does it work? (Decision Tree, Random Forest)
3. What is the difference between Bagging and Random Forest?
4. Advantages and Disadvantages of Random Forest
5. Solving a Problem
   • Parameter Tuning in Random Forest

What is the Random Forest algorithm?

Random forest is a tree-based algorithm which involves building several trees (decision trees), then combining their output to improve generalization ability of the model. The method of combining trees is known as an ensemble method. Ensembling is nothing but a combination of weak learners (individual trees) to produce a strong learner.

Say, you want to watch a movie. But you are uncertain of its reviews. You ask 10 people who have watched the movie. 8 of them said "the movie is fantastic." Since the majority is in favor, you decide to watch the movie. This is how we use ensemble techniques in our daily life too.

Random Forest can be used to solve regression and classification problems. In regression problems, the dependent variable is continuous. In classification problems, the dependent variable is categorical.

Trivia: The random Forest algorithm was created by Leo Breiman and Adele Cutler in 2001.

How does it work? (Decision Tree, Random Forest)

To understand the working of a random forest, it's crucial that you understand a tree. A tree works in the following way:

1. Given a data frame (n x p), a tree stratifies or partitions the data based on rules (if-else). Yes, a tree creates rules. These rules divide the data set into distinct and non-overlapping regions. These rules are determined by a variable's contribution to the homogeneity or pureness of the resultant child nodes (X2, X3).

2. In the image above, the variable X1 resulted in highest homogeneity in child nodes, hence it became the root node. A variable at root node is also seen as the most important variable in the data set.

3. But how is this homogeneity or pureness determined? In other words, how does the tree decide at which variable to split?

• In regression trees (where the output is predicted using the mean of observations in the terminal nodes), the splitting decision is based on minimizing RSS. The variable which leads to the greatest possible reduction in RSS is chosen as the root node. The tree splitting takes a top-down greedy approach, also known as recursive binary splitting. We call it "greedy" because the algorithm cares to make the best split at the current step rather than saving a split for better results on future nodes.
• In classification trees (where the output is predicted using mode of observations in the terminal nodes), the splitting decision is based on the following methods:
  • Gini Index - It's a measure of node purity. If the Gini index takes on a smaller value, it suggests that the node is pure. For a split to take place, the Gini index for a child node should be less than that for the parent node.
  • Entropy - Entropy is a measure of node impurity. For a binary class (a, b), the formula to calculate it is shown below. Entropy is maximum at p = 0.5. For p(X=a)=0.5 or p(X=b)=0.5 means a new observation has a 50%-50% chance of getting classified in either class. The entropy is minimum when the probability is 0 or 1.

Entropy = - p(a)*log(p(a)) - p(b)*log(p(b))

In a nutshell, every tree attempts to create rules in such a way that the resultant terminal nodes could be as pure as possible. Higher the purity, lesser the uncertainty to make the decision.

But a decision tree suffers from high variance. "High Variance" means getting high prediction error on unseen data. We can overcome the variance problem by using more data for training. But since the data set available is limited to us, we can use resampling techniques like bagging and random forest to generate more data.

Building many decision trees results in a forest. A random forest works the following way:

1. First, it uses the Bagging (Bootstrap Aggregating) algorithm to create random samples. Given a data set D1 (n rows and p columns), it creates a new dataset (D2) by sampling n cases at random with replacement from the original data. About 1/3 of the rows from D1 are left out, known as Out of Bag (OOB) samples.
2. Then, the model trains on D2. OOB sample is used to determine unbiased estimate of the error.
3. Out of p columns, P ≪ p columns are selected at each node in the data set. The P columns are selected at random. Usually, the default choice of P is p/3 for regression tree and √p for classification tree.
4. Unlike a tree, no pruning takes place in random forest; i.e., each tree is grown fully. In decision trees, pruning is a method to avoid overfitting. Pruning means selecting a subtree that leads to the lowest test error rate. We can use cross-validation to determine the test error rate of a subtree.
5. Several trees are grown and the final prediction is obtained by averaging (for regression) or majority voting (for classification).

Each tree is grown on a different sample of original data. Since random forest has the feature to calculate OOB error internally, cross-validation doesn't make much sense in random forest.

What is the difference between Bagging and Random Forest?

Many a time, we fail to ascertain that bagging is not the same as random forest. To understand the difference, let's see how bagging works:

1. It creates randomized samples of the dataset (just like random forest) and grows trees on a different sample of the original data. The remaining 1/3 of the sample is used to estimate unbiased OOB error.
2. It considers all the features at a node (for splitting).
3. Once the trees are fully grown, it uses averaging or voting to combine the resultant predictions.

Aren't you thinking, "If both the algorithms do the same thing, what is the need for random forest? Couldn't we have accomplished our task with bagging?" NO!

The need for random forest surfaced after discovering that the bagging algorithm results in correlated trees when faced with a dataset having strong predictors. Unfortunately, averaging several highly correlated trees doesn't lead to a large reduction in variance.

But how do correlated trees emerge? Good question! Let's say a dataset has a very strong predictor, along with other moderately strong predictors. In bagging, a tree grown every time would consider the very strong predictor at its root node, thereby resulting in trees similar to each other.

The main difference between random forest and bagging is that random forest considers only a subset of predictors at a split. This results in trees with different predictors at the top split, thereby resulting in decorrelated trees and more reliable average output. That's why we say random forest is robust to correlated predictors.

Advantages and Disadvantages of Random Forest

Advantages are as follows:

1. It is robust to correlated predictors.
2. It is used to solve both regression and classification problems.
3. It can also be used to solve unsupervised ML problems.
4. It can handle thousands of input variables without variable selection.
5. It can be used as a feature selection tool using its variable importance plot.
6. It takes care of missing data internally in an effective manner.

Disadvantages are as follows:

1. The Random Forest model is difficult to interpret.
2. It tends to return erratic predictions for observations out of the range of training data. For example, if the training data contains a variable x ranging from 30 to 70, and the test data has x = 200, random forest would give an unreliable prediction.
3. It can take longer than expected to compute a large number of trees.

Solving a Problem (Parameter Tuning)

Let's take a dataset to compare the performance of bagging and random forest algorithms. Along the way, I'll also explain important parameters used for parameter tuning. In R, we'll use MLR and data.table packages to do this analysis.

I've taken the Adult dataset from the UCI machine learning repository. You can download the data from here.

This dataset presents a binary classification problem to solve. Given a set of features, we need to predict if a person's salary is <=50K or >=50K. Since the given data isn't well structured, we'll need to make some modification while reading the dataset.

# set working directory
path <- "~/December 2016/RF_Tutorial"
setwd(path)

# Set working directory
path <- "~/December 2016/RF_Tutorial"
setwd(path)

# Load libraries
library(data.table)
library(mlr)
library(h2o)

# Set variable names
setcol <- c("age",
            "workclass",
            "fnlwgt",
            "education",
            "education-num",
            "marital-status",
            "occupation",
            "relationship",
            "race",
            "sex",
            "capital-gain",
            "capital-loss",
            "hours-per-week",
            "native-country",
            "target")

# Load data
train <- read.table("adultdata.txt", header = FALSE, sep = ",", 
                    col.names = setcol, na.strings = c(" ?"), stringsAsFactors = FALSE)
test <- read.table("adulttest.txt", header = FALSE, sep = ",", 
                   col.names = setcol, skip = 1, na.strings = c(" ?"), stringsAsFactors = FALSE)

After we've loaded the dataset, first we'll set the data class to data.table. data.table is the most powerful R package made for faster data manipulation.

>setDT(train)
>setDT(test)

Now, we'll quickly look at given variables, data dimensions, etc.

>dim(train)
>dim(test)
>str(train)
>str(test)

As seen from the output above, we can derive the following insights:

1. The train dataset has 32,561 rows and 15 columns.
2. The test dataset has 16,281 rows and 15 columns.
3. Variable target is the dependent variable.
4. The target variable in train and test data is different. We'll need to match them.
5. All character variables have a leading whitespace which can be removed.

We can check missing values using:

# Check missing values in train and test datasets
>table(is.na(train))
# Output:
#  FALSE   TRUE 
#  484153  4262

>sapply(train, function(x) sum(is.na(x)) / length(x)) * 100

table(is.na(test))
# Output:
#  FALSE  TRUE 
#  242012 2203

>sapply(test, function(x) sum(is.na(x)) / length(x)) * 100

As seen above, both train and test datasets have missing values. The sapply function is quite handy when it comes to performing column computations. Above, it returns the percentage of missing values per column.

Now, we'll preprocess the data to prepare it for training. In R, random forest internally takes care of missing values using mean/mode imputation. Practically speaking, sometimes it takes longer than expected for the model to run.

Therefore, in order to avoid waiting time, let's impute the missing values using median/mode imputation method; i.e., missing values in the integer variables will be imputed with median and in the factor variables with mode (most frequent value).

We'll use the impute function from the mlr package, which is enabled with several unique methods for missing value imputation:

# Impute missing values
>imp1 <- impute(data = train, target = "target", 
              classes = list(integer = imputeMedian(), factor = imputeMode()))

>imp2 <- impute(data = test, target = "target", 
              classes = list(integer = imputeMedian(), factor = imputeMode()))

# Assign the imputed data back to train and test
>train <- imp1$data
>test <- imp2$data

Being a binary classification problem, you are always advised to check if the data is imbalanced or not. We can do it in the following way:

# Check class distribution in train and test datasets
setDT(train)[, .N / nrow(train), target]
# Output:
#    target     V1
# 1: <=50K   0.7591904
# 2: >50K    0.2408096

setDT(test)[, .N / nrow(test), target]
# Output:
#    target     V1
# 1: <=50K.  0.7637737
# 2: >50K.   0.2362263

If you observe carefully, the value of the target variable is different in test and train. For now, we can consider it a typo error and correct all the test values. Also, we see that 75% of people in the train data have income <=50K. Imbalanced classification problems are known to be more skewed with a binary class distribution of 90% to 10%. Now, let's proceed and clean the target column in test data.

# Clean trailing character in test target values
test[, target := substr(target, start = 1, stop = nchar(target) - 1)]

We've used the substr function to return the substring from a specified start and end position. Next, we'll remove the leading whitespaces from all character variables. We'll use the str_trim function from the stringr package.

> library(stringr)
> char_col <- colnames(train)[sapply(train, is.character)]
> for(i in char_col)
>     set(train, j = i, value = str_trim(train[[i]], side = "left"))

Using sapply function, we've extracted the column names which have character class. Then, using a simple for - set loop we traversed all those columns and applied the str_trim function.

Before we start model training, we should convert all character variables to factor. MLR package treats character class as unknown.

> fact_col <- colnames(train)[sapply(train,is.character)]
>for(i in fact_col)
			set(train,j=i,value = factor(train[[i]]))
>for(i in fact_col)
	     set(test,j=i,value = factor(test[[i]]))

Let's start with modeling now. MLR package has its own function to convert data into a task, build learners, and optimize learning algorithms. I suggest you stick to the modeling structure described below for using MLR on any data set.

#create a task
> traintask <- makeClassifTask(data = train,target = "target")
> testtask <- makeClassifTask(data = test,target = "target")

#create learner
> bag <- makeLearner("classif.rpart",predict.type = "response")
> bag.lrn <- makeBaggingWrapper(learner = bag,bw.iters = 100,bw.replace = TRUE)

I've set up the bagging algorithm which will grow 100 trees on randomized samples of data with replacement. To check the performance, let's set up a validation strategy too:

#set 5 fold cross validation
> rdesc <- makeResampleDesc("CV", iters = 5L)

For faster computation, we'll use parallel computation backend. Make sure your machine / laptop doesn't have many programs running in the background.

#set parallel backend (Windows)
> library(parallelMap)
> library(parallel)
> parallelStartSocket(cpus = detectCores())