Developer Insights

HackerEarth is committed to honoring its users’ rights to data privacy and protection. We have a privacy-conscious culture, and GDPR is an opportunity for us to strengthen this further. Being GDPR-ready has been of the highest priority this past year, and our product and legal teams have devoted a lot of extra hours to adhere to its requirements, give users more control over their data, and explain what we do with the data. (PS: To further our crusade toward data protection, we are also in the process of the getting the ISO 27001 certification.)

What is GDPR?

General Data Protection Regulation (GDPR), which will go into effect on May 25, 2018, replaces the 1995 Data Protection Directive. Designed to give EU citizens more control over their data, it aims to use one all-encompassing privacy and security law to safeguard personal data. Regardless of their location, relevant controllers or processors dealing with EU residents’ personal data are required to update or craft new policies ahead of the date or be prepared for penalties.

What is personal data?

Article 4 in GDPR definition states that ‘Personal data’ means any information relating to an identified or identifiable natural person (‘data subject’); an identifiable natural person is one who can be identified, directly or indirectly, in particular by reference to an identifier such as a name, an identification number, location data, an online identifier or to one or more factors specific to the physical, physiological, genetic, mental, economic, cultural or social identity of that natural person.Both personally identifiable information (PII) and information which can be cross-referenced with other information to identify a person is included in the definition. Examples of sensitive PII include medical information, biometric information, social security ID, license number, birth date, etc. The personal data collected should be pseudonymized and/or encrypted.

How is HackerEarth getting ready for GDPR?

In our efforts to get the organization ready for sustainable compliance, HackerEarth has taken several steps—from raising awareness in the organization about the principles of GDPR and our data protection policy to training employees to responsibly handle user data and auditing.Also, to make sure our sub-processors do no breach the regulation, we are assessing our third-party service providers and partners and fine-tuning the contracts.

Product preparation

We have assessed HackerEarth Sprint, our innovation management software, and HackerEarth Recruit, our Technical Recruitment software, against the requirements of the GDPR and have implemented features that will help users achieve compliance.Our application teams strongly believe in letting the end users exercise their rights with respect to privacy. We are working to give you more control over the data you store in our systems. These provisions may vary based on your requirement, product characteristics, and mutually agreed upon statement of work. Our teams are working on these features and enhancements, which will be rolled out in phases.How HackerEarth enables customers to be GDPR compliant:

• • We have revised our privacy policy and terms of service.
  • We are encrypting all data in transit and at rest.
  • We are identifying and creating multiple delete profile use cases, including administrators having the control to delete users.

HackerEarth is also taking care of many more such features to ensure the customers are compliant and users have complete control over their data.

Process preparation

Based on our data flows and data handling practices, we have revised our privacy policy and added further information on the personal information we collect, why we collect it, how we will use it, how long we will store it, and so on. Moreover, we are reviewing our databases to make sure we have only the latest and most accurate information.We have put together a glossary of the terms and information on when HackerEarth acts as a data processor or a data controller. Additionally, we have appointed internal privacy champions for all our teams.

What happens in the event of a data breach?

In case a personal data breach occurs, we will send breach notifications in accordance with our internal incident response policy.We will notify our customers within 72 hours of us discovering the breach.
We will notify users through our blogs and social media for general incidents.
We will notify the concerned party through email (using the primary email address) for incidents specific to an individual user or an organization.

We have a whole series of blogs planned, with more updates and information to come. Please feel free to ask questions and share your concerns with us at vr-gdpr@hackerearth.com.

***For more information, see our Privacy Policy here.

Welcome to Part II of the series on data visualization. In the last blog post, we explored different ways to visualize continuous variables and infer information. If you haven’t visited that article, you can find it here.In this blog, we will expand our exploration to categorical variables and investigate ways in which we can visualize and gain insights from them, in isolation and in combination with variables (both categorical and continuous).

Before we dive into the different graphs and plots, let’s define a categorical variable. In statistics, a categorical variable is one which has two or more categories, but there is no intrinsic ordering to them, for example, gender, color, cities, age group, etc. If there is some kind of ordering between the categories, the variables are classified as ordinal variables, for example, if you categorize car prices by cheap, moderate and expensive. Although these are categories, there is a clear ordering between the categories.

# Importing the necessary libraries.  
import numpy as np  
import pandas as pd  
import seaborn as sns  
import matplotlib.pyplot as plt  
%matplotlib inline  

We will be using the Adult data set, which is an extraction of the 1994 census dataset. The prediction task is to determine whether a person makes more than 50K a year. Hereis the link to the dataset. In this blog, we will be using the dataset only for data analysis.

# Since the dataset doesn't contain the column header, we need to specify it manually.   
cols = ['age', 'workclass', 'fnlwgt', 'education', 'education-num', 'marital-status', 'occupation', 'relationship', 'race', 'gender', 'capital-gain', 'capital-loss', 'hours-per-week', 'native-country', 'annual-income']  

# Importing dataset   
data = pd.read_csv('adult dataset/adult.data', names=cols)  

# The first five columns of the dataset.   
data.head()  

Bar graph

A bar chart or graph is a graph with rectangular bars or bins that are used to plot categorical values. Each bar in the graph represents a categorical variable and the height of the bar is proportional to the value represented by it.

Bar graphs are used:

• To make comparisons between variables
• To visualize any trend in the data, i.e., they show the dependence of one variable on another
• Estimate values of a variable

# Let's start by visualizing the distribution of gender in the dataset.  
fig, ax = plt.subplots()  
x = data.gender.unique()  
# Counting 'Males' and 'Females' in the dataset  
y = data.gender.value_counts()  
# Plotting the bar graph  
ax.bar(x, y)  
ax.set_xlabel('Gender')  
ax.set_ylabel('Count')  
plt.show()  

From the figure, we can infer that there are more number of males than females in the dataset. Next, we will use the bar graph to visualize the distribution of annual income based on both gender and hours per week (i.e. the number of hours they work per week).

# For this plot, we will be using the seaborn library as it provides more flexibility with dataframes.   
sns.barplot(data.gender, data['hours-per-week'], hue=data['annual-income'])  
plt.show()

So from the figure above, we can infer that males and females with annual income less than 50K tend to work more per week.

Countplot

This is a seaborn-specific function which is used to plot the count or frequency distribution of each unique observation in the categorical variable. It is similar to a histogram over a categorical rather than quantitative variable.

So, let’s plot the number of males and females in the dataset using the countplot function.

# Using Countplot to count number of males and females in the dataset.  
sns.countplot(data.gender)  
plt.show()  

Earlier, we plotted the same thing using a bar graph, and it required some external calculations on our part to do so. But we can do the same thing using the countplot function in just a single line of code. Next, we will see how we can use countplot for deeper insights.

# ‘hue’ is used to visualize the effect of an additional variable to the current distribution.  
sns.countplot(data.gender, hue=data['annual-income'])  
plt.show()  

From the figure above, we can count that number of males and females whose annual income is <=50 and > 50K. We can see that the approximate number of

• Males with annual income <=50K : 15,000
• Males with annual income > 50K: 7000
• Females with annual income <=50K: 9000
• Females with annual income > 50K: 1000

So, we can infer that out of 32,500 (approx) people, only 8000 people have income greater than 50K, out of which only 1000 of them are females.

Box plot

Box plots are widely used in data visualization. Box plots, also known as box and whisker plots are used to visualize variations and compare different categories in a given set of data. It doesn’t display the distribution in detail but is useful in detecting whether a distribution is skewed and detect outliers in the data. In a box and whisker plot:

• the box spans the interquartile range
• a vertical line inside the box represents the median
• two lines outside the box, the whiskers, extending to the highest and the lowest observations represent the possible outliers in the data

Let’s use a box and whisker plot to find a correlation between ‘hours-per-week’ and ‘relationship’ based on their annual income.

# Creating a box plot  
fig, ax = plt.subplots(figsize=(15, 8))  
sns.boxplot(x='relationship', y='hours-per-week', hue='annual-income', data=data, ax=ax)  
ax.set_title('Annual Income of people based on relationship and hours-per-week')  
plt.show()  

We can interpret some interesting results from the box plot. People with the same relationship status and an annual income more than 50K often work for more hours per week. Similarly, we can also infer that people who have a child and earn less than 50K tend to have more flexible working hours.
Apart from this, we can also detect outliers in the data. For example, people with relationship status ‘Not in family’ (see Fig 6.) and an income less than 50K have a large number of outliers at both the high and low ends. This also seems to be logically correct as a person who earns less than 50K annually may work more or less depending on the type of job and employment status.

Strip plot

Strip plot is a data analysis technique used to plot the sorted values of a variable along one axis. It is used to represent the distribution of a continuous variable with respect to the different levels of a categorical variable. For example, a strip plot can be used to show the distribution of the variable ‘gender’, i.e., males and females, with respect to the number of hours they work each week. A strip plot is also a good complement to a box plot or a violin plot in cases where you want to showcase all the observations along with some representation of the underlying distribution.

# Using Strip plot to visualize the data.  
fig, ax= plt.subplots(figsize=(10, 8))  
sns.stripplot(data['annual-income'], data['hours-per-week'], jitter=True, ax=ax)  
ax.set_title('Strip plot')  
plt.show()  

In the figure, by looking at the distribution of the data points, we can deduce that most of the people with an annual income greater than 50K work between 40 and 60 hours per week. While those with income less than 50K work can work between 0 and 60 hours per week.

Violin plot

Sometimes the mean and median may not be enough to understand the distribution of the variable in the dataset. The data may be clustered around the maximum or minimum with nothing in the middle. Box plots are a great way to summarize the statistical information related to the distribution of the data (through the interquartile range, mean, median), but they cannot be used to visualize the variations in the distributions.

A violin plot is a combination of a box plot and kernel density function (KDE, described in Part I of this blog series) which can be used to visualize the probability distribution of the data. Violin plots can be interpreted as follows:

• The outer layer shows the probability distribution of the data points and indicates 95% confidence interval. The thicker the layer, the higher the probability of the data points, and vice-versa.
• The second layer shows a box plot indicating the interquartile range.
• The third layer, or the dot, indicates the median of the data.

  

Let’s now build a violin plot. To start with, we will analyze the distribution of annual income of the people w.r.t. the number of hours they work per week.

fig, ax = plt.subplots(figsize=(10, 8))  
sns.violinplot(x='annual-income', y='hours-per-week', data=data, ax=ax)  
ax.set_title('Violin plot')  
plt.show()  

In Fig 9, the median number working hours per week is same (40 approximately) for both people earning less than 50K and greater than 50K. Although people earning less than 50K can have a varied range of the hours they spend working per week, most of the people who earn more than 50K work in the range of 40 – 80 hours per week.

Next, we can visualize the same distribution, but this grouping them according to their gender.

# Violin plot  
fig, ax = plt.subplots(figsize=(10, 8))  
sns.violinplot(x='annual-income', y='hours-per-week', hue='gender', data=data, ax=ax)  
ax.set_title('Violin plot grouped according to gender')  
plt.show()  

Adding the variable ‘gender’, gives us insights into how much each gender spends working per week based upon their annual income. From the figure, we can infer that males with annual income less than 50K tends to spend more hours working per week than females. But for people earning greater than 50K, both males and females spend an equal amount of hours per week working.

Violin plots, although more informative, are less frequently used in data visualization. It may be because they are hard to grasp and understand at first glance. But their ability to represent the variations in the data are making them popular among machine learning and data enthusiasts.

PairGrid

PairGrid is used to plot the pairwise relationship of all the variables in a dataset. This may seem to be similar to the pairplot we discussed in part I of this series. The difference is that instead of plotting all the plots automatically, as in the case of pairplot, Pair Grid creates a class instance, allowing us to map specific functions to the different sections of the grid.

Let’s start by defining the class.

# Creating an instance of the pair grid plot.  
g = sns.PairGrid(data=data, hue='annual-income')  

The variable ‘g’ here is a class instance. If we were to display ‘g’, then we will get a grid of empty plots. There are four grid sections to fill in a Pair Grid: upper triangle, lower triangle, the diagonal, and off-diagonal. To fill all the sections with the same plot, we can simply call ‘g.map’ with the type of plot and plot parameters.

# Creating a scatter plots for all pairs of variables.  
g = sns.PairGrid(data=data, hue='capital-gain')  
g.map(plt.scatter)  

The ‘g.map_lower’ method only fills the lower triangle of the grid while the ‘g.map_upper’ method only fills the upper triangle of the grid. Similarly, ‘g.map_diag’ and ‘g.map_offdiag’ fills the diagonal and off-diagonal of the grid, respectively.

#Here we plot scatter plot, histogram and violin plot using Pair grid.  
g = sns.PairGrid(data=data, vars = ['age', 'education-num', 'hours-per-week'])  
# with the help of the vars parameter we can select the variables between which we want the plot to be constructed.  

g.map_lower(plt.scatter, color='red')  
g.map_diag(plt.hist, bins=15)  
g.map_upper(sns.violinplot)  

Thus with the help of Pair Grid, we can visualize the relationship between the three variables (‘hours-per-week’, ‘education-num’ and ‘age’) using three different plots all in the same figure. Pair grid comes in handy when visualizing multiple plots in the same figure.

Conclusion

Let’s summarize what we learned. So, we started with visualizing the distribution of categorical variables in isolation. Then, we moved on to visualize the relationship between a categorical and a continuous variable. Finally, we explored visualizing relationships when more than two variables are involved. Next week, we will explore how we can visualize unstructured data. Finally, I encourage you to download the given census data (used in this blog) or any other dataset of your choice and play with all the variations of the plots learned in this blog. Till then, Adiós!

This is a series of blogs dedicated to different data visualization techniques used in various domains of machine learning. Data Visualization is a critical step for building a powerful and efficient machine learning model. It helps us to better understand the data, generate better insights for feature engineering, and, finally, make better decisions during modeling and training of the model.

For this blog, we will use the seaborn and matplotlib libraries to generate the visualizations. Matplotlib is a MATLAB-like plotting framework in python, while seaborn is a python visualization library based on matplotlib. It provides a high-level interface for producing statistical graphics. In this blog, we will explore different statistical graphical techniques that can help us in effectively interpreting and understanding the data. Although all the plots using the seaborn library can be built using the matplotlib library, we usually prefer the seaborn library because of its ability to handle DataFrames.

We will start by importing the two libraries. Here is the guide to installing the matplotlib library and seaborn library. (Note that I’ll be using matplotlib and seaborn libraries interchangeably depending on the plot.)

### Importing necessary library  
import random  
import numpy as np  
import pandas as pd  
import seaborn as sns  
import matplotlib.pyplot as plt  
%matplotlib inline  

Simple Plot

Let’s begin by plotting a simple line plot which is used to plot a mathematical. A line plot is used to plot the relationship or dependence of one variable on another. Say, we have two variables ‘x’ and ‘y’ with the following values:

x = np.array([ 0, 0.53, 1.05, 1.58, 2.11, 2.63, 3.16, 3.68, 4.21,  
        4.74, 5.26, 5.79, 6.32, 6.84])  
y = np.array([ 0, 0.51, 0.87, 1. , 0.86, 0.49, -0.02, -0.51, -0.88,  
        -1. , -0.85, -0.47, 0.04, 0.53])  

To plot the relationship between the two variables, we can simply call the plot function.

### Creating a figure to plot the graph.  
fig, ax = plt.subplots()  
ax.plot(x, y)  
ax.set_xlabel('X data')  
ax.set_ylabel('Y data')  
ax.set_title('Relationship between variables X and Y')  
plt.show() # display the graph  
### if %matplotlib inline has been invoked already, then plt.show() is automatically invoked and the plot is displayed in the same window.  

Here, we can see that the variables ‘x’ and ‘y’ have a sinusoidal relationship. Generally, .plot() function is used to find any mathematical relationship between the variables.

Histogram

A histogram is one of the most frequently used data visualization techniques in machine learning. It represents the distribution of a continuous variable over a given interval or period of time. Histograms plot the data by dividing it into intervals called ‘bins’. It is used to inspect the underlying frequency distribution (eg. Normal distribution), outliers, skewness, etc.

Let’s assume some data ‘x’ and analyze its distribution and other related features.

### Let 'x' be the data with 1000 random points.   
x = np.random.randn(1000)  

Let’s plot a histogram to analyze the distribution of ‘x’.

plt.hist(x)  
plt.xlabel('Intervals')  
plt.ylabel('Value')  
plt.title('Distribution of the variable x')  
plt.show()  

The above plot shows a normal distribution, i.e., the variable ‘x’ is normally distributed. We can also infer that the distribution is somewhat negatively skewed. We usually control the ‘bins’ parameters to produce a distribution with smooth boundaries. For example, if we set the number of ‘bins’ too low, say bins=5, then most of the values get accumulated in the same interval, and as a result they produce a distribution which is hard to predict.

plt.hist(x, bins=5)  
plt.xlabel('Intervals')  
plt.ylabel('Value')  
plt.title('Distribution of the variable x')  
plt.show()  

Similarly, if we increase the number of ‘bins’ to a high value, say bins=1000, each value will act as a separate bin, and as a result the distribution seems to be too random.

plt.hist(x, bins=1000)  
plt.xlabel('Intervals')  
plt.ylabel('Value')  
plt.title('Distribution of the variable x')  
plt.show()  

Kernel Density Function

Before we dive into understanding KDE, let’s understand what parametric and non-parametric data are.

Parametric Data: When the data is assumed to have been drawn from a particular distribution and some parametric test can be applied to it

Non-Parametric Data: When we have no knowledge about the population and the underlying distribution

Kernel Density Function is the non-parametric way of representing the probability distribution function of a random variable. It is used when the parametric distribution of the data doesn’t make much sense, and you want to avoid making assumptions about the data.

The kernel density estimator is the estimated pdf of a random variable. It is defined as

Similar to histograms, KDE plots the density of observations on one axis with height along the other axis.

### We will use the seaborn library to plot KDE.  
### Let's assume random data stored in variable 'x'.  
fig, ax = plt.subplots()  
### Generating random data  
x = np.random.rand(200)   
sns.kdeplot(x, shade=True, ax=ax)  
plt.show()  

Distplot combines the function of the histogram and the KDE plot into one figure.

### Generating a random sample  
x = np.random.random_sample(1000)  
### Plotting the distplot  
sns.distplot(x, bins=20)  

So, the distplot function plots the histogram and the KDE for the sample data in the same figure. You can tune the parameters of the displot to only display the histogram or kde or both. Distplot comes in handy when you want to visualize how close your assumption about the distribution of the data is to the actual distribution.

Scatter Plot

Scatter plots are used to determine the relationship between two variables. They show how much one variable is affected by another. It is the most commonly used data visualization technique and helps in drawing useful insights when comparing two variables. The relationship between two variables is called correlation. If the data points fit a line or curve with a positive slope, then the two variables are said to show positive correlation. If the line or curve has a negative slope, then the variables are said to have a negative correlation.

A perfect positive correlation has a value of 1 and a perfect negative correlation has a value of -1. The closer the value is to 1 or -1, the stronger the relationship between the variables. The closer the value is to 0, the weaker the correlation.

For our example, let’s define three variables ‘x’, ‘y’, and ‘z’, where ‘x’ and ‘z’ are randomly generated data and ‘y’ is defined as
We will use a scatter plot to find the relationship between the variables ‘x’ and ‘y’.

### Let's define the variables we want to find the relationship between.  
x = np.random.rand(500)  
z = np.random.rand(500)  
### Defining the variable 'y'  
y = x * (z + x)  
fig, ax = plt.subplots()  
ax.set_xlabel('X')  
ax.set_ylabel('Y')  
ax.set_title('Scatter plot between X and Y')  
plt.scatter(x, y, marker='.')  
plt.show()  

From the figure above we can see that the data points are very close to each other and also if we fit a curve, along with the points, it will have a positive slope. Therefore, we can infer that there is a strong positive correlation between the values of the variable ‘x’ and variable ‘y’.

Also, we can see that the curve that best fits the graph is quadratic in nature and this can be confirmed by looking at the definition of the variable ‘y’.

Joint Plot

Jointplot is seaborn library specific and can be used to quickly visualize and analyze the relationship between two variables and describe their individual distributions on the same plot.

Let’s start with using joint plot for producing the scatter plot.

### Defining the data.   
mean, covar = [0, 1], [[1, 0,], [0, 50]]  
### Drawing random samples from a multivariate normal distribution.  
### Two random variables are created, each containing 500 values, with the given mean and covariance.  
data = np.random.multivariate_normal(mean, covar, 500)  
### Storing the variables in a dataframe.  
df = pd.DataFrame(data=data, columns=['X', 'Y'])  

### Joint plot between X and Y  
sns.jointplot(df.X, df.Y, kind='scatter')  
plt.show()  

Next, we can use the joint point to find the best line or curve that fits the plot.

sns.jointplot(df.X, df.Y, kind='reg')  
plt.show()  

Apart from this, jointplot can also be used to plot ‘kde’, ‘hex plot’, and ‘residual plot’.

PairPlot

We can use scatter plot to plot the relationship between two variables. But what if the dataset has more than two variables (which is quite often the case), it can be a tedious task to visualize the relationship between each variable with the other variables.

The seaborn pairplot function does the same thing for us and in just one line of code. It is used to plot multiple pairwise bivariate (two variable) distribution in a dataset. It creates a matrix and plots the relationship for each pair of columns. It also draws a univariate distribution for each variable on the diagonal axes.

### Loading a dataset from the sklearn toy datasets  
from sklearn.datasets import load_linnerud  
### Loading the data  
linnerud_data = load_linnerud()  
### Extracting the column data  
data = linnerud_data.data  

Sklearn stores data in the form of a numpy array and not data frames, thereby storing the data in a dataframe.

### Creating a dataframe  
data = pd.DataFrame(data=data, columns=diabetes_data.feature_names)  
### Plotting a pairplot  
sns.pairplot(data=data)  

So, in the graph above, we can see the relationships between each of the variables with the other and thus infer which variables are most correlated.

Conclusion

Visualizations play an important role in data analysis and exploration. In this blog, we got introduced to different kinds of plots used for data analysis of continuous variables. Next week, we will explore the various data visualization techniques that can be applied to categorical variables or variables with discrete values. Next, I encourage you to download the iris dataset or any other dataset of your choice and apply and explore the techniques learned in this blog.

Have anything to say? Feel free to comment below for any questions, suggestions, and discussions related to this article. Till then, Sayōnara.

The meteoric rise of artificial intelligence in the last decade has spurred a huge demand for AI and ML skills in today’s job market. ML-based technology is now used in almost every industry vertical from finance to healthcare. In this blog, we have compiled a list of best frameworks and libraries that you can use to build machine learning models.

1) TensorFlow
Developed by Google, TensorFlow is an open-source software library built for deep learning or artificial neural networks. With TensorFlow, you can create neural networks and computation models using flowgraphs. It is one of the most well-maintained and popular open-source libraries available for deep learning. The TensorFlow framework is available in C++ and Python. Other similar deep learning frameworks that are based on Python include Theano, Torch, Lasagne, Blocks, MXNet, PyTorch, and Caffe. You can use TensorBoard for easy visualization and see the computation pipeline. Its flexible architecture allows you to deploy easily on different kinds of devices
On the negative side, TensorFlow does not have symbolic loops and does not support distributed learning. Further, it does not support Windows.

2)Theano
Theano is a Python library designed for deep learning. Using the tool, you can define and evaluate mathematical expressions including multi-dimensional arrays. Optimized for GPU, the tool comes with features including integration with NumPy, dynamic C code generation, and symbolic differentiation. However, to get a high level of abstraction, the tool will have to be used with other libraries such as Keras, Lasagne, and Blocks. The tool supports platforms such as Linux, Mac OS X, and Windows.

3) Torch
The Torch is an easy to use open-source computing framework for ML algorithms. The tool offers an efficient GPU support, N-dimensional array, numeric optimization routines, linear algebra routines, and routines for indexing, slicing, and transposing. Based on a scripting language called Lua, the tool comes with an ample number of pre-trained models. This flexible and efficient ML research tool supports major platforms such as Linux, Android, Mac OS X, iOS, and Windows.

4) Caffe
Caffe is a popular deep learning tool designed for building apps. Created by Yangqing Jia for a project during his Ph.D. at UC Berkeley, the tool has a good Matlab/C++/ Python interface. The tool allows you to quickly apply neural networks to the problem using text, without writing code. Caffe partially supports multi-GPU training. The tool supports operating systems such as Ubuntu, Mac OS X, and Windows.

5) Microsoft CNTK
Microsoft cognitive toolkit is one of the fastest deep learning frameworks with C#/C++/Python interface support. The open-source framework comes with powerful C++ API and is faster and more accurate than TensorFlow. The tool also supports distributed learning with built-in data readers. It supports algorithms such as Feed Forward, CNN, RNN, LSTM, and Sequence-to-Sequence. The tool supports Windows and Linux.

6) Keras
Written in Python, Keras is an open-source library designed to make the creation of new DL models easy. This high-level neural network API can be run on top of deep learning frameworks like TensorFlow, Microsoft CNTK, etc. Known for its user-friendliness and modularity, the tool is ideal for fast prototyping. The tool is optimized for both CPU and GPU.



7) SciKit-Learn
SciKit-Learn is an open-source Python library designed for machine learning. The tool based on libraries such as NumPy, SciPy, and matplotlib can be used for data mining and data analysis. SciKit-Learn is equipped with a variety of ML models including linear and logistic regressors, SVM classifiers, and random forests. The tool can be used for multiple ML tasks such as classification, regression, and clustering. The tool supports operating systems like Windows and Linux. On the downside, it is not very efficient with GPU.

8)Accord.NET
Written in C#, Accord.NET is an ML framework designed for building production-grade computer vision, computer audition, signal processing and statistics applications. It is a well-documented ML framework that makes audio and image processing easy. The tool can be used for numerical optimization, artificial neural networks, and visualization. It supports Windows.

9)Spark MLIib
Apache Spark’s MLIib is an ML library that can be used in Java, Scala, Python, and R. Designed for processing large-scale data, this powerful library comes with many algorithms and utilities such as classification, regression, and clustering. The tool interoperates with NumPy in Python and R libraries. It can be easily plugged into Hadoop workflows.

10) Azure ML Studio
Azure ML studio is a modern cloud platform for data scientists. It can be used to develop ML models in the cloud. With a wide range of modeling options and algorithms, Azure is ideal for building larger ML models. The service provides 10GB of storage space per account. It can be used with R and Python programs.

11) Amazon Machine Learning
Amazon Machine Learning (AML) is an ML service that provides tools and wizards for creating ML models. With visual aids and easy-to-use analytics, AML aims to make ML more accessible to developers. AML can be connected to data stored in Amazon S3, Redshift, or RDS.

Machine learning frameworks come with pre-built components that are easy to understand and code. A good ML framework thus reduces the complexity of defining ML models. With these open-source ML frameworks, you build your ML models easily and quickly.

Know an ML framework that should be on this list? Share them in comments below.

Data visualization is helping companies worldwide to identify patterns, predict outcomes, and improve business returns. Visualization is an important aspect of data analysis. Simply put, data visualization conveys outcomes of tabular or spatial data in a visual format. Images have the power to capture attention and convey ideas clearly. This aids decision making and drives action for improvements.

With the use of the right tools, you can sketch a convincing visual story from your raw data. Here are some free and open source tools for data visualization:

1) Candela

If you know Javascript, then you can use this open source tool to make rich data visualizations. Candela is an open-source suite of interoperable web visualization components.

2) Charted

Charted is a free data visualization tool that lets you create line graphs or bar charts from CSV files and Google spreadsheets. The toll comes with integrated components including LineUp component, UpSet component, OnSet component, Vega visualizations, and GeoJS geospatial visualizations. The tool does not store the data or manipulate it. Focused purely on visualization, it comes with basic features to create a line or stacked charts with labels and notes.

3) Datawrapper

Datawrapper is a mobile-friendly data visualization tool that lets you create charts and reports in seconds. The free version of the tool meant for a single user supports 10,000 monthly chart views. Using the tool, you can create different types of visualization such as bar chart, split chart, stacked chart, dot plot, arrow plot, area chart, scatter plot, symbol map, and choropleth map. You don’t need coding or designing skills to use the tool.

4) Google Data Studio

Google’s data visualization tool is free and easy to set up if you have a Gmail account. You can connect it easily with Google products such as Google AdWords, Google Analytics, YouTube Analytics, and Google Sheets.

5) Google Charts

Another simple and free data visualization tool by Google is the Google chart tool. The tool comes with interactive charts and data tools for visualization.

6) Leaflet

The leaflet is an open-source JavaScript library that allows you to make mobile-friendly interactive maps. The tool has a lot of plugins for adding features and works well on various desktop and mobile platforms.

7) MyHeatMap

MyHeatMap is a free tool to view your geographic data interactively. The free version of the tool offers only public maps and you can add only 20 data points for each of those free maps. The tool makes it easy to understand the data with color-coded heat maps. You can also switch between data sets within the same map.

8) Openheatmap

This free tool lets you turn your spreadsheet into a map. You can upload your CSV file or Google sheet to create an interactive online map in seconds. The tool can be used to explain data like customer demographics by zip codes.

9) Palladio

Palladio is a free tool designed to visualize complex historical data. It comes with features like map view, graph view, list view and gallery view. You can use the tool to visualize data in CSV, TAB, or TSV files. With the graph view, you can visualize the relationship between dimensions of your data. The data is displayed as nodes connected by lines. The list view, on the other hand, allows you to arrange data to make customized lists. The tool also has a gallery view to display data within a grid.

10) RawGraphs

RawGraphs is an open-source platform that helps you visualize TSV, CSV, DSV, or JSON data. The free tool is simple to use and helps in converting data to charts.

11) Tableau Public

Tableau Public is a free business intelligence tool that allows users to create and share interactive charts, graphs, maps, and app. The free version of the tool comes with 10 GB of storage. You can connect it to data sources like Google Sheets, Microsoft Excel, Text files, JSON files, Spatial files, Web Data Connectors, OData, and statistical files such as SAS (*.sas7bdat), SPSS (*.sav), and R (*.rdata, *.rda).

12) Timeline

A timeline is a free tool that allows you to create timelines for reports. You can connect your Google Drive account to create a timeline from Google Spreadsheet using the templates given in the tool. Using JSON, you can create custom installations.

13) Chartist.js

Chartist.js is a free data visualization that allows you to create responsive charts fast and easy.
The tool offers great flexibility and is customizable. You can even use CSS animations and transitions to your SVG elements.

14) ColorBrewer

ColorBrewer is a free tool that can be used to make your maps better in terms of color schemes. The tool makes it easy to differentiate colors on a complex map.

15) D3.JS

D3.JS is a free JavaScript library that helps you create images using data. The tool enables you to connect arbitrary data to a Document Object Model (DOM), and then apply data-driven transformations to the document. With DOM programming API, programmers can access documents as objects.

16) Plotly

Plotly is an open-source tool that allows you to compose, edit and share interactive data visualizations. You can use the tool to create D3.js charts and maps by uploading CSV files or connecting to the SQL database. You can also create charts with R or Python.

17) Polymaps

Polymaps is a free javascript library for creating dynamic, interactive maps in browsers. You can use the tool to get a display of multi-zoom datasets over maps. The tool uses scalable vector graphics (SVG) to display images, thus enabling you to define the design using CSS.

18) Weave

The Weave is a free data visualization platform that is ADA-compliant. The tool comes with a full keyboard and assistive device navigation and complete screen reader support. The tool also automatically gives descriptions of the images in real time.

19) Dygraphs

Dygraphs is an open-source charting library based on JavaScript. This free tool can be used to analyze dense data sets. The tool is highly customizable and works well in all browsers. The tool offers strong support for error bars/ confidence intervals.

Dygraphs

20) GanttPro

Apart from these, there are many data visualization tools that offer a free trial for a limited time GanttPro, a project management tool, for instance, helps you create charts for projects for free during their 15-day trial period.

Data visualization is crucial for accurate data analysis. With the right tools in hand, you can easily summarize and explain complex data to your stakeholders. By leveraging actionable insights generated from data, companies can make big profits and savings. Just how big are we talking about? Netflix saved around $1 billion in 2017 with its ML algorithm that recommends personalized TV shows and movies to subscribers. When used right, data analysis and visualization have the power to change the way people live their lives.

Know a great open source tool for data visualization? Share it in comments below.

“We didn’t do anything wrong, but somehow, we lost,” said Nokia’s CEO Stephen Elop in his speech soon after Nokia’s announcement of being acquired by Microsoft in September 2013. The mobile giant that was once valued at $222 billion at its peak was acquired for just $7.2 billion by Microsoft that year. Failure to adapt to new trends in smartphone technology drove the legendary mobile company from market domination to sell-off.

The biggest lesson for from Nokia’s collapse in the mobile industry is to adapt to new tech trends before your consumers abandon you. Fast innovation is the key to keeping up with new technologies. By discerning the latest trends, companies can leverage the right technology and adapt in time to succeed. Here are the top tech trends that will define the IT landscape in 2018:

1) The incredible AI

Artificial intelligence (AI) is going to get better and smarter in 2018. Thanks to artificial intelligence, your everyday appliances from the security system to entertainment console will become more automated and smarter. Using software algorithms and sensors, artificial intelligence based devices will be able to do more complex actions and will play a significant role in several domains including healthcare, smart cars, and personal security. From making the financial sector more accurate and secure to powering smart personal assistants like Siri, you will see more of artificial intelligence based technology in your everyday life.

2) Planet of the robots

With more research and investment in robotics, you will also see more of intelligent drones and robots in 2018. According to IDC’s FutureScape: Worldwide Robotics 2017 Predictions report, 45% of the 200 leading global e-commerce and omnichannel commerce companies will deploy robotics systems in their order fulfillment, warehousing, and delivery operations. 2018 is going to witness the emergence of robotics in fields including agriculture, manufacturing, and medicine. Lifelike robots like Sophia could even play the role of human companions and help take care of children, the elderly and people with special needs. Sophia who looks like the late actor Audrey Hepburn was even awarded full citizenship of Saudi Arabia. Those worried about a robotic invasion can put their fears to rest. Robotics of the future will be designed to help people achieve more.

3) Mixed reality: The rise of immersive experience

With the digital revolution, today we have access to more content than ever before. Earlier, augmented reality (AR) and virtual reality (VR) defined the way people interacted with the digital world. The coming days will see the emergence of mixed reality (MR) that combines both augmented reality and virtual reality. The mixed reality technology enables users to interact in an environment where both physical and digital objects co-exist. 2018 will witness more application of the mixed reality technology in fields ranging from simulation-based learning, military training, aviation, healthcare to interactive product content management. According to the research firm Reportbuyer.com, the global mixed reality market size is expected to touch $2.8 billion by 2023.

4) Blockchain: The force awakens

The blockchain is the fundamental technology behind cryptocurrencies like bitcoin. The exponential rise of bitcoins has rekindled everyone’s interest in blockchain technology. The technology provides a secure way of sharing encrypted data on anything, from money to medical records, between companies, people, and institutions. Blockchain has the potential to revolutionize the financial sector and the world economy. In 2018, companies are going to invest heavily in developing their blockchain and fintech capabilities. As we move toward a digital, technologically advanced financial world, blockchain will play a crucial role in making our financial systems faster, more secure and efficient.

5) The age of machine learning

Businesses will increasingly leverage on machine learning to gain a competitive advantage in 2018. Machine learning deals with the technology that enables computers to learn explicitly without being programmed. The technology can be used to analyze large volumes of data and predict patterns. In the coming days, machine learning will be widely used in fields including data security, personal security, financial trading, healthcare, personalized marketing and online search.

6) IoT: Unchained

IoT is here to stay and thrive. According to Business Insider, the business spends on IoT solutions will reach $6 trillion by 2021. Internet of Things (IoT) is a network of devices that is embedded with software and sensors that enable these devices to connect and exchange data. The technology is applied in verticals including wearables, connected cars, connected homes, connected cities and industrial internet. 2018 will see the rise of “digital twins,” the next step in IoT-based technology. Companies will rely on the technology to predict problems through data analysis and simulations. Another disruptive technology that will emerge will be based on a combination of IoT and Blockchain technologies. The technology will be applied in domains including warehousing, healthcare, and financial sectors. There will be connected devices everywhere.

As the waves of new and emerging technologies hit the world, it is a good idea to start learning more about these new and upcoming tech domains. In today’s high-tech era, leveraging the right technology can mean the difference between success and failure. Consumers are quick to punish those who don’t innovate fast.

Nokia did not do anything wrong except miss out on the latest trends in mobile technology. While the competitors quickly caught on the demand for smarter phones, Nokia was left far behind. In the end, it is the lack of innovation and the failure to adapt to emerging technologies that force companies out of business. The lesson for everyone is to adapt and innovate before it is too late.