Developer Insights

With the beginning of new year, every startup’s open mailing list starts receiving emails from graduate students for summer internship. I have read hundreds of applications that follow a generic template, somewhat like this:

Dear Sir/Ma'am,

My name is and I am currently pursuing my in from . I am well-versed with C/C++ and have started learning Java. I’m also learning about Android and iOS apps. I have good knowledge of HTML, CSS, JS too.

If given chance, I’ll give my 100% at work. I am confident that I will be a valuable asset to your team.

Please find the attachment of my CV

Kind regards,

Seems like a decent email, however, there’s one major red flag about this application. The applicant is not talking anything about the startup he is applying to and how he will benefit them. This shows that the applicant has not researched about the company's business and engineering. So, how to write an application email for Software Engineering Internships at startups?

Find the right point of contact

• Try to address the right person in the company. When you begin with “Hi” or “Hey there” or “Sir/Ma’am” you’re diffusing the responsibility of a reply, a lot of emails are lost in the haystack because everybody in the group thinks that it is not meant for them or someone else will reply. Connect with someone in the company and ask them for the right person to contact for the purpose. Address them in your application email.

Keep the intro short

• Keep the subject line and introduction short. Begin with your name, major and institute. That’d be all. Next you should be talking about what you know about the company and how you both can benefit each other.

Talk more about what you can give to the company, less about what you’ll take away

• Research about the company’s engineering. Many tech companies have engineering blogs. You’ll know what technologies they use. Now you have what technologies you should highlight in the application, only if you know them well.
• Check out the open source projects of the company. Every good tech company knows what engineering tasks they are going to accomplish in the upcoming year. You can contribute to their public repositories. And then highlight this in your application.
• Use the product or services offered by the company. Find all sorts of possible improvements and suggest solutions for them in your application email. Every user out there is pointing out problems to them via support tickets, so suggesting solutions for them is an important part.

Sure that’s going to make your application a little longer than usual, however, it will make you stand out among the crowd. Ideally, such an application should be addressed to one of the engineers at the company.

Be objective about why you’d be a good addition to the team

Terms like “I’ll be a good addition to the team”, “Fast learner” etc. are subjective in nature. You have to prove it objectively. This is why many companies have started giving away take-home projects. Candidates can work on it for a week and depending on the work accomplished, companies get a good idea of how the candidate will perform during the internship. Some companies think that the candidate can cheat by asking someone else to do the take-home project. So they keep the candidate for a week-long project on trial. To overcome all of this, you, as a candidate, can think of features or apps that you can build which could be of value to the company. You can also make use of any APIs exposed by the company. You can then highlight such contributions to the company in your application, which shows that you’re reliable because you have a good past record.

Have an online presence

LinkedIn and AngelList are extensively used by Talent Acquisition team because they are quite familiar with their profile format and features. A Github account with few repositories and regular commits helps you in two ways: It shows that 1. You know how to use git 2. You’re consistent at work. Engineers may check your code to judge you by its quality.

You may also create a portfolio for yourself. You can extensively highlight all your projects in the portfolio. You can also write tech blog posts about challenges you faced during some project and how you solved them. By thy way, if you’re good at communication (written or verbal) you will do well in any job in the world. And your application email is one of the things by which every company is going to judge your communication skills.

Résumé

Lastly, prepare a short résumé that highlights your strengths and modify it for the target company. The resume format suggested by careercup seems adequate. However, it is suggested to apply your creativity to make it stand out. People don’t spend more than 30 seconds to go through any resume. Make sure you make those 30 seconds count.

All of the above points are “DOs”, let’s have a look at few of the “DON'Ts”:

• Don’t send blind emails in bulk in `to` or `cc` or `bcc` expecting at least one of them to respond. Target few companies and write personalized emails.
• Don’t write a subject line longer than 5-8 words.
• Don’t share your email format with your friend who is going to send the same email to the company by replacing a few things. You both will be rejected.
• Don’t mention all the technologies you know, a company is not interested in what you know, they are interested in what you know that they use. Do your research well.
• Don’t include subjective statements which cannot be proved like “I always give my 100%” and similar statements.
• Don’t send reminder emails on your application email frequently. Give it a week for them to respond. They receive hundreds of emails every day.
• Don’t say that “Review my CV and match it to open roles in your company”. It is candidate’s duty to target a particular position.

The startup hiring process includes resume filtering, phone screening, face-to-face interviews (plus take-home projects in some cases) and final interview with founders. Many good candidates are not able to get past the first hurdle. To all those candidates, next time you apply for an internship, use this post as a checklist. All the best!Till next time. Evíva!

Variation in gradient descent with learning rate-

Summary

In this article, we learned about the basics of gradient descent algorithm and its types. These optimization algorithms are being widely used in neural networks these days. Hence, it's important to learn. The image below shows a quick comparison in all 3 types of gradient descent algorithms:

Every time you try to create a new account on any of the websites, you begin with your name and, more often than not, you get the response “Username is already taken.”

Then, you add “your name + date of birth”, to realize it also has been “already taken” to finally end up with “your name + date of birth + license plate + graduation” to create the account.

I’m sure a lot of you are nodding and saying “been there, done that.”

But how many of you have wondered how these giant sites like Facebook, Instagram, and Gmail verify whether the username is taken or not?

Let’s start with the two possible approaches

A linear search may not be a good idea

Let’s assume that Facebook stores all the data in its directory.

And the software simply checks each name on the list one at a time and if it doesn’t find a match, it tells you your desired username is available.

Doesn’t sound sensible, does it?

The software has to look at each name every time a username needs to be verified.

The technique is unreasonable when you compare it with the Facebook database, which has over 1.5 billion users, and Twitter, which has 300 million users.

What if they use a Binary search?

This makes more sense, with all the brains working at Facebook.

Facebook keeps all the data sorted and arranged in an alphabetized list.

The list is 1.5 billion characters long, stored like a, aa,aaa……xyy, XYZ, yaa,yaa,yxz, zaa, zac and is very similar to your dictionary.

When you enter a name, it matches the entry with the username exactly in the middle of the list. If it matches, the software rejects the new username.

If it doesn’t match (which has a lot of possibilities), the next question the software addresses are “ If searched alphabetically, does the requested username come before or after this username in the middle?”

If it comes before, then the software knows that all the 750 million people after the username found in the middle of the list is of no use for the current search.

That eliminates 750 million possibilities in a single comparison.

If the desired username comes after the name in the middle (alphabetically), it eliminates all the names before it.

Either way, the software eliminates almost 750 million names for search in the first comparison.

Next, it takes the selected half of the list and immediately matches the requested username with the name in the middle of the remaining list.

If it matches, the requested name is rejected and if it doesn’t, the requested name is again checked for the possibility of it occurring before or after the name in the middle.

If it is before, reject the 350 million names after the name.

And go ahead with divide and conquer for the rest of the names as done earlier.

If the requested name is after the middle string, reject the names before it and try with the 350 million remaining names.

By dividing the list every time, you can compare the required username with the names in the list quite quickly…

But the question is…how quickly?

You will continue dividing the list into two until you can no longer do so.

And when you are left with one name in the database, you match it with your desired username.

This would be the last step before you find whether the username “chosen” is available or not.

For data as big as 1.5 billion, this method would need no more than 30 steps. 2 to the power of 31 gives you 2.14 billion, which is closest to our expectation of 1.5 billion users on Facebook.

This means fewer steps and complications for the same data when searched with a linear search.

What if the developers are very smart and use a Bloom filter as the solution?

Before you understand Bloom filters, you need to understand the concept of Hashing.

Hashing is like the license plate of your car.

A hash function takes data of any length as input and gives you a smaller identifier of a smaller, fixed length, which is often used to identify or compare data.

Bloom filters work simply – Test and Add.

Test whether the element is present in the list:

• If it returns false, the element is definitely not on the list.
• If it returns true, the element might probably be on the list. This false positive (will discuss it below) is a function of the Bloom filter and depends on the size and is independent of the hash function used.

A Bloom filter divides the memory area into buckets, which are filled with various hashes generated from one or many hash functions.

Let’s understand with an example.

Suppose, you have a memory bucket of size 10 and 3 hash functions which will give you three unique identifiers.

Suppose, you enter Ronaldo into this memory bucket.

Ronaldo, when passed through these hash functions, gives the value of 1,4, and 5. The filter quickly fills the memory in the bucket with these identifiers.

1

4

5

Now, you enter Messi into the memory bucket. Messi, when passed through the hash function, gives its own unique identifier. In this case, it is 3,7, and 8 and the filter fills the bucket.

1

3

4

5

7

8

As the functions always return the same value for similar inputs, we can be sure that when the name Ronaldo is given to the filter, it would check in locations 1,4, and 5 to find it full, which means that the name Ronaldo is already on the list.

Let’s continue with another example of entering Rooney into the memory.

Rooney, when passed through the hash function, returns 2,6, and 8. The filters check the memory to find that though 8 is full 2 and 6 are empty, which means you don’t have Rooney in the memory.

Therefore, the name is available.

But when the name Neymar is passed through the hash functions, it returns the value of 1,3 and 7 which eventually makes the filter believe that the name Neymar is already present on the list.

This scenario explains the concept of false positives used in Bloom filters. One can control the false positive by controlling the size of the Bloom filter.

More space is inversely proportional to false positives.

Each of the above-mentioned techniques comes with its own advantages and disadvantages.

With technology and computers getting smarter and faster every day, even the brute-force method seems feasible.

But with space and time complexity, many companies, such as Reddit, prefer Binary search, whereas some others, such as Medium, use Bloom filters smartly to suggest articles for you without repeating them again on your timeline.

Register now before your username is taken on the HackerEarth platform.

Introduction

Machine Learning is tricky. No matter how many books you read, tutorials you finish or problems you solve, there will always be a data set you might come across where you get clueless. Specially, when you are in your early days of Machine Learning. Isn’t it ?

In this blog post, you’ll learn some essential tips on building machine learning models which most people learn with experience.These tips were shared by Marios Michailidis(a.k.a Kazanova), Kaggle Grandmaster, Current Rank #3 in a webinar happened on 5th March 2016. The webinar had three aspects:

1. Video – Watch Here.
2. Slides – Slides used in the video were shared by Marios. Indeed, an enriching compilation of machine learning knowledge. Below are the slides.
3. Q & As – This blog enlists all the questions asked by participants at webinar.

The key to succeeding in competitions is perseverance. Marios said, ‘I won my first competition (Acquired valued shoppers challenge) and entered kaggle’s top 20 after a year of continued participation on 4 GB RAM laptop (i3)’.Were you planning to give up ?

While reading Q & As, if you have any questions, please feel free to drop them in comments!

Questions & Answers

1. What are the steps you follow for solving a ML problem? Please describe from scratch.

Following are the steps I undertake while solving any ML problem:

1. Understand the data – After you download the data, start exploring features. Look at data types. Check variable classes. Create some univariate – bivariate plots to understand the nature of variables.
2. Understand the metric to optimize – Every problem comes with a unique evaluation metric. It’s imperative for you to understand it, specially how does it change with target variable.
3. Decide cross validation strategy – To avoid overfitting, make sure you’ve set up a cross validation strategy in early stages. A nice CV strategy willhelp you get reliable score on leaderboard.
4. Start hyper parameter tuning– Once CV is at place, try improving model’s accuracy using hyper parameter tuning. It further includes the following steps:
   • Data transformations: It involve steps like scaling, removing outliers, treating null values, transform categorical variables, do feature selections, create interactions etc.
   • Choosing algorithms and tuning their hyper parameters: Try multiple algorithms to understand how model performance changes.
   • Saving results: From all the models trained above, make sure you save their predictions. They will be useful for ensembling.
   • Combining models: At last, ensemble the models, possibly on multiple levels. Make sure the models are correlated for best results.

2. What are the model selection and data manipulation techniques you follow to solve a problem?

Generally, I try (almost) everything for most problems. In principle for:

• Time series: I use GARCH, ARCH, regression, ARIMA models etc.
• Image classification: I use deep learning (convolutional nets) in python.
• Sound Classification :Common neural networks
• High cardinality categorical (like text data): I use linear models, FTRL, Vowpal wabbit, LibFFM, libFM, SVD etc.

For everything else,I use Gradient boosting machines (like XGBoost and LightGBM) and deep learning (like keras, Lasagne, caffe, Cxxnet). I decide what model to keep/drop in Meta modelling with feature selection techniques.Some of the feature selection techniques I use includes:

• Forward (cv or not) – Start from null model. Add one feature at a time and check CV accuracy. If it improves keep the variable, else discard.
• Backward (cv or not) – Start from full model and remove variables one by one. It CV accuracy improves by removing any variable, discard it.
• Mixed (or stepwise) – Use a mix of above to techniques.
• Permutations
• Using feature importance – Use random forest, gbm, xgboost feature selection feature.
• Apply some stats’ logic such as chi-square test, anova.

Data manipulation could be different for every problem :

• Time series : You can calculate moving averages, derivatives. Remove outliers.
• Text : Useful techniques are tfidf, countvectorizers, word2vec, svd (dimensionality reduction). Stemming, spell checking, sparse matrices, likelihood encoding, one hot encoding (or dummies), hashing.
• Image classification: Here you can do scaling, resizing, removing noise (smoothening), annotating etc
• Sounds : Calculate Furrier Transforms , MFCC (Mel frequency cepstral coefficients), Low pass filters etc
• Everything else : Univariate feature transformations (like log +1 for numerical data), feature selections, treating null values, removing outliers, converting categorical variables to numeric.

3. Can you elaborate cross validation strategy?

Cross validation means that from my main set, I create RANDOMLY 2 sets. I built (train) my algorithm with the first one (let’s call it training set) and score the other (let’s call it validation set). I repeat this process multiple times and always check how my model performs on the test set in respect to the metric I want to optimize.

The process may look like:

• For 10 (you choose how many X) times
• Split the set in training (50%-90% of the original data)
• And validation (50%-10% of the original data)
• Then fit the algorithm on the training set
• Score the validation set.
• Save the result of that scoring in respect to the chosen metric.
• Calculate the average of these 10 (X) times. That how much you expect this score in real life and is generally a good estimate.
• Remember to use a SEED to be able to replicate these X splits

Other things to consider is Kfold and stratified KFold . Read here.For time sensitive data, make certain you always the rule of having past predicting future when testing’s.

4. Can you please explain sometechniques usedfor cross validation?

• Kfold
• Stratified Kfold
• Random X% split
• Time based split
• For large data, just one validation set could suffice (like 20% of the data – you don’t need to do multiple times).

5. How did you improve your skills in machine learning? What training strategy did you use?

I did a mix of stuff in 2. Plus a lot of self-research. Alongside,programming and software (in java) and A LOT of Kaggling ☺

6. Which are the most useful python libraries for a data scientist ?

Below are some libraries which I find most useful in solving problems:

• Data Manipulation
  • Numpy
  • Scipy
  • Pandas
• Data Visualization
  • Matplotlib
• Machine Learning / Deep Learning
  • Xgboost
  • Keras
  • Nolearn
  • Gensim
  • Scikit image
• Natural Language Processing
  • NLTK

7. What are useful ML techniques / strategies to impute missing values or predict categorical label when all the variables are categorical in nature.

Imputing missing values is a critical step. Sometimes you may find a trend in missing values. Below are some techniques I use:

• Use mean, mode, median for imputation
• Use a value outside the range of the normal values for a variable. like -1 ,or -9999 etc.
• Replace witha likelihood – e.g. something that relates to the target variable.
• Replace with something which makes sense. For example: sometimes null may mean zero
  • Try to predict missing values based on subsets of know values
  • You may consider removing rows with many null values

8. Can you elaborate what kind of hardware investment you have done i.e. your own PC/GPU setup for Deep learning related tasks? Or were you using more cloud based GPU services?

I won my first competition (Acquired valued shoppers challenge) and entered kaggle’s top 20 after a year of continued participation on 4 GB RAM laptop (i3). I was using mostly self-made solutions up to this point (in Java). That competition it had something like 300,000,000 rows of data of transactions you had to aggregate so I had to parse the data and be smart to keep memory usage at a minimum.

However since then I made some good investments to become Rank #1. Now, I have access to linux servers of 32 cores and 256 GBM of RAM. I also have a geforce 670 machine (for deep learning /gpu tasks) . Also, I use mostly Python now. You can consider Amazon’s AWS too, however this is mostly if you are really interested in getting to the top, because the cost may be high if you use it a lot.

9. Do you use high performing machine like GPU. or for example do you do thing like grid search for parameters for random forest(say), which takes lot of time, so which machine do you use?

I use GPUs (not very fast, like a geforce 670) for every deep learning training model. I have to state that for deep learning GPU is a MUST. Training neural nets on CPUs takes ages, while a mediocre GPU can make a simple nn (e.g deep learning) 50-70 times faster. I don’t like grid search. I do this fairly manually. I think in the beginning it might be slow, but after a while you can get to decent solutions with the first set of parameters! That is because you can sort of learn which parameters are best for each problem and you get to know the algorithms better this way.

10. How do people built around 80+ models is it by changing the hyper parameter tuning ?

It takes time. Some people do it differently. I have some sets of params that worked in the past and I initialize with these values and then I start adjusting them based on the problem at hand. Obviously you need to forcefully explore more areas (of hyper params in order to know how they work) and enrich this bank of past successful hyper parameter combinations for each model. You should consider what others are doing too. There is NO only 1 optimal set of hyper params. It is possible you get a similar score with a completely different set of params than the one you have.

11. How does one improve their kaggle rank? Sometimes I feel hopeless while working on any competition.

It’s not an overnight process. Improvement on kaggle or anywhere happens with time. There are no shortcuts. You need to just keep doing things. Below are some of the my recommendations:

• Learn better programming: Learn python if you know R.
• Keep learning tools (listed below)
• Read some books.
• Play in ‘knowledge’ competitions
• See what the others are doing in kernels or in past competitions look for the ‘winning solution sections’
• Team up with more experience users, but you need to improve your ranking slightly before this happens
• Create a code bank
• Play … a lot!

12. Can you tellus about some usefultools used in machine learning ?

Below is the list of my favourite tools:

• Liblinear : For linear models
• LibSvm for Support Vector machines
• Scikit Learn for all machine learning models
• Xgboost for fast scalable gradient boosting
• LightGBM
• Vowpal Wabbit for fast memory efficient linear models
• http://www.heatonresearch.com/encog encog for neural nets
• H2O in R for many models
• LibFm
• LibFFM
• Weka in Java (has everything)
• Graphchi for factorizations
• GraphLab for lots of stuff
• Cxxnet : One of the best implementation of convolutional neural nets out there. Difficult to install and requires GPU with NVDIA Graphics card.
• RankLib: The best library out there made in java suited for ranking algorithms (e.g. rank products for customers) that supports optimization fucntions like NDCG.
• Kerasand Lasagnefor neural nets. This assumes you have Theanoor Tensorflow.

13. How to start with machine learning?

I like these slides from the university of utah in terms of understanding some basic algorithms and concepts about machine learning. This book for python. I like this book too. Don’t forget to follow the wonderful scikit learn documentation. Use jupyter notebook from anaconda.

You can find many good links that have helped me in kaggle here. Look at ‘How Did you Get Better at Kaggle’

In addition, you should do Andrew Ng’s machine learning course. Alongside, you can follow some good blogs such as mlwave, fastml, analyticsvidhya. But the best way is to get your hands dirty. do some kaggle! tackle competitions that have the “knowledge” flag first and then start tackling some of the main ones. Try to tackle some older ones too.

14. What techniques perform best on large data sets on Kaggle and in general ? How to tackle memory issues ?

Big data sets with high cardinality can be tackled well with linearmodels. Consider sparse models. Tools like vowpal wabbit. FTRL , libfm, libffm, liblinear are good tools matrices in python (things like csr matrices). Consider ensembling (like combining) models trained on smaller parts of the data.

15. What is the SDLC (Sofware Development Life Cycle) of projects involving Machine Learning ?

• Give a walk-through on an industrial project and steps involved, so that we can get an idea how they are used. Basically, I am in learning phase and would expect to get an industry level exposure.
• Business questions: How to recommend products online to increase purchases.
• Translate this into an ml problem. Try to predict what the customer will buy in the future given some data available at the time the customer is likely to make the click/purchase, given some historical exposures to recommendations
• Establish a test /validation framework.
• Find best solutions to predict best what customer chose.
• Consider time/cost efficiency as well as performance
• Export model parameters/pipeline settings
• Apply these in an online environment. Expose some customers but NOT all. Keep test and control groups
• Assess how well the algorithm is doing and make adjustments over time.

16. Which is your favorite machine learning algorithm?

It has to be Gradient Boosted Trees. All may be good though in different tasks.

15. Which language is best for deep learning, R or Python?

I prefer Python. I think it is more program-ish . R is good too.

16. What would someone trying to switch careers in data science need to gain aside from technical skills? As I don’t have a developer background would personal projects be the best way to showcase my knowledge?

The ability to translate business problems to machine learning, and transforming them into solvable problems.

17. Do you agree with the statement that in general feature engineering (so exploring and recombining predictors) is more efficient than improving predictive models to increase accuracy?

In principle – Yes. I think model diversity is better than having a few really strong models. But it depends on the problem.

18. Are the skills required to get to the leaderboard top on Kaggle also those you need for your day-to day job as a data scientist? Or do they intersect or are somewhat different? Can I make the idea of what a data scientist’s job is based on Kaggle competitions? And if a person does well on Kaggle does it follow that she will be a successful data scientist in her career ?

There is some percentage of overlap especially when it comes to making predictive models, working with data through python/R and creating reports and visualizations. What Kaggle does not offer (but you can get some idea) is:

• How to translate a business question to a modelling (possibly supervised) problem
• How to monitor models past their deployment
• How to explain (many times) difficult concepts to stake holders.
• I think there is always room for a good kaggler in the industry world. It is just that data science can have many possible routes. It may be for example that not everyone tends to be entrepreneurial in their work or gets to be very client facing, but rather solving very particular (technical) tasks.

19. Which machine learning concepts are must to have to perform well in a kaggle competition?

• Data interrogation/exploration
• Data transformation – pre-processing
• Hands on knowledge of tools
• Familiarity with metrics and optimization
• Cross Validation
• Model Tuning
• Ensembling

20. How do you see the future of data scientist job? Is automation going to kill this job?

No – I don’t think so. This is what they used to say about automation through computing. But ended up requiring a lot of developers to get the job done! It may be possible that data scientists focus on softer tasks over time like translating business questions to ml problems and generally becoming shepherds’ of the process – as in managers/supervisors of the modelling process.

21. How to use ensemble modelling in R and Python to increase the accuracy of prediction. Please quote some real life examples?

You can see my github script as I explain different Machine leaning methods based on a Kaggle competition. Also, check this ensembling guide.

22. What is best python deep learning libraries or framework for text analysis?

I like Keras (because now supports sparse data), Gensim (for word 2 vec).

23. How valuable is the knowledge gained through these competitions in real life? Most often I see competitions won by ensembling many #s of models … is this the case in real life production systems? Or are interpretable models more valuable than these monster ensembles in real productions systems?

In some cases yes – being interpretable or fast (or memory efficient) is more important. Butthis is likely to change over time as people will be less afraid of black box solutions and focus on accuracy.

24. Should I worry about learning about the internals about the machine learning algorithms or just go ahead and try to form an understanding of the algorithms and use them (in competitions and to solve real life business problems) ?

You don’t need the internals. I don’t know all the internals. It is good if you do, but you don’t need to. Also there are new stuff coming out every day – sometimes is tough to keep track of it. That is why you should focus on the decent usage of any algorithm rather than over investing in one.

25. Which are the best machine learning techniques for imbalanced data?

I don’t do a special treatment here. I know people find that strange. This comes down to optimizing the right metric (for me). It is tough to explain in a few lines. There are many techniques for sampling, but I never had to use. Some people are using Smote. I don’t see value in trying to change the principal distribution of your target variable. You just end up with augmented or altered principal odds. If you really want a cut-off to decide on whether you should act or not – you may set it based on the principal odds.

I may not be the best person to answer this. I personally have never found it (significantly) useful to change the distribution of the target variable or the perception of the odds in the target variable. It may just be that other algorithms are better than others when dealing with this task (for example tree-based ones should be able to handle this).

26. Typically, marketing research problems have been mostly handled using standard regression techniques – linear and logistic regression, clustering, factor analyses, etc…My question is how useful are machine learning and deep learning techniques/algorithms useful to marketing research or business problems? For example how useful is say interpreting the output of a neural network to clients? Are there any resources you can refer to?

They are useful in the sense that you can most probably improve accuracy (in predicting let’s say marketing response) versus linear models (like regressions). Interpreting the output is hard and in my opinion it should not be necessary as we are generally moving towards more black box and complicated solutions.

As a data scientist you should put effort in making certain that you have a way to test how good your results are on some unobserved (test) data rather trying to understand why you get the type of predictions you are getting. I do think that decompressing information from complicating models is a nice topic (and valid for research), but I don’t see it as necessary.

On the other hand, companies, people, data scientists, statisticians and generally anybody who could be classified as a ‘data science player’ needs to get educated to accept black box solutions as perfectly normal. This may take a while, so it may be good to run some regressions along with any other modelling you are doing and generally try to provide explanatory graphs and summarized information to make a case for why your models perform as such.

27. How to build teams for collaboration on Kaggle ?

You can ask in forums (i.e in kaggle) . This may take a few competitions though before ’people can trust you’. Reason being, they are afraid of duplicate accounts (which violate competition rules), so people would prefer somebody who is proven to play fair. Assuming some time has passed, you just need to think of people you would like play with, people you think you can learn from and generally people who are likely to take different approaches than you so you can leverage the benefits of diversity when combining methods.

28. I have gone through basic machine learning course(theoretical) . Now I am starting up my practical journey , you just recommended to go through sci-kit learn docs & now people are saying TENSORFLOW is the next scikit learn , so should I go through scikit or TF is a good choice ?

I don’t agree with this statement ‘people are saying TENSORFLOW is the next scikit learn’. Tensorflow is a framework to do well certain machine learning tasks (like for deep learning). I think you can learn both, but I would start with scikit. I personally don’t know TensorFlow , but I use tools that are based on tensor flow (for example Keras). I am lazy I guess!

29. The main challenge that I face in any competition is cleaning the data and making it usable for prediction models. How do you overcome it ?

Yeah. I join the club! After a while you will create pipelines that could handle this relatively quicker. However…you always need to spend time here.

30. How to compute big data without having powerful machine?

You should consider tools like vowpal wabbit and online solutions, where you parse everything line by line. You need to invest more in programming though.

31. What is Feature Engineering?

In short, feature engineering can be understood as:

• Feature transformation (e.g. converting numerical or categorical variables to other types)
• Feature selections
• Exploiting feature interactions (like should I combine variable A with variable B?)
• Treating null values
• Treating outliers

32. Which maths skills are important in machine learning?

Some basic probabilities along with linear algebra (e.g. vectors). Then some stats help too. Like averages, frequency, standard deviation etc.

33. Can you share your previous solutions?

See some with code and some without (just general approach).

• https://www.kaggle.com/c/malware-classification/discussion/13863
• http://blog.kaggle.com/2015/05/11/microsoft-malware-winners-interview-2nd-place-gert-marios-aka-kazanova/
• https://github.com/kaz-Anova/ensemble_amazon
• http://blog.kaggle.com/2015/12/03/dato-winners-interview-1st-place-mad-professors/
• http://blog.kaggle.com/2016/04/08/homesite-quote-conversion-winners-write-up-1st-place-kazanova-faron-clobber/
• https://mlwave.com/how-we-won-3rd-prize-in-crowdanalytix-copd-competition/
• http://blog.kaggle.com/2016/08/31/avito-duplicate-ads-detection-winners-interview-2nd-place-team-the-quants-mikel-peter-marios-sonny/
• http://blog.kaggle.com/2016/12/15/bosch-production-line-performance-competition-winners-interview-3rd-place-team-data-property-avengers-darragh-marios-mathias-stanislav/

34. How long should it take for you to build your first machine learning predictor ?

Depends on the problem (size, complexity, number of features). You should not worry about the time. Generally in the beginning you might spend much time on things that could be considered much easier later on. You should not worry about the time as it may be different for each person, given the programming, background or other experience.

35. Are there any knowledge competitions that you can recommend where you are not necessarily competing on the level as Kaggle but building your skills?

From here, both titanic and digit recognizer are good competitions to start. Titanic is better because it assumes a flat file. Digit recognizer is for image classification so it might be more advanced.

36. What is your opinion about using Weka and/or R vs Python for learning machine learning?

I like Weka. It has a good documentation– especially if you want to learn the algorithms. However I have to admit that it is not as efficient as some of the R and Python implementations. It has good coverage though. Weka has some good visualizations too – especially for some tree-based algorithms. I would probably suggest you to focus on R and Python at first unless your background is strictly in Java.

Summary

In short, succeeding in machine learning competition is all about learning new things, spending a lot of time training, feature engineering and validating models. Alongside, interact with community on forums, read blogs and learn from approach of fellow competitors.

Success is imminent, given that if you keep trying. Cheers!

Introduction

“This hot new field promises to revolutionize industries from business to government, health care to academia,” says the New York Times. People have woken up to the fact that without analyzing the massive amounts of data that’s at their disposal and extracting valuable insights, there really is no way to successfully sustain in the coming years.

Touted as the most promising profession of the century, data science needs business savvy people who have listed data literacy and strategic thinking as their key skills. Anjul Bhambri, VP of Architecture at Adobe, says, “A Data Scientist is somebody who is inquisitive, who can stare at data and spot trends. It’s almost like a Renaissance individual who really wants to learn and bring change to an organization.” (She was previously IBM’s VP of Big Data Products.)

How do we get value from this avalanche of data in every sector in the economy? Well, we get persistent and data-mad personnel skilled in math, stats, and programming to weave magic using reams of letters and numbers.

Over the last few years, people have moved away from the umbrella term, data scientist. Companies now advertise for a diverse set of job roles such as data engineers, data architects, business analysts, MIS reporting executives, statisticians, machine learning engineers, and big data engineers.

In this post, you’ll get a quick overview about these exciting positions in the field of analytics. But do remember that companies often tend to define job roles in different ways based on the inner workings rather than market descriptions.

List of Job Roles in Data Science / Big Data

1. MIS Reporting Executive

Business managers rely on Management Information System reports to automatically track progress, make decisions, and identify problems. Most systems give you on-demand reports that collate business information, such as sales revenue, customer service calls, or product inventory, which can be shared with key stakeholders in an organization.

Skills Required:

MIS reporting executives typically have degrees in computer science or engineering, information systems, and business management or financial analysis. Some universities also offer degrees in MIS. Look at this image from the University of Arizona which clearly distinguishes MIS from CS and Engineering.

Roles & Responsibilities:

MIS reporting executives meet with top clients and co-workers in public relations, finance, operations, and marketing teams in the company to discuss how far the systems are helping the business achieve its goals, discern areas of concern, and troubleshoot system-related problems including security.

They are proficient in handling data management tools and different types of operating systems, implementing enterprise hardware and software systems, and in coming up with best practices, quality standards, and service level agreements. Like they say, an MIS executive is a “communication bridge between business needs and technology.”

2. Business Analyst

Although many of their job tasks are similar to that of data analysts, business analysts are experts in the domain they work in. They try to narrow the gap between business and IT. Business analysts provide solutions that are often technology-based to enhance business processes, such as distribution or productivity.

Organizations need these “information conduits” for a plethora of things such as gap analysis, requirements gathering, knowledge transfer to developers, defining scope using optimal solutions, test preparation, and software documentation.

Skills Required:

Apart from a degree in business administration in the field of your choice, say, healthcare or finance, aspiring business analysts need to have knowledge of data visualization tools such as Tableau and requisite IT know-how, including database management and programming.

You could also major in computer science with additional courses that include statistics, organizational behavior, and quality management. Or you could get professional certifications such as the Certified Business Analysis Professional (CBAP®) or PMI Professional in Business Analysis (PBA). Many universities offer degrees in business intelligence, business analytics, and analytics. Check out the courses in the U.S/India.

Roles & Responsibilities:

Business analysts identify business needs, crystallizing the data for easy understanding, manipulation, and analysis via clear and precise requirements documentation, process models, and wireframes. They identify key gaps, challenges, and potential impacts of a solution or strategy.

In a day, a business analyst could be doing anything from defining a business case or eliciting information from top management to validating solutions or conducting quality testing. Business analysts need to be effective communicators and active listeners, resilient and incisive, to translate tech speak or statistical analysis into business intelligence.

They use predictive, prescriptive, and descriptive analysis to transform complex data into easily understood actionable insights for the users. A change manager, a process analyst, and a data analyst could well be doing business analysis tasks in their everyday work.

3. Data Analyst

Unlike data scientists, data analysts are more of generalists. Udacity calls them junior data scientists. They play a gamut of roles, from acquiring massive amounts of data to processing and summarizing it.

Skills Required:

Data analysts are expected to know R, Python, HTML, SQL, C++, and Javascript. They need to be more than a little familiar with data retrieval and storing systems, data visualization and data warehousing using ETL tools, Hadoop-based analytics, and business intelligence concepts. These persistent and passionate data miners usually have a strong background in math, statistics, machine learning, and programming.

Roles & Responsibilities:

Data analysts are involved in data munging and data visualization. If there are requests from stakeholders, data analysts have to query databases. They are in charge of data that is scraped, assuring the quality and managing it. They have to interpret data and effectively communicate the findings.

Optimization is must-know skill for a data analyst. Designing and deploying algorithms, culling information and recognizing risk, extrapolating data using advanced computer modeling, triaging code problems, and pruning data are all in a day’s work for a data analyst. For more information about how a data analyst is different from a data scientist.

4. Statistician

Statisticians collect, organize, present, analyze, and interpret data to reach valid conclusions and make correct decisions. They are key players in ensuring the success of companies involved in market research, transportation, product development, finance, forensics, sport, quality control, environment, education, and also in governmental agencies. A lot of statisticians continue to enjoy their place in academia and research.

Skills Required:

Typically, statisticians need higher degrees in statistics, mathematics, or any quantitative subject. They need to be mini-experts of the industries they choose to work in. They need to be well-versed in R programming, MATLAB, SAS, Python, Stata, Pig, Hive, SQL, and Perl.

They need to have strong background in statistical theories, machine learning and data mining and munging, cloud tools, distributed tools, and DBMS. Data visualization is a hugely useful skill for a statistician. Aside from industry knowledge and problem-solving and analytical skills, excellent communication is a must-have skill to report results to non-statisticians in a clear and concise manner.

Roles & Responsibilities:

Using statistical analysis software tools, statisticians analyze collected or extracted data, trying to identify patterns, relationships, or trends to answer data-related questions posed by administrators or managers. They interpret the results, along with strategic recommendations or incisive predictions, using data visualization tools or reports.

Maintaining databases and statistical programs, ensuring data quality, and devising new programs, models, or tools if required also come under the purview of statisticians. Translating boring numbers into exciting stories is no easy task!

5. Data Scientist

One of the most in-demand professionals today, data scientists rule the roost of number crunchers. Glassdoor says this is the best job role for someone focusing on work-life balance. Data scientists are no longer just scripting success stories for global giants such as Google, LinkedIn, and Facebook.

Almost every company has some sort of a data role on its careers page.Job Descriptions for data scientists and data analysts show a significant overlap.

Skills Required:

They are expected to be experts in R, SAS, Python, SQL, MatLab, Hive, Pig, and Spark. They typically hold higher degrees in quantitative subjects such as statistics and mathematics and are proficient in Big Data technologies and analytical tools. Using Burning Glass’s tool Labor Insight, Rutgers students came up with some key insights after running a fine-toothed comb through job postings data in 2015.

Roles & Responsibilities:

Like Jean-Paul Isson, Monster Worldwide, Inc., says, “Being a data scientist is not only about data crunching. It’s about understanding the business challenge, creating some valuable actionable insights to the data, and communicating their findings to the business.” Data scientists come up with queries.

Along with predictive analytics, they also use coding to sift through large amounts of unstructured data to derive insights and help design future strategies. Data scientists clean, manage, and structure big data from disparate sources. These “curious data wizards” are versatile to say the least—they enable data-driven decision making often by creating models or prototypes from trends or patterns they discern and by underscoring implications.

6. Data Engineer/Data Architect

“Data engineers are the designers, builders and managers of the information or “big data” infrastructure.” Data engineers ensure that an organization’s big data ecosystem is running without glitches for data scientists to carry out the analysis.

Skills Required:

Data engineers are computer engineers who must know Pig, Hadoop, MapReduce, Hive, MySQL, Cassandra, MongoDB, NoSQL, SQL, Data streaming, and programming. Data engineers have to be proficient in R, Python, Ruby, C++, Perl, Java, SAS, SPSS, and Matlab.

Other must-have skills include knowledge of ETL tools, data APIs, data modeling, and data warehousing solutions. They are typically not expected to know analytics or machine learning.

Roles & Responsibilities:

Data infrastructure engineers develop, construct, test, and maintain highly scalable data management systems. Unlike data scientists who seek an exploratory and iterative path to arrive at a solution, data engineers look for the linear path. Data engineers will improve existing systems by integrating newer data management technologies.

They will develop custom analytics applications and software components. Data engineers collect and store data, do real-time or batch processing, and serve it for analysis to data scientists via an API. They log and handle errors, identify when to scale up, ensure seamless integration, and “build human-fault-tolerant pipelines.” The career path would be Data Engineer?Senior Data Engineer?BI Architect?Data Architect.

7. Machine Learning Engineer

Machine learning (ML) has become quite a booming field with the mind-boggling amount of data we have to tap into. And, thankfully, the world still needs engineers who use amazing algorithms to make sense of this data.

Skills Required:

Engineers should focus on Python, Java, Scala, C++, and Javascript. To become a machine learning engineer, you need to know to build highly-scalable distributed systems, be sure of the machine learning concepts, play around with big datasets, and work in teams that focus on personalization.

ML engineers are data- and metric-driven and have a strong foundation in mathematics and statistics. They are expected to have experience in Elasticsearch, SQL, Amazon Web Service, and REST APIs. As always, great communication skills are vital to interpret complex ML concepts to non-experts.

Roles & Responsibilities:

Machine learning engineers have to design and implement machine learning applications/algorithms such as clustering, anomaly detection, classification, or prediction to address business challenges. ML engineers build data pipelines, benchmark infrastructure, and do A/B testing.

They work collaboratively with product and development teams to improve data quality via tooling, optimization, and testing. ML engineers have to monitor the performance and ensure the reliability of machine learning systems in the organization.

8. Big Data Engineer

What a big data solutions architect designs, a big data engineer builds, says DataFloq founder Mark van Rijmenam. Big data is a big domain, every kind of role has its own specific responsibilities.

Skills Required:

Big data engineers, who have computer engineering or computer science degrees, need to know basics of algorithms and data structures, distributed computing, Hadoop cluster management, HDFS, MapReduce, stream-processing solutions such as Storm or Spark, big data querying tools such as Pig, Impala and Hive, data integration, NoSQL databases such as MongoDB, Cassandra, and HBase, frameworks such as Flume and ETL tools, messaging systems such as Kafka and RabbitMQ, and big data toolkits such as H2O, SparkML, and Mahout.

They must have experience with Hortonworks, Cloudera, and MapR. Knowledge of different programming and scripting languages is a non-negotiable skill. Usually, people with 1 to 3 years of experience handling databases and software development is preferred for an entry-level position.

Roles & Responsibilities:

Rijmenam says “Big data engineers develop, maintain, test, and evaluate big data solutions within organizations. Most of the time they are also involved in the design of big data solutions, because of the experience they have with Hadoop[-]based technologies such as MapReduce, Hive, MongoDB or Cassandra.”

To support big data analysts and meet business requirements via customization and optimization of features, big data engineers configure, use, and program big data solutions. Using various open source tools, they “architect highly scalable distributed systems.” They have to integrate data processing infrastructure and data management.

It is a highly cross-functional role. With more years of experience, the responsibilities in development and operations; policies, standards and procedures; communication; business continuity and disaster recovery; coaching and mentoring; and research and evaluation increase.

Summary

Companies are running helter-skelter looking for experts to draw meaningful conclusions and make logical predictions from mammoth amounts of data. To meet these requirements, a slew of new job roles have cropped up, each with slightly different roles & responsibilities and skill requirements.

Blurring boundaries aside, these job roles are equally exciting and as much in demand. Whether you are a data hygienist, data explorer, data modeling expert, data scientist, or business solution architect, ramping up your skill portfolio is always the best way forward.

Look at these trends from Indeed.com

If you know exactly what you want to do with your coveted skillset comprising math, statistics, and computer science, then all you need to do is hone the specific combination that will make you a name to reckon with in the field of data science or data engineering.

To read more informative posts about data science and machine learning, go here.