I've recently done a q&a session on machine learning. Here's an excerpt of what I answered. You can find the full session here.

What is a formal definition of machine learning? 

A system whose performance with respect to some error function increases with experience.

In what ways are machine learning overrated?

That depends on the rating system.

For example, it’s my impression that most people who know and use machine learning have a quite accurate and balanced view of the possibilities, and more importantly the limitations of machine learning.

The problem arises when authorities who may not have adequate experience such as most of mainstream media (and me) start making claims and predictions without being very careful about not misinterpreting or misrepresenting their sources.

Moreover, even an objective reporting on the events may lead to an incorrect picture of the capabilities.

For example, when chatbots, self-driving cars, spam filtering all are collected under the same ‘machine learning’ umbrella term, it’s not unreasonable to think that they are the same problems, and extrapolate that we’re just a few years away from super intelligence.

This is strengthened by a marketing incentive for showcasing only primal examples and steer away from fringe cases. This is especially seen in natural language based “virtual assistants” where a small number of ‘advanced queries’ for which they have been specifically optimized are showcased; misleading people to assume that it’s a representative sample. False representativity is a problem for not just machine learning, but for nefarious claims in general.

While actually, even just making a robot do something without screwing everything else up is really difficult. There’s also currently no real solution for adversarial examples.

In conclusion, it’s possible that the average technological interested, but not necessarily literate, person tends to overestimate the capabilities of machine learning. In particular, people tend to overestimate the generalizability of current machine learning algorithms, and compare it to that of humans.

Can a neural network really do anything with the right weights and hyperparameters?

No, neural networks cannot do everything and anything.

Neural networks are good at mapping continuous geometric relationships given a large amount of labeled data. In fact, they can approximate any continuous relationship given enough data.

This ability has led to some incredible achievements in many fields.

However, neural networks still have several limitations among which are:

• Adversarial examples
• Biased/Untruthful training data
• Weak generalization
• Limited mapping
• Require a lot of human-annotated training data
• Limited transfer learning

Furthermore, there are things that cannot be done using neural networks; not now, not ever. For example:

• Anything discontinuous
• Anything for which data cannot be collected or simulated
• Symbolic manipulation (at least without a supporting agent)
• Beat entropy; no new novel information (not even by synthesising information with generative networks)
  
• Neural networks do not imply P=NP with forward propagation being P.
  
• This implies that neural networks cannot perfectly encode NP problems in finite space.

For a beginner who is not a coder, is it necessary to learn Python before learning machine learning or data science?

You don’t need to know Python in order to learn about machine learning, or data science.

In fact, you don’t even need to know a programming language in order to understand the theory.

But if you want to actually implement, or use anything you learn, you will need to learn some programming eventually.

However, while Python is probably the most used programming language in the data science community, there are a few good alternatives, and more less used (bad) alternatives.

The largest competitor is R. R is focused on statistical computing, and while R is turing complete, there’s less focus on general purpose programming in the R community which means you’ll be at a disadvantage compared to Python when doing anything but data science. (That is not necessarily a bad thing though).

Some years ago, MATLAB was quite popular, but it’s not open source, and requires a commercial license to use, so it has fallen out of favor recently.

Today, most popular programming language has some sort of machine learning library available. The most popular; Tensorflow, however, is fully supported by Python with more languages coming.

If your question instead was if it is necessary to learn the basics of any language before learning machine learning (assuming you don’t know the theory very well), I until recently would have said no as most programming languages are sufficiently simple to learn while you’re learning something else.

But when I mentored some people in learning both machine learning and a programming language at the same time, I noticed that they confused programming, machine learning, and the custom libraries they are using.

This lead to more than a few inefficiencies that could have been avoided had they learned the basics of the respectable domains beforehand why I now recommend getting familiar with the basics of programming before applying data science concepts.

More questions include:

• How should I choose between becoming a full stack JavaScript developer and a machine learning engineer?
• What should I concentrate on after learning machine learning if I am not interested in deep learning?
• What does 'deep' mean in machine learning?
• What type of optimization is needed to train a neural network? And what is your favorite course for the types of optimization we need for NN?
• How do you calculate gradients in a feed forward neural network using matrices?
• Is automatic differentiation in deep learning the same as numerical differentiation?
• How should I fill category missing values to apply machine learning?
• And more

You can find the full session here.