All about Deep Learning Tutorial

This is the eleventh tutorial in the series. In this tutorial, we will be studying Deep Learning. Deep learning is a subset of machine learning in artificial intelligence (AI) that has networks capable of learning unsupervised from data that is unstructured or unlabeled. Also known as deep neural learning or deep neural network

Introduction

 
In the previous article, we studied Machine Learning. One thing that I believe is that if we can correlate anything with us or our life, there are greater chances of understanding the concept. So I will try to explain everything by relating it to humans.
 

What is Deep Learning? 

 
Deep learning is an artificial intelligence function that imitates the workings of the human brain in processing data and creating patterns for use in decision making. Deep learning is a subset of machine learning in artificial intelligence (AI) that has networks capable of learning unsupervised from data that is unstructured or unlabeled. Also known as deep neural learning or deep neural network. 
 
To grasp the idea of deep learning, imagine a family, with an infant and parents. The toddler points objects with his little finger and always says the word 'cat.' As its parents are concerned about his education, they keep telling him 'Yes, that is a cat' or 'No, that is not a cat.' The infant persists in pointing objects but becomes more accurate with 'cats.' The little kid, deep down, does not know why he can say it is a cat or not. He has just learned how to hierarchies complex features coming up with a cat by looking at the pet overall and continue to focus on details such as the tails or the nose before to make up his mind.
 
A neural network works quite the same. Each layer represents a deeper level of knowledge, i.e., the hierarchy of knowledge. A neural network with four layers will learn more complex features than with that with two layers.
 
The learning occurs in two phases.
  • The first phase consists of applying a nonlinear transformation of the input and create a statistical model as output.
  • The second phase aims at improving the model with a mathematical method known as derivative.
The neural network repeats these two phases hundreds to thousands of times until it has reached a tolerable level of accuracy. The repeat of this two-phase is called an iteration.
 

Who invented Deep Learning? 

 
The history of Deep Learning can be traced back to 1943 when Walter Pitts and Warren McCulloch created a computer model based on the neural networks of the human brain. They used a combination of algorithms and mathematics they called “threshold logic” to mimic the thought process 
 

Advantages/Features of Deep Learning

 
1. Features are automatically deduced and optimally tuned for the desired outcome. Features are not required to be extracted ahead of time. This avoids time-consuming machine learning techniques.
2. Robustness to natural variations in the data is automatically learned.
3. The same neural network-based approach can be applied to many different applications and data types.
4. Massive parallel computations can be performed using GPUs and are scalable for large volumes of data. Moreover, it delivers better performance results when the amount of data is huge. 
5. The deep learning architecture is flexible to be adapted to new problems in the future. 
 

Disadvantages/Shortcomings of Deep Learning 

 
1. It requires a very large amount of data to perform better than other techniques.
2. It is extremely expensive to train due to complex data models. Moreover, deep learning requires expensive GPUs and hundreds of machines. This increases the cost to the users.
3. There is no standard theory to guide you in selecting the right deep learning tools as it requires knowledge of topology, training method, and other parameters. As a result, it is difficult to be adopted by less skilled people.
4. It is not easy to comprehend output based on mere learning and requires classifiers to do so. Convolutional neural network-based algorithms perform such tasks.
 

Drivers of Deep Learning

 
The drivers of Deep Learning are listed below:
  • Availability of multi-layered learning network
  • Ability to leverage Big Data
  • Expanded use of high-performing graphics processing units (GPU)
  • An improved scale of data and size of neural networks
  • Improved performance of Neural Networks
  • Availability of a large amount of labeled data
  • The ability of GPUs to perform parallel computing 

Difference Between Machine Learning and Deep Learning?

 
  Machine Learning Deep Learning
Formal Definition  A computer program is said to learn from experience E concerning some class of tasks T and performance measure P if its performance at tasks in T, as measured by P, improves with experience E Deep learning is a particular kind of machine learning that achieves great power and flexibility by learning to represent the world as a nested hierarchy of concepts, with each concept defined concerning simpler concepts, and more abstract representations computed in terms of less abstract ones. 
Data When Data is small, machine learning is used When data is large, deep learning is used, 
Hardware Requirements  Machine Learning algorithms can work on CPUs Deep Learning algorithms require GPUs and TPUs to work 
Feature Engineering
In Machine learning, most of the applied features need to be identified by an expert and then hand-coded as per the domain and data type.
Deep learning reduces the task of developing new feature extractor for every problem
Problem Solving Approach  Machine Learning tries to break the problem into parts and solve each part separately Deep Learning solves the problem in an end-to-end manner
Execution Time The execution time is comparatively less i.e. a few hours to a week The execution time is comparatively more i.e up to weeks
Interpretability Machine Learning algorithms are easy to interpret Deep Learning algorithms are not that easy to understand, or we can say difficult to impossible
Number of Algorithms present  We currently have a good number of Machine learning algorithms  We currently have a very few numbers of Deep Learning algorithms 
Training Time Short Long 
 
The following image shows a comparison between the popularity of Machine Learning and Deep Learning.
Legend:
Blue: Machine  Learning
Red: Deep Learning 
 
mlvsdl 
Source: Google Trends 
 

Python Deep Learning Libraries 

 

1. TensorFlow

 
TensorFlow is a free and open-source software library for dataflow and differentiable programming across a range of tasks. It is a symbolic math library and is also used for machine learning applications such as neural networks. It is used for both research and production at Google.
 
For a detailed tutorial on TensorFlow, please click 
 

2. Keras

 
Keras is a high-level neural networks API, written in Python and capable of running on top of TensorFlow, CNTK, or Theano. It was developed with a focus on enabling fast experimentation. Being able to go from idea to result with the least possible delay is key to doing good research.
 

3. Apache mxnet

 
Apache MXNet is an open-source deep learning software framework, used to train, and deploy deep neural networks. It has wide applicability across various OS and using various languages.
 

4. Caffe

 
Caffe is a deep learning framework made with expression, speed, and modularity in mind. It is developed by Berkeley AI Research (BAIR) and by community contributors. Yangqing Jia created the project during his Ph.D. at UC Berkeley. Caffe is released under the BSD 2-Clause license.
 

5. Theano

 
Theano is a Python library and optimizing compiler for manipulating and evaluating mathematical expressions, especially matrix-valued ones. In Theano, computations are expressed using a NumPy-Esque syntax and compiled to run efficiently on either CPU or GPU architectures.
 

6. Microsoft Cognitive Toolkit

 
Azure Cognitive Services are APIs, SDKs, and services available to help developers build intelligent applications without having direct AI or data science skills or knowledge. Azure Cognitive Services enable developers to easily add cognitive features into their applications. The goal of Azure Cognitive Services is to help developers create applications that can see, hear, speak, understand, and even begin to reason. The catalog of services within Azure Cognitive Services can be categorized into five main pillars - Vision, Speech, Language, Web Search, and Decision.
 

7. PyTorch

 
PyTorch is an open-source machine learning library based on the Torch library, used for applications such as computer vision and natural language processing. It is primarily developed by Facebook's artificial intelligence research group. It is free and open-source software released under the Modified BSD license.
 

8. Eclipse DeepLearning4J (DL4J)

 
Eclipse Deeplearning4j is the first commercial-grade, open-source, distributed deep-learning library written for Java and Scala. Integrated with Hadoop and Apache Spark, DL4J brings AI to business environments for use on distributed GPUs and CPUs.
 

9. Lasagne

 
Lasagne is a lightweight library to build and train neural networks in Theano. Its main features are:
  • Supports feed-forward networks such as Convolutional Neural Networks (CNNs), recurrent networks including Long Short-Term Memory (LSTM), and any combination thereof
  • Allows architectures of multiple inputs and multiple outputs, including auxiliary classifiers
  • Many optimization methods including Nesterov momentum, RMSprop, and ADAM
  • Freely definable cost function and no need to derive gradients due to Theano's symbolic differentiation
  • Transparent support of CPUs and GPUs due to Theano's expression compiler 

10. noLearn

 
nolearn contains a number of wrappers and abstractions around existing neural network libraries, most notably Lasagne, along with a few machine learning utility modules. All code is written to be compatible with scikit-learn.
 

11. PyLearn2

 
Pylearn2 is a machine learning library. Most of its functionality is built on top of Theano. This means you can write Pylearn2 plugins (new models, algorithms, etc) using mathematical expressions, and Theano will optimize and stabilize those expressions for you, and compile them to a backend of your choice (CPU or GPU).
 

Artificial Neural Networks

 
typesOfNN
 
A neural network is a series of algorithms that endeavors to recognize underlying relationships in a set of data through a process that mimics the way the human brain operates. In this sense, neural networks refer to systems of neurons, either organic or artificial. Neural networks can adapt to changing input; so the network generates the best possible result without needing to redesign the output criteria. The concept of neural networks, which has its roots in artificial intelligence, is swiftly gaining popularity in the development of trading systems.
 

Artificial Neural Networks Process

 
Artificial Neural Networks consist of the following four main parts:
 

1. Neuron

 
Artificial Neural Networks contain layers of neurons. A neuron is a computational unit that calculates a piece of information based on weighted input parameters. Inputs accepted by the neuron are separately weighted.
 
Inputs are summed and passed through a non-linear function to produce output. Each layer of neurons detects some additional information, such as edges of things in a picture or tumors in a human body. Multiple layers of neurons can be used to detect additional information about input parameters.
 

2. Nodes

 
Artificial Neural Network is an interconnected group of nodes akin to the vast network of layers of neurons in a brain. Each circular node represents an artificial neuron and an arrow represents a connection from the output of one neuron to the input of another.
 

3. Inputs

 
Inputs are passed into the first layer. Individual neurons receive the inputs, with each of them receiving a specific value. After this, an output is produced based on these values.
 

4. Outputs

 
The outputs from the first layer are then passed into the second layer to be processed. This continues until the final output is produced. The assumption is that the correct output is predefined.
 
Each time data is passed through the network, the result is compared with the correct one, and tweaks are made to their values until the network creates the correct final output each time.
 

Classification of Artificial Neural Networks

 
1. Shallow neural network
The Shallow neural network has only one hidden layer between the input and output.
 
2. Deep neural network:
Deep neural networks have more than one layer. For instance, the Google LeNet model for image recognition counts 22 layers.
 

Types of Deep Learning Networks

 

1. Feed-forward Neural Networks

 
The simplest type of artificial neural network. With this type of architecture, information flows in only one direction, forward. It means, the information flows starts at the input layer, goes to the "hidden" layers, and ends at the output layer. The network does not have a loop. Information stops at the output layers.
 

2. Recurrent neural networks (RNNs)

 
RNN is a multi-layered neural network that can store information in context nodes, allowing it to learn data sequences and output a number or another sequence. In simple words it an Artificial neural network whose connections between neurons include loops. RNNs are well suited for processing sequences of inputs.
 
For example, if the task is to predict the next word in the sentence "Do you want a…………?
  • The RNN neurons will receive a signal that points to the start of the sentence.
  • The network receives the word "Do" as an input and produces a vector of the number. This vector is fed back to the neuron to provide a memory to the network. This stage helps the network to remember it received "Do" and it received it in the first position.
  • The network will similarly proceed to the next words. It takes the word "you" and "want." The state of the neurons is updated upon receiving each word.
  • The final stage occurs after receiving the word "a." The neural network will provide a probability for each English word that can be used to complete the sentence. A well-trained RNN probably assigns a high probability to "café", "drink," "burger," etc.
Common uses of RNN
  • Help securities traders to generate analytic reports
  • Detect abnormalities in the contract of financial statement
  • Detect fraudulent credit-card transaction
  • Provide a caption for images
  • Power chatbots
  • The standard uses of RNN occur when the practitioners are working with time-series data or sequences (e.g., audio recordings or text).

3. Convolutional neural networks (CNN)

 
CNN is a multi-layered neural network with a unique architecture designed to extract increasingly complex features of the data at each layer to determine the output. CNN's are well suited for perceptual tasks.
 
CNN is mostly used when there is an unstructured data set (e.g., images) and the practitioners need to extract information from it
 
For instance, if the task is to predict an image caption:
  • The CNN receives an image of let's say a cat, this image, in computer term, is a collection of the pixel. Generally, one layer for the greyscale picture and three layers for a color picture.
  • During the feature learning (i.e., hidden layers), the network will identify unique features, for instance, the tail of the cat, the ear, etc.
  • When the network thoroughly learned how to recognize a picture, it can provide a probability for each image it knows. The label with the highest probability will become the prediction of the network.

4. Reinforcement Learning

 
Reinforcement learning is a subfield of machine learning in which systems are trained by receiving virtual "rewards" or "punishments," essentially learning by trial and error.
Google's DeepMind has used reinforcement learning to beat a human champion in the Go games. Reinforcement learning is also used in video games to improve the gaming experience by providing smarter bot.
 
One of the most famous algorithms are:
  • Q-learning
  • Deep Q network
  • State-Action-Reward-State-Action (SARSA)
  • Deep Deterministic Policy Gradient (DDPG)
The following is a list of Neural Networks with their corresponding use cases
 
Neural Network
Use Case
ANN (Artificial Neural Network)
Computational Neuroscience
CNN (Convolutional Neural Network)
Image Processing
RNN (Recurrent Neural Network)
Speech Recognition
DNN (Deep Neural Network)
Acoustic Modeling
DBN (Deep Brief Network)
Drug Discovery
 

Computational Graph  

 
Computational graphs are a nice way to think about mathematical expressions. For example, consider the expression e=(a+b)∗(b+1). There are three operations: two additions and one multiplication. To help us talk about this, let’s introduce two intermediary variables, c and d so that every function’s output has a variable. We now have:
c=a+b
d=b+1
e=c∗d
 
To create a computational graph, we make each of these operations, along with the input variables, into nodes. When one node’s value is the input to another node, an arrow goes from one to another.
 COMPUTATIONAL1
 
These sorts of graphs come up all the time in computer science, especially in talking about functional programs. They are very closely related to the notions of dependency graphs and call graphs. They’re also the core abstraction behind the popular deep learning framework Theano.

We can evaluate the expression by setting the input variables to certain values and computing nodes up through the graph. For example, let’s set a=2 and b=1 
 
COMPUTATIONAL2
 
The output comes out to be 6. 
 

Python Implementation of Computational Graph 

  1. from dask import delayed,compute    
  2. import dask    
  3. @delayed    
  4. def cube(num):    
  5.      print("Cube function:",num)    
  6.      print()    
  7.      return num*num    
  8. @delayed    
  9. def sum_list(args):    
  10.      print("Sum_list function:",args)    
  11.      return sum(args)    
  12. items=[1,2,3]    
  13. computation_graph = sum_list([cube(i) for i in items])    
  14. computation_graph.visualize()     
Output  
 
computational_python 
 

Examples of deep learning applications

 

1. AI in Finance

 
The financial technology sector has already started using AI to save time, reduce costs, and add value. Deep learning is changing the lending industry by using more robust credit scoring. Credit decision-makers can use AI for robust credit lending applications to achieve faster, more accurate risk assessment, using machine intelligence to factor in the character and capacity of applicants.
 
Underwrite is a Fintech company providing an AI solution for credit makers company. underwrite.ai uses AI to detect which applicant is more likely to pay back a loan.
  

2. AI in HR

 
Under Armour, a sportswear company revolutionizes hiring and modernizes the candidate experience with the help of AI. They had, on average, 30000 resumes a month. Reading all of those applications and begin to start the screening and interview process was taking too long. 
 
At that time, Under Armour had all of the 'must-have' HR technology in place such as transactional solutions for sourcing, applying, tracking and onboarding but those tools weren't useful enough. Under armor choose HireVue, an AI provider for HR solution, for both on-demand and live interviews. The results were bluffing; they managed to decrease by 35% the time to fill. In return, the hired higher quality staff.
 

3. AI in Marketing:

 
AI is a valuable tool for customer service management
and personalization challenges. Improved speech recognition in call-center management and call routing as a result of the application of AI techniques allows a more seamless experience for customers.
 
For example, deep-learning analysis of audio allows systems to assess a customer's emotional tone. If the customer is responding poorly to the AI chatbot, the system can be rerouted the conversation to real, human operators that take over the issue.
 

Conclusion

 
In this article, we studied deep learning, what is deep learning, who invented deep learning, advantages of deep learning, disadvantages of deep learning, drivers of deep learning, the difference between machine learning and deep learning, python deep learning libraries, artificial neural network, artificial neural network process, classification of artificial neural network, types of deep learning networks, computational graph and examples of deep learning applications . Hope you were able to understand each and everything. For any doubts, please comment on your query.
 
In the next article, we will learn about Data Science.
 
Congratulations!!! you have climbed your next step in becoming a successful ML Engineer.
 
Next Article In this Series >> Data Science