You are reading the article Discover The Applications Of Deep Learning In Healthcare updated in November 2023 on the website Eastwest.edu.vn. We hope that the information we have shared is helpful to you. If you find the content interesting and meaningful, please share it with your friends and continue to follow and support us for the latest updates. Suggested December 2023 Discover The Applications Of Deep Learning In Healthcare
The applications of DL in healthcare is many, from improved diagnostics to enhance disease predictionDeep learning, a subset of artificial intelligence (AI), has emerged as a revolutionary technology with profound applications in the healthcare industry. In healthcare, deep learning algorithms can be trained to analyze medical images, such as X-rays, MRIs, and CT scans, with remarkable accuracy, aiding in the early detection of diseases and improving diagnostic outcomes. These deep learning algorithms have also shown promise in drug discovery, genomics research, and clinical decision support systems. By harnessing the power of deep learning, healthcare providers can revolutionize patient care, improve efficiency, and ultimately save lives. This article will explore the diverse applications of deep learning in healthcare and their potential impact.
Deep learning, a subset of artificial intelligence (AI), has emerged as a powerful tool with transformative potential across various industries, including healthcare. With its ability to analyze vast amounts of complex data, deep learning holds promise for revolutionizing healthcare by improving diagnostics, personalized medicine, drug discovery, disease prediction, and more.
1. Medical ImagingDeep learning algorithms have demonstrated remarkable capabilities in interpreting medical images such as X-rays, CT scans, MRI scans, and mammograms. By training on large datasets, deep learning models can accurately identify patterns, anomalies, and early signs of diseases. This technology can assist radiologists in diagnosing conditions like cancer, cardiovascular diseases, and neurological disorders, leading to faster and more accurate diagnoses.
2. Disease Diagnosis and PredictionDeep learning models can aid in diagnosing diseases by analyzing patient data, including medical records, symptoms, genetic information, and laboratory results. By leveraging this information, deep learning algorithms can identify disease patterns and provide more accurate and timely diagnoses. Additionally, these models can predict the likelihood of developing certain diseases based on risk factors, allowing for early intervention and prevention.
3. Drug Discovery and DevelopmentThe process of discovering and developing new drugs is time-consuming and costly. Deep learning can accelerate this process by analyzing vast amounts of biomedical data, including molecular structures, genomic data, and clinical trial results. By predicting the efficacy and safety of potential drug candidates, deep learning can help researchers identify promising molecules, optimize drug design, and reduce the time and cost associated with bringing new drugs to market.
4. Personalized Medicine 5. Electronic Health Records (HER) AnalysisDeep learning can unlock valuable insights from electronic health records containing vast patient data. By analyzing this data, deep learning models can identify patterns, predict disease progression, and enable early intervention. Moreover, deep learning algorithms can improve HER accuracy by automatically extracting relevant information, reducing errors, and enhancing healthcare providers’ ability to make informed decisions.
6. Clinical Decision Support SystemsDeep learning can be integrated into clinical decision support systems to aid healthcare professionals in making informed decisions. By analyzing patient data, medical literature, and treatment guidelines, deep learning models can provide recommendations on diagnosis, treatment plans, and medication options. These systems can enhance clinical decision-making, improve patient safety, and reduce medical errors.
7. Disease Outbreak PredictionDeep learning can analyze vast amounts of data, including social media feeds, news articles, and sensor data, to detect early signs of disease outbreaks. By identifying patterns and correlations, deep learning algorithms can predict disease spread, helping public health authorities allocate resources, implement preventive measures, and mitigate the impact of epidemics.
You're reading Discover The Applications Of Deep Learning In Healthcare
Gearing Up To Dive Into Mariana Trench Of Deep Learning
This article was published as a part of the Data Science Blogathon
This article would try to make an effort to take the deepest possible plunge in the ocean of deep learning. Mariana Trench is the deepest trench on earth located in the pacific ocean, so in the ocean of deep learning, we shall try to reach as close to the Mariana Trench. This is a continuation of the previous article, the link of which has been shared below for reference-
This article would cover overfitting and underfitting, and drop out and batch normalization using ‘heart dataset’. The dataset can be downloaded for reference using the following link-
IntroductionUnderfitting and Overfitting – Taking care of underfitting and overfitting enable performance enhancement either by adding capacity or stopping early.
Dropout and Normalization – Take care of underfitting and overfitting. So, let’s discuss the two very important concepts.
Underfitting and Overfitting
Image Source:Kaggle
The above image represents validation loss which gives an idea of an unseen error on invisible data. During the training of a model, the loss on the training set is plot epoch by epoch. To this, we have added another parameter validation data. The condition in which the training loss will go down when the model learns signal or it learns noise. For a nearly ideal situation, the model needs a negotiation with the signal as well as noise which is not enough the signal and not enough noise.
Criteria for underfitting and Overfitting2. Overfitting when the loss of signal is not very low as the model has learned enough too much noise.
Image Source:Kaggle
Method’s to reduce the amount of noise and to get more signal out of training dataMinutes of the concepts would be better comprehended with the help of the lines of code that follows along with the outputs.
import pandas as pd Cardiology = pd.read_csv('heart.csv') Cardiology.head()df_train = Cardiology.sample(frac=0.7, random_state=5) df_valid = Cardiology.drop(df_train.index) display(df_train.head(4)) max_ = df_train.max(axis=0) min_ = df_train.min(axis=0) df_train = (df_train - min_) / (max_ - min_) df_valid = (df_valid - min_) / (max_ - min_)
X_train = df_train.drop('target', axis=1) X_valid = df_valid.drop('target', axis=1) y_train = df_train['target'] y_valid = df_valid['target']
input_shape = [X_train.shape[1]] print("Input shape: {}".format(input_shape)) from tensorflow import keras from tensorflow.keras import layers from tensorflow.keras import callbacks
model = keras.Sequential([ layers.Dense(1, input_shape=input_shape), ]) optimizer='adam', loss='mae', ) history = model.fit( X_train, y_train, validation_data=(X_valid, y_valid), batch_size=512, epochs=50, verbose=0, ) history_df = pd.DataFrame(history.history) history_df.loc[0:, ['loss', 'val_loss']].plot() print("Minimum Validation Loss: {:0.4f}".format(history_df['val_loss'].min())); history_df.loc[10:, ['loss', 'val_loss']].plot() print("Minimum Validation Loss: {:0.4f}".format(history_df['val_loss'].min())); model = keras.Sequential([ layers.Dense(128, activation='relu', input_shape=input_shape), layers.Dense(64, activation='relu'), layers.Dense(1) ]) optimizer='adam', loss='mae', ) history = model.fit( X_train, y_train, validation_data=(X_valid, y_valid), batch_size=512, epochs=50, ) history_df = pd.DataFrame(history.history) history_df.loc[:, ['loss', 'val_loss']].plot() print("Minimum Validation Loss: {:0.4f}".format(history_df['val_loss'].min())); from tensorflow.keras.callbacks import EarlyStopping early_stopping = EarlyStopping( min_delta=0.001,patience=5,restore_best_weights=True,)
model = keras.Sequential([ layers.Dense(128, activation='relu', input_shape=input_shape), layers.Dense(64, activation='relu'), layers.Dense(1) ]) optimizer='adam', loss='mae', ) history = model.fit( X_train, y_train, validation_data=(X_valid, y_valid), batch_size=512, epochs=50, callbacks=[early_stopping] ) history_df = pd.DataFrame(history.history) history_df.loc[:, ['loss', 'val_loss']].plot() print("Minimum Validation Loss: {:0.4f}".format(history_df['val_loss'].min()));
At the outset, we have loaded the dataset. The dataset was then split into the training part and the testing part. The target is the output variable and the rest 13 are all input variables. In the next step, we imported keras, layers, and callback from tensorflow. After importing the necessary libraries and modules, we have started by training a low-capacity linear model. In the output, we can see a huge gap between the loss and the validation loss curve, indicating that the network is overfitting.
After that, we have added capacity to the network by incorporating 3 hidden layers with each having a unit value of 128. We can observe that validation loss and training loss have begun to come very close. So, this suggests that the network is about to underfit.
At this point, we define early stopping callback with patience = 5 epochs, change in validation loss, min_delta=0.001, and setting restore_best_weights=True. In the output, we observed that the early stopping callback stopped the training once the network began underfitting. In addition, with the inclusion of restore_best_weights, the model could be kept where the validation loss was lowest.
Dropout and Batch NormalizationBeyond dense layers, there exist special layers too. Dropout and Batch Normalization are 2 special types of layers. On their own, these layers do not contain any neurons but add valuable functionalities which are beneficial for the model.
1.Dropout– It can rectify overfitting. Overfitting results in spurious patterns in the training data, so to detect these, the network relies on specific combinations of weight. This is also known as conspiracies of weight. The dropout helps in removing such conspiracies by dropping out some fraction of the layer’s input units during each step of training.
Image Source:Kaggle
In the above image, 50% dropout addition has taken place between 2 hidden layers.
2. Batch Normalization – It enables to rectify the training that is either slow or not stable. For convenience, it is important to put all the data on a common scale-like scikit-learn’s StandardScaler as SGD(Stochastic Gradient Descent) shifts the network weights in sync with the largeness of the activation the data produces. A batch normalization layer allows us to do this inside the network by looking at each batch as it comes in.
Minutes of the concepts would be better comprehended with the help of the lines of code that follows along with the outputs.
model = keras.Sequential([ layers.Dense(128, activation='relu', input_shape=input_shape), layers.Dropout(0.3), layers.Dense(64, activation='relu'), layers.Dropout(0.3), layers.Dense(1) ])optimizer=’adam’, loss=’mae’, ) history = model.fit( X_train, y_train, validation_data=(X_valid, y_valid), batch_size=512, epochs=50, verbose=0 ) history_df = pd.DataFrame(history.history) history_df.loc[0:, [‘loss’, ‘val_loss’]].plot() print(“Minimum Validation Loss: {:0.4f}”.format(history_df[‘val_loss’].min()));
history_df.loc[10:, ['loss', 'val_loss']].plot() print("Minimum Validation Loss: {:0.4f}".format(history_df['val_loss'].min()));In the heart model, we have added 2 dropout layers. The layers have been added one each after the Dense layer with 128 units and another Dense layer with 64 units. The drop-out rate in both cases has been set to 0.3. Now, we have run lines of code that are exactly similar to the one we ran previously where the model tended to overfit the data. Here, the addition of dropout seems to have helped in closing the gap.
model = keras.Sequential([ layers.Dense(512, activation='relu', input_shape=input_shape), layers.Dense(512, activation='relu'), layers.Dense(512, activation='relu'), layers.Dense(1), ]) optimizer='sgd', # SGD is more sensitive to differences of scale loss='mae', metrics=['mae'], ) history = model.fit( X_train, y_train, validation_data=(X_valid, y_valid), batch_size=64, epochs=100, verbose=0, ) history_df = pd.DataFrame(history.history) history_df.loc[0:, ['loss', 'val_loss']].plot() print(("Minimum Validation Loss: {:0.4f}").format(history_df['val_loss'].min()))This dataset got trained properly, so did manifest with a minimum validation loss. A certain dataset will fail the training of this network. Let’s try with ‘spotify’ dataset. The link can be found below-
model = keras.Sequential([ layers.Dense(512, activation='relu', input_shape=input_shape), layers.Dense(512, activation='relu'), layers.Dense(512, activation='relu'), layers.Dense(1), ]) optimizer='sgd', # SGD is more sensitive to differences of scale loss='mae', metrics=['mae'], ) history = model.fit( X_train, y_train, validation_data=(X_valid, y_valid), batch_size=64, epochs=100, verbose=0, ) history_df = pd.DataFrame(history.history) history_df.loc[0:, ['loss', 'val_loss']].plot() print(("Minimum Validation Loss: {:0.4f}").format(history_df['val_loss'].min()))Image Source: Kaggle
In this dataset, training the dataset failed as it is converging to a very large network. Here, the role of batch normalization becomes very prominent.
model = keras.Sequential([ layers.BatchNormalization(), layers.Dense(512, activation='relu', input_shape=input_shape), layers.BatchNormalization(), layers.Dense(512, activation='relu'), layers.BatchNormalization(), layers.Dense(512, activation='relu'), layers.BatchNormalization(), layers.Dense(1), ])optimizer=’sgd’, loss=’mae’, metrics=[‘mae’], ) EPOCHS = 100 history = model.fit( X_train, y_train, validation_data=(X_valid, y_valid), batch_size=64, epochs=EPOCHS, verbose=0, ) history_df = pd.DataFrame(history.history) history_df.loc[0:, [‘loss’, ‘val_loss’]].plot() print((“Minimum Validation Loss: {:0.4f}”).format(history_df[‘val_loss’].min())) Image Source: Kaggle
We have added 4 BatchNormalization layers preceding the dense layers. It could be concluded that the addition of batch normalization layers helped to adaptively scale the data while passing through the network. On a difficult dataset, unlike the heart dataset, batch normalization can prove to be an asset.
ConclusionDeep learning is a key component of Artificial Intelligence and has the potential to overhaul many aspects of life including the medical and healthcare sectors. This article covered most of the important concepts of deep learning shortly and crisply. Practicing with the different datasets is important to learn deeper.
Thanks a lot for going through this article. I hope this article did add value to the time you have devoted!
References
The media shown in this article are not owned by Analytics Vidhya and are used at the Author’s discretion.
Related
Transformation Of Healthcare: How Technology Redefines One Of The Oldest Industries In The World
Since healthcare is one of the few industries that are crucial for all people worldwide, no wonder it sees huge spending and even bigger investment. As estimated, in 2023 the global annual healthcare spending may reach $8.7 trillion. And in the first half of 2023, $10.6 billion were poured into the healthcare startups by venture capitalists, as per . These numbers are clear indicators that health care today is a whole field of opportunities for both savvy startups and huge enterprises that are willing to deploy cutting-edge technology for healthcare purposes. However, the implementation of technology within the medical sector started a long time ago, back in the early 1960ies. Since then, healthcare went through a massive transformation, both in terms of adopted practices and equipment. And this is not the end of its development – today the healthcare industry stands on the verge of change once again due to massive deployment of artificial intelligence, RPA, and other technologies.
The pain points of the past and first implementation of technologyOne of the biggest pain points for all the industries at all times have been: • The overwhelming amount of paperwork • High possibility of human error • Mundane and routine tasks that consumed too much time One of the pioneers in the field of medicine digitization was – an American cardiologist who was among the first ones to use computers for decision support in the mid-1950s. In the late 1960s, Warner and his colleagues came up with the monitoring systems that became a globally used standard today and were a true breakthrough back then. In 1980s, the networking technologies for computer connection gained incredible popularity. And in 1991, the EMRs (electronic medical records) were examined and adopted as well. Since then, the healthcare industry has been steadily transforming towards more efficient and accurate processes. The implementation of computers and other technologies freed medical specialists from hours, spent on routine work, and allowed them to focus on tasks that are more critical. As well, computers significantly decreased the possibility of an error and allowed to keep all processes within one place (or network). Within the last few years, the world saw the rise of IoT, AI, and Big Data. Without a doubt, these technologies influenced healthcare significantly and took it to the next level.
Healthcare today: robotic assistants and disease predictionsModern healthcare is heavily influenced by the following technologies: the Internet of Things, Artificial Intelligence, Machine Learning, and Big Data. Each of them offers unique solutions that eliminate risks, decrease the possibility of an error, increase the efficiency of work processes and contribute to medical research.
Data ScienceAs can be guessed from the name, data science is all about analyzing the data and extracting
Medical image processingIn healthcare, image processing plays a vital role in diagnostics. The earlier the disease is found, the higher the chances are that it will be successfully cured. The examples of such image techniques include X-ray, MRI, computed topography. What data science does is that it offers medical specialists an opportunity for more accurate and thorough analysis. Being able to classify the organ texture, find non-obvious disease indicators etc., data science contributes greatly to the detection of severe conditions, like tumor or artery stenosis.
Drug creationAccording to , it may take about 12 years for a drug to get submitted – and think of all the resources and finances invested into the development. But thanks to data science, the time and investment can be reduced significantly. With the help of special algorithms, data science makes it possible to forecast how the drug will act and what will be the expected success rate. Such an approach eliminates the necessity to conduct lab experiments by enabling the tests in carefully crafted simulations.
Computer VisionThis technology offers accurate analysis and inspection and found its niche in the sphere of image analysis. We’ve discussed above that data science contributes greatly to image processing. Due to its ability of detecting the slightest anomalies, computer vision is also used for patient’s scan analysis and increases the accuracy of the image analysis. This, however, is not the only application of the computer vision in the healthcare. A company, for example, developed a software that is able to detect the patient’s blood loss in real time – all thanks to the computer vision. This proves that if a company has an innovative idea and uses the corresponding technology, it can make an impact on the future of healthcare.
Machine learningA subset of AI, machine learning, is capable of learning and recognizing certain patterns in the data and building predictive models upon that. This technology is widely deployed in the financial industry but in healthcare it plays a crucial role as well, helping medical specialists predict and identify the possible disease and prevent its development. The deployment of machine learning in healthcare plays an important role indeed as many illnesses always go unnoticed during the first stage and it may become too late in the future. But with the accurate predictive models, doctors will have a bigger chance to fight cancer, for example, rather than treating the already progressing illness.
Trends for 2023 The Internet of Medical ThingsThe Internet of Things is slowly turning into the IoMT – the Internet of Medical Things. This term implies a wide adaptation of smart and medical wearables, including ECG monitors. It is expected that the number of deployed IoMT devices will range between 20 – 30 billion by 2023, according to Allied Market Research. However, IoMT still has several issues to resolve: like the question of security or efficient communication between the devices.
AR and VRFirst, AR is great for educating medical students and putting them into the environments that would be maximally close to real-life situations. But that’s not the only way to use this (or VR) technology. In psychology, the usage of VR with the patients helps to put them in a situation that would be unavailable in the current environment, thus helping to unlock certain memories or identify the pain points. As for AR, it is used to identify the patient’s veins and place a digital map over it, thus making it easier for clinicians to identify a spot for injection.
Big opportunity for tech companiesFor technological and app development companies today is the perfect time to join the race and come up with an innovative and efficient solution that can be offered to the healthcare industry. As healthcare is ready to adopt cutting-edge technology, the biggest issue for the development company is to create a solution that indeed will be useful and helpful. As for the medical institutions and healthcare companies, they need to find a reliable and trustworthy development company with a proven record of developing and completing successful and healthcare-related projects. Because healthcare is aimed at one’s well-being, there is no room for a mistake at all. So the reputation and experience of a development company indeed will play the biggest role when healthcare representatives will be choosing their provider.
Difference Between Deep Learning And Neural Network
Deep learning and neural networks are both machine learning methods that are used to identify patterns and make predictions. While the two terms are often used interchangeably, there are important differences between them that can have significant implications for their use.
What is Deep Learning?Deep learning is a broader category of machine learning that encompasses neural networks and other approaches. Deep learning involves training models to recognize patterns in data by processing multiple layers of information. These models can learn from vast amounts of data and can recognize patterns that are too complex for humans to identify.
What is Neural Network?The term “Neural Networks” is used to describe a system of virtual neurons or nodes that is loosley modelled after the neural networks that make up the brains of various animals. A lot of today’s AI has its roots in this technique. In fact, research indicates that the current implications and applications of AI are just the result of the evolution of the special qualities of neural networks (such as machine learning, deep learning, etc.).
Computer science, physics, information science, psychology, and engineering all have a hand in developing and refining the neural network paradigm. Neural networks are networks of nodes whose functioning is inspired by animal neurons but only in a very general way. Neural networks are widely employed in many fields today, from issue solving and consumer research to data validation and sales forecasting and risk management.
Differences: Deep Learning and Neural NetworkOne of the key differences between neural networks and deep learning is their complexity. Neural networks are relatively simple compared to deep learning models. They typically consist of a single layer of neurons that are connected to each other. These networks are effective at recognizing simple patterns in data, but they are not able to handle complex data sets.
Deep learning models, on the other hand, can process multiple layers of data and can recognize complex patterns that are not immediately visible to humans. This makes them ideal for applications like image and speech recognition, where the data is highly complex and requires sophisticated processing.
Another important difference between neural networks and deep learning is the way that they are trained. Neural networks are typically trained using a process called backpropagation, which involves adjusting the weights of the connections between neurons based on the error between the predicted output and the actual output.
Deep learning models, on the other hand, are trained using a process called gradient descent, which involves adjusting the weights of the connections between layers of neurons based on the gradient of the error function. This allows the model to learn more complex patterns in the data and to make more accurate predictions.
The following table highlights the major differences between Neural Network and Deep Learning −
Characteristics
Neural Network
Deep Learning
Concept
Neural network, also called artificial neural network, is an information processing model that stimulates the mechanism of learning biological organisms.
It is inspired by the idea of how the nervous system operates. The nervous system contains cells which are referred to as neurons.
Similarly, neural networks consist of nodes which mimic the biological function of neurons.
Deep learning, on the other hand, is much broader concept than artificial neural networks and includes several different areas of connected machines.
Deep learning is an approach to AI and a technique that enables computer systems to improve with experience and data.
Architecture
Neural networks are simple architectural models based on how the nervous system works and are divided into single-layer and multi-layer neural networks. The simple instantiation of a neural network is also referred to as the perceptron.
In the single-layer network, a set of inputs is mapped directly onto an output using generalized variation of a linear function. In multi-layer networks, as the name suggests, the neurons are arranged in layers, in which a layer of neutrons is sandwiched between the input layer and output layer, which is called the hidden layer.
Deep learning architecture, on the other hand, is based on artificial neural networks.
Applications
Neural networks allow modeling of non-linear processes, so they make great tools for solving several different problems such as classification, pattern recognition, clustering, prediction and analysis, control and optimization, machine translation, decision making, machine learning, deep learning and more.
Deep learning models can be applied to various fields including speech recognition, natural language processing, self-driving vehicles, computer-aided diagnosis, voice assistant, sound creation, robotics, computer games, image recognition, brain cancer detection, social network filtering, pattern recognition, biomedicine, and more.
ConclusionThe main difference between deep learning and neural networks is the complexity of the models and the way that they are trained. Neural networks are simpler and more limited in their capabilities, while deep learning models are more complex and can handle more complex data sets.
Both approaches have their strengths and weaknesses, and the choice between them will depend on the specific application and the type of data that is being analyzed.
What Are The Applications Of Electrolysis?
What is Electrolysis?
The process in which ionic substances are decomposed into simple substances by passing an electric current through them is known as electrolysis.
In other words, the process based on the fact that electrical energy can produce chemical changes is known as electrolysis.
Applications of ElectrolysisNowadays the electrolytic process is widely used in various industrial applications. The major applications of the electrolysis are given below.
Extraction of Metal from their Ores
The electrolytic process is used for extracting out the pure metal from their ores, this process is known as electro-extraction. In the electro-extraction, the metal ore is treated with strong acid or is melted and then a DC current is passed through the resulting solution, the solution is decomposed and pure metal is deposited on the cathode.
Refining of Metals
Electrolysis is also used for refining of metals and the process is termed as electro-refining. In electro-refining, the anode of impure metal is placed in a suitable electrolytic solution. When DC current is passed through the solution, pure metal is deposited on the cathode.
Manufacturing of Chemicals
The electrolytic process is also used for manufacturing of various chemicals. When an electric current is passed through the solution of some compound, the compound gets breakdown into its constituent components which are liberated at the anode and cathode, which in turn can be collected.
Electro-Deposition
The electro-deposition is an electrolytic process, in which one metal is deposited over the other metal or non-metal. The electro-deposition is usually used for the decorative, protective and functional purposes.
Electroplating
An electrolytic process in which a metal is deposited over any metallic or non-metallic surface is called the electroplating. Electroplating is usually used to protect the metals from corrosion by atmospheric air and moisture.
Electro-deposition of Rubber
Electrolysis is also employed for electro-deposition of rubber. The rubber latex obtained from the tree consists of very fine colloidal particles of rubber suspended in water. These particles of rubber are negatively charged. On electrolysis of the solution, these rubber particles move towards the anode and deposit on it.
Electro-Metallization
The electrolytic process in which the metal is deposited on a conducting base for decorative and for protective purposes is termed as electro-metallization. Also, by using the electro-metallization process, any non-conductive base is made conductive by depositing a layer of graphite over it.
Electro-Facing
An electrolytic process in which a metallic surface is coated with a harder metal by electro-deposition in order to increase its durability is known as electro-facing.
Electro-Forming
Electrolysis is also used for electro-forming, it is the reproduction of an object by electro-deposition in order to increase its durability.
In the electro-forming, i.e. reproduction of medals, coins, etc., a mould is made by impressing the object in wax. The wax surface having exact impression of the object is coated by powdered graphite to make it conducting. This mould is then dipped in an electro-forming cell as cathode. After obtaining a coating of desired thickness, the article is removed and the wax core is melted out of the metal shell.
Electro-Typing
The electrotyping is an electrolytic process for forming metal parts that exactly reproduce a model. It is a special application of electro-forming and is mainly used to reproduce printing, set up tying, medals, etc.
Anodizing
The electrolysis process of deposition of an oxide film on a metal surface is known as anodizing. It is mainly used to increase the thickness of the natural oxide layer on the surface of the metal parts.
Electro-Polishing
The electro-polishing is an electrolytic process that removes materials from a metallic workpiece. It is also known as electrochemical polishing or electrolytic polishing.
Electro-polishing uses a combination of rectified current and a blended chemical electrolyte bath to remove flaws from the surface of a metal part.
Electro-Refining
Electro-refining is a method for purifying a metal using electrolysis. In the electro-refining process, the anode is made of impure metal and the impurities must be lost during the passage of metal from the anode to cathode during electrolysis.
Electro-Parting
An electrolytic process of separation of two or more metals is known as electro-parting or electro-stripping.
Electro-Cleaning
Electro-cleaning is the process of removing soil, scale or corrosion from a metallic surface. It is also known as electro-pickling. It is a form of electroplating which can be applied to all electrically conductive materials.
Playing Super Mario Bros With Deep Reinforcement Learning
This article was published as a part of the Data Science Blogathon
IntroductionFrom this article, you will learn how to play Super Mario Bros with Deep Q-Network and Double Deep Q-Network (with code!).
Photo by Cláudio Luiz Castro on Unsplash
Super Mario Bros is a well-known video game title developed and published by Nintendo in the 1980s. It is one of the classical game titles that lived through the years and need no explanations. It is a 2D side-scrolling game, allowing the player to control the main character — Mario.
The game environment was taken from the OpenAI Gym using the Nintendo Entertainment System (NES) python emulator. In this article, I will show how to implement the Reinforcement Learning algorithm using Deep Q-Network (DQN) and Deep Double Q- Network (DDQN) algorithm using PyTorch library to examine each of their performance. The experiments conducted on each algorithm were then evaluated.
Data understanding and preprocessingThe original observation space for Super Mario Bros is 240 x 256 x 3 an RGB image. And the action space is 256 which means the agent is able to take 256 different possible actions. In order to speed up the training time of our model, we have used the gym’s wrapper function to apply certain transformations to the original environment:
Repeating each action of the agent over 4 frames and reducing the video frame size, i.e. each state in the environment is a 4 x 84 x 84 x 1 (a list of 4 continuous 84 x 84 grayscale pixel frames)
Normalizing the pixel value to the range from 0 to 1
Reducing the number of actions to 5 (Right only), 7 (Simple movement) and 12 (Complex movement)
Theoretical ResultsInitially, I am thinking to perform an experiment using Q-learning which uses a 2-D array to store all possible combinations of state and action pair values. However, in this environment setting, I realized that it is impossible to apply Q-learning as there is a requirement to store a very large Q-table and this is not feasible.
Therefore, this project used the DQN algorithm as the baseline model. DQN algorithms use Q- learning to learn the best action to take in the given state and a deep neural network to estimate the Q- value function.
The type of deep neural network I used is a 3 layers convolutional neural network followed by two fully connected linear layers with a single output for each possible action. This network works like the Q-table in the Q-Learning algorithm. The objective loss function we used is Huber loss or smoothed mean absolute error on Q-values. Huber loss combines both MSE and MAE to minimize the objective function. The optimizer we used to optimize the objective function is Adam.
However, the DQN network has the problem of overestimation.
Figure 1: Illustration of how the DQN network is overestimated
There are 2 main reasons for overestimation as shown in Fig1. The first reason is due to the maximization function used to calculate the target value. Let’s assume the True action-values are denoted by: x(a₁) … x(aₙ). Noisy estimations made by DQN are denoted by Q(s,a₁;w), … Q(s, aₙ;w) Mathematically,
therefore it is an overestimation over true Q-value.
The second reason is that overestimated Q-values are again being used to update the weights of the Q-Network through backward propagation. This made overestimation more severe.
The main drawback of the overestimation is due to non-uniform overestimation done by DQN. The intuition is that the more frequent a specific state, action pairs appear in the replay buffer, the more overestimation is done on that state-action pairs.
To obtain a more accurate Q-value, we would like to use the DDQN network on our problem then compare the experiment result against the previous DQN network. To alleviate the overestimation caused by maximization, DDQN uses 2 Q-network, one for getting actions and another for updating the weights through backpropagation. The DDQN Q-learning update equation is:
Q* is the one for updating weight and Q^ is the one for getting actions for a specific state. Q^ simply copies the values of Q* every n step.
Experiment ResultsThere were 5 experiments conducted based on different movements of the agent using 2 algorithms, DQN and DDQN. The different movements are complex movements, simple movements, and right-only movements.
The parameters settings are as follows :
Observation space: 4 x 84 x 84 x 1
Action space: 12 (Complex Movement) or 7 (Simple Movement) or 5 (Right only movement)
Loss function: HuberLoss with δ = 1
Optimizer: Adam with lr = 0.00025
betas = (0.9, 0.999)
Batch size = 64 Dropout = 0.2
gamma = 0.9
Max memory size for experience replay = 30000
For epsilon greedy: Exploration decay = 0.99, Exploration min = 0.05
At the beginning of exploration, max = 1, the agent will take random action. After each episode, it will decay by the exploration decay rate until it reaches an exploration min of 0.05.
Experiment 1The first experiment conducted was to compare DDQN and DQN algorithms for the complex movement of the agent.
Experiment 2 Experiment 3From the above 3 experiment results, we can see that in all cases, DQN’s performance at episode 10,000 is approximately the same as DDQN’s performance at episode 2,000. So, we can conclude that the DDQN network helps to eliminate the overestimation problem caused by the DQN network.
A further experiment conducted using DDQN and DQN for the 3 different movements.
Experiment 4The fourth experiment conducted was using the DDQN algorithm on all 3 different movements.
Experiment 5From the above 2 experiment results, we can conclude that the network is able to train better on right-only movement action space which only allows the agent to move to the right.
Codes import torch import chúng tôi as nn import random from nes_py.wrappers import JoypadSpace import gym_super_mario_bros from tqdm import tqdm import pickle from gym_super_mario_bros.actions import RIGHT_ONLY, SIMPLE_MOVEMENT, COMPLEX_MOVEMENT import gym import numpy as np import collections import cv2 import matplotlib.pyplot as plt %matplotlib inline import time import pylab as pl from IPython import display class MaxAndSkipEnv(gym.Wrapper): """ Each action of the agent is repeated over skip frames return only every `skip`-th frame """ def __init__(self, env=None, skip=4): super(MaxAndSkipEnv, self).__init__(env) # most recent raw observations (for max pooling across time steps) self._obs_buffer = collections.deque(maxlen=2) self._skip = skip def step(self, action): total_reward = 0.0 done = None for _ in range(self._skip): obs, reward, done, info = self.env.step(action) self._obs_buffer.append(obs) total_reward += reward if done: break max_frame = np.max(np.stack(self._obs_buffer), axis=0) return max_frame, total_reward, done, info def reset(self): """Clear past frame buffer and init to first obs""" self._obs_buffer.clear() obs = self.env.reset() self._obs_buffer.append(obs) return obs class MarioRescale84x84(gym.ObservationWrapper): """ Downsamples/Rescales each frame to size 84x84 with greyscale """ def __init__(self, env=None): super(MarioRescale84x84, self).__init__(env) self.observation_space = gym.spaces.Box(low=0, high=255, shape=(84, 84, 1), dtype=np.uint8) def observation(self, obs): return MarioRescale84x84.process(obs) @staticmethod def process(frame): if chúng tôi == 240 * 256 * 3: img = np.reshape(frame, [240, 256, 3]).astype(np.float32) else: assert False, "Unknown resolution." # image normalization on RBG img = img[:, :, 0] * 0.299 + img[:, :, 1] * 0.587 + img[:, :, 2] * 0.114 resized_screen = cv2.resize(img, (84, 110), interpolation=cv2.INTER_AREA) x_t = resized_screen[18:102, :] x_t = np.reshape(x_t, [84, 84, 1]) return x_t.astype(np.uint8) class ImageToPyTorch(gym.ObservationWrapper): """ Each frame is converted to PyTorch tensors """ def __init__(self, env): super(ImageToPyTorch, self).__init__(env) old_shape = self.observation_space.shape self.observation_space = gym.spaces.Box(low=0.0, high=1.0, shape=(old_shape[-1], old_shape[0], old_shape[1]), dtype=np.float32) def observation(self, observation): return np.moveaxis(observation, 2, 0) class BufferWrapper(gym.ObservationWrapper): """ Only every k-th frame is collected by the buffer """ def __init__(self, env, n_steps, dtype=np.float32): super(BufferWrapper, self).__init__(env) self.dtype = dtype old_space = env.observation_space self.observation_space = gym.spaces.Box(old_space.low.repeat(n_steps, axis=0), old_space.high.repeat(n_steps, axis=0), dtype=dtype) def reset(self): self.buffer = np.zeros_like(self.observation_space.low, dtype=self.dtype) return self.observation(self.env.reset()) def observation(self, observation): self.buffer[:-1] = self.buffer[1:] self.buffer[-1] = observation return self.buffer class PixelNormalization(gym.ObservationWrapper): """ """ def observation(self, obs): return np.array(obs).astype(np.float32) / 255.0 def create_mario_env(env): env = MaxAndSkipEnv(env) env = MarioRescale84x84(env) env = ImageToPyTorch(env) env = BufferWrapper(env, 4) env = PixelNormalization(env) return JoypadSpace(env, SIMPLE_MOVEMENT) class DQNSolver(nn.Module): """ Convolutional Neural Net with 3 conv layers and two linear layers """ def __init__(self, input_shape, n_actions): super(DQNSolver, self).__init__() chúng tôi = nn.Sequential( nn.Conv2d(input_shape[0], 32, kernel_size=8, stride=4), nn.ReLU(), nn.Conv2d(32, 64, kernel_size=4, stride=2), nn.ReLU(), nn.Conv2d(64, 64, kernel_size=3, stride=1), nn.ReLU() ) conv_out_size = self._get_conv_out(input_shape) chúng tôi = nn.Sequential( nn.Linear(conv_out_size, 512), nn.ReLU(), nn.Linear(512, n_actions) ) def _get_conv_out(self, shape): o = self.conv(torch.zeros(1, *shape)) return int(np.prod(o.size())) def forward(self, x): conv_out = self.conv(x).view(x.size()[0], -1) return self.fc(conv_out) class DQNAgent: def __init__(self, state_space, action_space, max_memory_size, batch_size, gamma, lr, dropout, exploration_max, exploration_min, exploration_decay, double_dqn, pretrained): # Define DQN Layers self.state_space = state_space self.action_space = action_space self.double_dqn = double_dqn self.pretrained = pretrained self.device = 'cuda' if torch.cuda.is_available() else 'cpu' # Double DQN network if self.double_dqn: self.local_net = DQNSolver(state_space, action_space).to(self.device) self.target_net = DQNSolver(state_space, action_space).to(self.device) if self.pretrained: self.local_net.load_state_dict(torch.load("DQN1.pt", map_location=torch.device(self.device))) self.target_net.load_state_dict(torch.load("DQN2.pt", map_location=torch.device(self.device))) self.optimizer = torch.optim.Adam(self.local_net.parameters(), lr=lr) chúng tôi = 5000 # Copy the local model weights into the target network every 5000 steps chúng tôi = 0 # DQN network else: chúng tôi = DQNSolver(state_space, action_space).to(self.device) if self.pretrained: self.dqn.load_state_dict(torch.load("DQN.pt", map_location=torch.device(self.device))) self.optimizer = torch.optim.Adam(self.dqn.parameters(), lr=lr) # Create memory self.max_memory_size = max_memory_size if self.pretrained: self.STATE_MEM = torch.load("STATE_MEM.pt") self.ACTION_MEM = torch.load("ACTION_MEM.pt") self.REWARD_MEM = torch.load("REWARD_MEM.pt") self.STATE2_MEM = torch.load("STATE2_MEM.pt") self.DONE_MEM = torch.load("DONE_MEM.pt") with open("ending_position.pkl", 'rb') as f: self.ending_position = pickle.load(f) with open("num_in_queue.pkl", 'rb') as f: self.num_in_queue = pickle.load(f) else: self.STATE_MEM = torch.zeros(max_memory_size, *self.state_space) self.ACTION_MEM = torch.zeros(max_memory_size, 1) self.REWARD_MEM = torch.zeros(max_memory_size, 1) self.STATE2_MEM = torch.zeros(max_memory_size, *self.state_space) self.DONE_MEM = torch.zeros(max_memory_size, 1) self.ending_position = 0 self.num_in_queue = 0 self.memory_sample_size = batch_size # Learning parameters self.gamma = gamma self.l1 = nn.SmoothL1Loss().to(self.device) # Also known as Huber loss self.exploration_max = exploration_max self.exploration_rate = exploration_max self.exploration_min = exploration_min self.exploration_decay = exploration_decay def remember(self, state, action, reward, state2, done): """Store the experiences in a buffer to use later""" self.STATE_MEM[self.ending_position] = state.float() self.ACTION_MEM[self.ending_position] = action.float() self.REWARD_MEM[self.ending_position] = reward.float() self.STATE2_MEM[self.ending_position] = state2.float() self.DONE_MEM[self.ending_position] = done.float() self.ending_position = (self.ending_position + 1) % self.max_memory_size # FIFO tensor self.num_in_queue = min(self.num_in_queue + 1, self.max_memory_size) def batch_experiences(self): """Randomly sample 'batch size' experiences""" idx = random.choices(range(self.num_in_queue), k=self.memory_sample_size) STATE = self.STATE_MEM[idx] ACTION = self.ACTION_MEM[idx] REWARD = self.REWARD_MEM[idx] STATE2 = self.STATE2_MEM[idx] DONE = self.DONE_MEM[idx] return STATE, ACTION, REWARD, STATE2, DONE def act(self, state): """Epsilon-greedy action""" if self.double_dqn: chúng tôi += 1 if random.random() < self.exploration_rate: return torch.tensor([[random.randrange(self.action_space)]]) if self.double_dqn: # Local net is used for the policy return torch.argmax(self.local_net(state.to(self.device))).unsqueeze(0).unsqueeze(0).cpu() else: return torch.argmax(self.dqn(state.to(self.device))).unsqueeze(0).unsqueeze(0).cpu() def copy_model(self): """Copy local net weights into target net for DDQN network""" self.target_net.load_state_dict(self.local_net.state_dict()) def experience_replay(self): """Use the double Q-update or Q-update equations to update the network weights""" if self.double_dqn and chúng tôi % chúng tôi == 0: self.copy_model() return # Sample a batch of experiences STATE, ACTION, REWARD, STATE2, DONE = self.batch_experiences() STATE = STATE.to(self.device) ACTION = ACTION.to(self.device) REWARD = REWARD.to(self.device) STATE2 = STATE2.to(self.device) DONE = DONE.to(self.device) self.optimizer.zero_grad() if self.double_dqn: # Double Q-Learning target is Q*(S, A) <- r + γ max_a Q_target(S', a) target = REWARD + torch.mul((self.gamma * self.target_net(STATE2).max(1).values.unsqueeze(1)), 1 - DONE) current = self.local_net(STATE).gather(1, ACTION.long()) # Local net approximation of Q-value else: # Q-Learning target is Q*(S, A) <- r + γ max_a Q(S', a) target = REWARD + torch.mul((self.gamma * self.dqn(STATE2).max(1).values.unsqueeze(1)), 1 - DONE) current = self.dqn(STATE).gather(1, ACTION.long()) loss = self.l1(current, target) loss.backward() # Compute gradients self.optimizer.step() # Backpropagate error self.exploration_rate *= self.exploration_decay # Makes sure that exploration rate is always at least 'exploration min' self.exploration_rate = max(self.exploration_rate, self.exploration_min) def show_state(env, ep=0, info=""): """While testing show the mario playing environment""" plt.figure(3) plt.clf() plt.imshow(env.render(mode='rgb_array')) plt.title("Episode: %d %s" % (ep, info)) plt.axis('off') display.clear_output(wait=True) display.display(plt.gcf()) def run(training_mode, pretrained, double_dqn, num_episodes=1000, exploration_max=1): env = gym_super_mario_bros.make('SuperMarioBros-1-1-v0') env = create_mario_env(env) # Wraps the environment so that frames are grayscale observation_space = env.observation_space.shape action_space = env.action_space.n agent = DQNAgent(state_space=observation_space, action_space=action_space, max_memory_size=30000, batch_size=32, gamma=0.90, lr=0.00025, dropout=0.2, exploration_max=1.0, exploration_min=0.02, exploration_decay=0.99, double_dqn=double_dqn, pretrained=pretrained) # Restart the enviroment for each episode num_episodes = num_episodes env.reset() total_rewards = [] if training_mode and pretrained: with open("total_rewards.pkl", 'rb') as f: total_rewards = pickle.load(f) for ep_num in tqdm(range(num_episodes)): state = env.reset() state = torch.Tensor([state]) total_reward = 0 steps = 0 while True: if not training_mode: show_state(env, ep_num) action = agent.act(state) steps += 1 state_next, reward, terminal, info = env.step(int(action[0])) total_reward += reward state_next = torch.Tensor([state_next]) reward = torch.tensor([reward]).unsqueeze(0) terminal = torch.tensor([int(terminal)]).unsqueeze(0) if training_mode: agent.remember(state, action, reward, state_next, terminal) agent.experience_replay() state = state_next if terminal: break total_rewards.append(total_reward) if ep_num != 0 and ep_num % 100 == 0: print("Episode {} score = {}, average score = {}".format(ep_num + 1, total_rewards[-1], np.mean(total_rewards))) num_episodes += 1 print("Episode {} score = {}, average score = {}".format(ep_num + 1, total_rewards[-1], np.mean(total_rewards))) # Save the trained memory so that we can continue from where we stop using 'pretrained' = True if training_mode: with open("ending_position.pkl", "wb") as f: pickle.dump(agent.ending_position, f) with open("num_in_queue.pkl", "wb") as f: pickle.dump(agent.num_in_queue, f) with open("total_rewards.pkl", "wb") as f: pickle.dump(total_rewards, f) if agent.double_dqn: torch.save(agent.local_net.state_dict(), "DQN1.pt") torch.save(agent.target_net.state_dict(), "DQN2.pt") else: torch.save(agent.dqn.state_dict(), "DQN.pt") torch.save(agent.STATE_MEM, "STATE_MEM.pt") torch.save(agent.ACTION_MEM, "ACTION_MEM.pt") torch.save(agent.REWARD_MEM, "REWARD_MEM.pt") torch.save(agent.STATE2_MEM, "STATE2_MEM.pt") torch.save(agent.DONE_MEM, "DONE_MEM.pt") env.close() # For training run(training_mode=True, pretrained=False, double_dqn=True, num_episodes=1, exploration_max = 1) # For Testing run(training_mode=False, pretrained=True, double_dqn=True, num_episodes=1, exploration_max = 0.05)Conclusion
DDQN takes much less episode to train in comparison with DQN. Thus, the DDQN network helps to eliminate overestimation issues found in the DQN network. Both DQN and DDQN networks are able to train better on right-only movement compared to simple & complex movement action space.
About the AuthorConnect with me on LinkedIn Here.
Thanks for giving your time!
The media shown in this article are not owned by Analytics Vidhya and are used at the Author’s discretion.
Related
Update the detailed information about Discover The Applications Of Deep Learning In Healthcare on the Eastwest.edu.vn website. We hope the article's content will meet your needs, and we will regularly update the information to provide you with the fastest and most accurate information. Have a great day!