AI / ML - History of Activations

 In one of my favorite film (Tamil) which deals with the greatness of a 5th Century  spiritual master  (Bodhi Dharma),  there is a wonderful dialogue - Film - Aezhaam Arivu (which means "7th sense")

" We started losing our science when we started ignoring our history and heritage"

***********************************************************************************

 In 1943, "Step" function was used to have the early neural network just to enable "fire" or "don't fire"  in other word - "give output or keep quiet based on the computation made" (no details covered in this post please). You can appreciate this is the most simplistic way of looking at things but the data scientists those days were sincere to mimick their limited understanding of human brain which they observed that only few of the neurons of the brain were activated at any point of time.

Understandably this milestone made their initial neural network to continue the data processing in neural networks - or what is referred as "forward propagation". However, it lacked the ability to support the reverse calculation of the neural network automatically which is critical to optimize the output. Can you believe the AI experts used to manually work on derivatives (calculus) to handle the back propagation since that was not enabled by "step" function ? It was cumbersome but there were no choice for them.

During 1970 - 80, a breakthrough was achieved by using sigmoid function in the basic neural networks (also referred as shallow networks which did not have many layers) which helped  optimization when the concept of gradient descent (a method to do the reverse calculation / backward propagation automatically) was developed. 

However, since - as we know - sigmoid function returns a value 0 to 1, there were challenges of "vanishing gradients" during the backward propagation process so the whole idea of optimization was stuck up. In the year 1980, a much better activation called tanh function started getting used which gives an output in the range -1 to 1 which helped to avoid the earlier challenge substantially. However still when there are situations of large values of inputs or very small values of inputs, optimization still suffered.

After quite a while which also marked the advent of "deep learning era" (supported by deep neural networks (allowing multiple hidden layers and large language models), during 2010, RELU activation function  (Proud to say one of the co-founder was Vinod Nair - though he lives in Canada) . This activation is very simple (output either maximum of the input value or zero which ever is higher) and the computational cost was minimal.  It has almost  taken out the woes of optimization and was a huge shot in the arm of deep learning. Even today, though there are many other sophisticated activation functions available, when in doubt or unsure, developing community goes for ReLU for hidden layer activation as a safe bet. Is it the best one available today ? It still has issue with returning values zeros some times.

During 2011-15, a very smart variation of RELU (returns either input value or .001 of input value) named as "Leaky Relu" was introduced to adjust against the issues faced by zero value of the computed value since now it wont return zero any more but a tiny value instead  and keep the backward propagation going with non zero values ! 

In parallel, we had softmax function introduced in deep neural network based on the need to return a probablistic output for a chosen set of values. We should be clear that this was more out of need than any logical progression in the history described so far.

After 2015, we have Swish, Mish , GELU and so on  which has made smoother activation possible for ultra deep networks ..and this is not an exhaustive list of activation functions.

 Well, my constant companion ChatGPT gave a nice idea to remember the history for people over 40 years of age (when obviously the neurons start dying quite rapidly)

*************************************************************************

 "Some Teachers Run Like Super Geniuses"

S = Step & Sigmoid

T = Tanh

R = ReLU

L = Leaky ReLU

S = SWISH

G = GELU

**************************************************************************

 Another Memory Anchor - 

First they 'stepped' (binary), then made it smooth (Sigmoid), then centered (tanh), then said 'lets forget curves, just 'cut'' (ReLU), fixed 'dying neurons' (Leaky ReLU) and finally started 'smart, curvy activations' (SWISH, GELU)

 

 

 

 

 

 

 

AI/ML ----> Paradox - "part 2" - Neural Network Activations

So, the moral of today's blog is that we get out of challenges only to get caught into newer issues. :-)  That is how human evolution too happened right ?? 

Did  by mistake the postscript of previous post  is copy pasted here ? When we deal with aspects of  truth which is so overwhelming, is it not natural to repeat, re-emphasis  and reiterate ?  

Paradox and Contradictions is in all aspects of our lives - so it is quite easy to understand few nuances of AI/ML if we relate it with this grand truth. I am continuing the same theme of last post in this post albeit with a different topic.

So lets deal with "Activation" function which is a "basic" concept in neural networks.

This concept has become a familiar one in machine learning era also when ML engineers started experimenting with neural networks as early as 1943 ! Before going down the memory lane let me give a contextual understanding so that switching gears will be smoother.

 The essential difference between the traditional Machine learning models - like Linear regression (which uses linear equation) & logistics regression (which uses sigmoid function)  and neural network is  what is referred as "hidden layers". Neural network uses hidden layers while the traditional ML models were not sophisticated enough to use it.If there is just 1 hidden layer we call it as "shallow" neural network positioned between input layer and output layer. Hidden layer may contain one or many neurons. (well, it could be a very rare situation to have just 1 neuron in a hidden layer but still technically feasible if practically required). 

Essentially what does such hidden layers do ? 

The hidden layers take an input value from previous layer, process it and pass on its output to next layer. As simple as that. 

Processing here essentially means two components - (1) computing operation (which is referred to calculating Z value) and (2) transforming that value (with the help of an activation function)

 The funny thing about activation is that we have multiple choices based on the requirements - the purpose of model, the volume and pattern of data, the complexity of design and also aspects like budget & infrastructure availability. These days the default activation function used is RELU (which is acronym of Rectified Linear Unit) but there are many other smart activation available too.

Now let me zero in on the intent of this post - paradox - isn't it ? When we say there are multiple options today, the paradox is that we don't use sigmoid function and linear function as activation layer in hidden layers.  Both of them are relevant only for output layers and sigmoid was used until a point of time when RELU was made public (Linear function was never attempted to be used in hidden layer since it won't "transform" the output. why to create hidden layers then ?)

I wanted to go deeper into the history of these activation functions but don't want to make this post any longer since I hit the bull's eye already. Do you see the paradox. The original functions that started off machine learning are out-dated and become useless with modern neural networks. In the next post let me explain the history of various activation functions & after that would like to jump in for another sequel to lament about philosophical aspect of this "activation" and what is the purpose of "activation"  in grand scheme of things.

By the way, with my limited exposure to AI/ML, I am convinced that this subject is not for those who have a dis-taste for philosophy. Come on, if we don't see the connection between the most materialistic things and the abstract aspects of life, I am sorry, you are missing life.

 So Lets jump to philosophy after exploring history.. ok ? Stay tuned.

 

 

 

                    

 

AI / ML ---> Amusing Paradox

"One of my PhD students from Stanford, many years after he'd already graduated from Stanford, once said to me that while he was studying at Stanford, he learned about bias and variance and felt like he got it, he understood it. But that subsequently, after many years of work experience in a few different companies, he realized that bias and variance is one of those concepts that takes a short time to learn, but takes a lifetime to master. Those were his exact words. Bias and variance is one of those very powerful ideas"  - Andrew N.G

 Let me talk about one amusing concept today in ML/AI learning - the trade off between Bias and  Variance.

Essentially all these Machine Learning / AI models consist of 3 steps - Create a model, train the same, Test/Validate it before actual release. Out of the entire data set, always a portion is reserved for testing / validation - generally referred as Dev set. That's the fundamental knowledge that you will need to go thru this post.

During Training we may face an issue of "under-fitting" or Bias which means that the model's is not doing so great with the training data. It could be due to simplistic assumptions made OR not considering the key features  / input parameters. Understandably when we train the model, it would come to light  very quick pointing the need for refinement of the model to make it more accurate in training stage itself.

Most of the times, a model quite successful during training  may not produce as good results during validation / testing. This is the other side of the coin - the challenge of Variance problem which is referred as "over-fitting". It may be an isolated issue or also as a consequence of too much of bias-treatment given during training stage.

To given an analogy, a child pampered too much at home (model treatment for "bias" problem) may end up not so-well-behaved in public isn't it ? We may consider the data set reserved for testing / validation to be equivalent to going public (or the future performance) & that goes for a task if we try to handle bias problem too much.

Historically, during pre-neural network days, this was always referred as "Bias-Variance" Trade off when we were using Linear regressions, Simple Decision Trees and K-Means algorithms. For example, in case of Linear regression, if we add a lot of input features, we can reduce bias substantially but we will find it reflecting adversely when we test the model with new set of data since the model gets too much "attached" to the training data set. Similarly when we use a higher order of polynomial function in linear regression, bias problem can be effectively reduced but it will reflect very badly during testing.

 With the Deep Learning era where we have complex and deeper neural network coupled with large volumes of data, are we better off now ? We still have the same issues that exist earlier but the way we handle them has changed with better tools in hand.

To begin with, it has been time and again proved that a deep neural network is capable of handling Bias problem effectively without impacting the variance issue in any manner WHEN we use appropriate regularization factor. While building neural network we don't get into the paradox type of situation any more - for instance, the high level thumb rule is "Solve bias problem with more complex neural network" and there after "Use more volume of data to tackle the variance problem". 

What is stated above is bit simplistic but we have more number of tools in hand - the hyper parameter tuning, batch normalization, drop outs, early stoppages - we are better equipped to handle the Bias Variance issues. There is of course a cost associated to this luxury - well, larger network means more cost and need for better and larger  IT infrastructure. Similarly on the side of having more data,  it may not be always possible to get large data for all kinds of situations.

So, the moral of today's blog is that we get out of challenges only to get caught into newer issues. :-)  That is how human evolution too happened right ?? 

Just musing..... 

 Suren

AI a.k.a Agandam

 A year back, just to cope up with official pressure (?) on up-skilling,  I took up Azure Fundamentals course from Microsoft and as a natural sequel, I also took up Azure AI Fundamentals. Amongst an array of courses offered by MS why I did choose AI ?  

Mainly because I felt AI could alone capture my imagination - the faculty that kept me going until now in spite of growing old - year after year.

Yep.. without having a clue on what AI is capable of, I just picked up a single criteria for choosing to get into it - it sounded creative from my layman's view.

Well, as I look back the past 1 year, I was taking up one course after other from a reputed university (Coursera - who has a tie up with Stanford University) where I learnt pretty much of conceptual knowledge during past 7- 8 months continually. after the MS AI fundamentals course. Every single concept of AI/M I learnt left only with awe and admiration.

Today as I look at this brief journey that I have completed so far, I find myself lot more humble with my learning than feeling "fullness" of what I learnt. May be because, i had a very scientific approach in my learning ? Lemme explain...

 As I was understanding the concepts of AI from the soft-spoken and pleasant-faced Andrew.NG - an expert at Coursera, I was also exploring specific nuances at more detail with the help of Chat GPT and other popular models, I was also trying my hands with Python with a feverish aim to "understand" the codes if not write them myself, I was taking assistance from an overseas nephew who is an AI Engineer and also had the patience to educate me on my silly doubt. Above all, I had the guts to watch Andrej Karpathy's you tube videos  which literally made my jaws wide open. I did not understand every thing he spoke about, but I was drawn helplessly towards his "Zero to Hero" series which showed me the peak of what I am trying to attempt with my baby steps.

May be this kind of varied approach of learning made the entire experience so far quite a joy and instilling me a craving to march further and farther.... and when the time came to take some decisive steps towards my dreams, I am helplessly drawn back to blogging.

Yes around 17-18  years back when I first started my blogging experience with a senior cousin he rightfully titled it as "Jolly Musings" and then I launched this very blog for myself by end of 2008 to continue my tantrums about the existence - so I titled it appropriately as "AGANDAM" - which in Tamil means vastness.

I was dishing out random thoughts, writing poems, wrote film reviews, giving my pearls of wisdom on little things about life but as I see those posts now, I can see it was rudderless. It contained lot of energy and sparks of intelligence, but it was not going any where tangible. Well, Sort of stopped writing 3 years back but then I got reminded today to re-start my blogging - to share my delight on my AI learning, shout out my bliss moments and move towards next plane of my "new" learning endeavour.

Here I am writing this at 1.45 AM on ShivRathri (a day before New moon day) which is meant to be the darkest night of entire month. Yes...Let the blog restart like a spark in this dark night and  glow bigger and bigger with sure-footed march ahead !! 

I am re-christened the name of the blog as "AI alias Agandam" which is just an extension of what I used to brag about my erstwhile name "Agandam". Well, scope is going to be narrowed down now - neigh - rather going to be more focussed

Krishnaarpanam - As always