Transfer learning

• 1. The First Neural Network Paper

  ›

  In 1976, Stevo Bozinovski and Ante Fulgosi published a paper addressing transfer learning in neural network training. Their work appeared in the Proceedings of Symposium Informatica 3-121-5 held in Bled. The original text was written in Croatian before translation efforts began. This document provided both a mathematical model and a geometrical model for the topic. It marked the earliest known attempt to describe how knowledge from one task could help another. A decade later, Bozinovski released a reminder about this first paper in 2020 within the journal Informica. That retrospective piece spanned pages 291 through 302 of volume 44. By 1981, researchers tested these ideas on actual image datasets containing letters from computer terminals. These experiments showed that transfer learning could produce positive results or negative ones depending on the setup.

  45

• 2. Domains And Predictive Functions

  ›

  A domain consists of a feature space and a marginal probability distribution where the two elements interact. Given a specific domain labeled D_s, a task requires two components: a label space Y and an objective predictive function f. The function f predicts the corresponding label y of a new instance x. This task T_s is learned from training data consisting of pairs (x_i, y_i). When comparing a source domain D_s with a target domain D_t, transfer learning aims to improve the learning of the target predictive function f_t. The improvement happens using knowledge found in D_s and T_s. If D_s equals D_t but T_s differs from T_t, the method still applies. Alternatively, if T_s equals T_t but D_s differs from D_t, the approach remains valid. The goal always centers on boosting performance for the target task.

• 3. Algorithms And Network Types

  ›

  Algorithms for transfer learning exist within Markov logic networks and Bayesian networks. These structures allow systems to move information between different problem spaces. Researchers applied these methods to cancer subtype discovery during the 32nd Conference on Neural Information Processing Systems in Montréal. That event took place in 2018 and featured work by Hajiramezanali, Dadaneh, Karbalayghareh, Zhou, and Qian. Other applications included building utilization and general game playing as described in a 2007 IJCAI paper by Bikramjit Banerjee and Peter Stone. Text classification tasks also benefited from these techniques alongside digit recognition efforts. Medical imaging projects used similar strategies while spam filtering operations adopted them too. Convolutional neural networks showed improved accuracy when exposed to another domain before any formal learning began. Results reached higher asymptotes at the end of the process compared to standard random weight distribution.

  1823

• 4. Muscle Signals And Brainwaves

  ›

  In 2020, researchers discovered that electromyographic signals from muscles could transfer knowledge to electroencephalographic brainwave classifications. This connection existed because both signal types shared similar physical natures. The system moved data from gesture recognition domains into mental state recognition domains. Experiments confirmed this relationship worked bidirectionally since EEG could classify EMG behaviors just as well. Neural network accuracy improved prior to any learning phase against standard random weight distributions. Final results showed better performance at the asymptote stage after exposure to another domain. End-users could change fully-connected layer structures to further boost outcomes. These findings demonstrated how cross-domain transfers enhanced predictive capabilities across biological signal types.

  23

• 5. Commercial Drivers And Self-Training

  ›

  Andrew Ng stated during his NIPS 2016 tutorial that transfer learning would become the next driver of machine learning commercial success. He positioned it as following the era dominated by supervised learning. By 2020, Zoph and colleagues published a paper titled Rethinking Pre-Training and self-training. Their report claimed pre-training could sometimes hurt accuracy in specific scenarios. They advocated for self-training methods instead of traditional pre-training approaches. This shift reflected changing priorities within the field toward more adaptive strategies. Recent surveys from 2009 and 2019 highlighted evolving theoretical foundations alongside practical applications. Cost-sensitive machine learning concepts remained relevant throughout these developments. Multi-objective optimization frameworks continued to support complex transfer tasks across industries.

  1012