Single neuron still has a lot to say In the post of the first neural network tutorial, we studied a perceptron as a simple supervised learning machine. The perceptron is an amazing structure to understanding inference. In the post of the first neural network tutorial, I said I would leave you to find the objective function and and draw the plot of it. I just introduce here. Objective function and its contour plot.

Continue reading

Efficient Monte Carlo sampling This post is on the extension of the post about Hamiltonian Monte Carlo method. Therefore, I assume the readers already read the post. Overrelaxation also reduces the random property of the Monte Carlo sampling, and speeds up the convergence of the Markov chain. Gibbs sampling In advance of studying over relaxation, we study Gibbs sampling. In the general case of a system with K variables, a single iteration involves sampling one parameter at a time.

Continue reading

Single neuron is amazing One of the lessons I had during physics program is that we should start to understand small thing deeply however complicated the system which you want to know is. Not just it is easier but also it helps a lot to understand the more complex ones. Neural network is often compared to black magic. We do not understand why and how exactly so effective it is, but it makes great estimations in some specific matters.

Continue reading

Generative model Generative model is a model for generating all variables including outputs. I will give a very simple example with strong assumptions. Data $\boldsymbol{x^{(n)} } $ are generated by an unknown matrix, $\boldsymbol{G}$. $$ \boldsymbol{x} = \boldsymbol{G}~\boldsymbol{s} $$ The goal is to find the source variable $\boldsymbol{s}$. we assume that the number of sources is equal to the number of observations We assume that the latent variables are independently distributed, with marginal distributions We assume that the vector $\boldsymbol{x}$ is generated without noise for simplicity.

Continue reading

I announce over and over that the chronicle ordering of the post are irrelevant for beginners' favor. There are many blanks I skipped. I would fill the holes later. Variational method During my physics coursework and researches, I used this method countlessly. I even had a book of the name. It is quite simple, but also as big topic as being a book. Simply put, it is a technique to find equations and solutions (sometimes approximate solutions) by extremizing functionals which is mainly just integrals of fields, and treat the functions in the integral, as parameters.

Continue reading