Hacker News new | ask | show | jobs
by sgentle 2873 days ago
You might find relative entropy, aka information gain, aka Kullback-Liebler divergence interesting: https://en.m.wikipedia.org/wiki/Kullback–Leibler_divergence

Also this treatment of K-L divergence as a measure of "Bayesian surprise": http://ilab.usc.edu/surprise/

That's all based on Shannon entropy (probabilistic), not Kolmogorov complexity (algorithmic), but there are a lot of connections between them. This paper is a pretty thorough summary: https://homepages.cwi.nl/~paulv/papers/info.pdf

And here's a paper defining algorithmic relative complexity by analogy to relative entropy: http://www.mdpi.com/1099-4300/13/4/902/pdf-vor

"We define the cross-complexity of an object x with respect to another object y as the amount of computational resources needed to specify x in terms of y, and the complexity of x related to y as the compression power which is lost when adopting such a description for x, compared to the shortest representation of x"