[dpflan]

I am not well versed in topological deep learning; but can topological concepts be learned by a graph neural network -- like can there a be topological layer(s) that learns the topology of the graphs, like how it's easy to visualize how a CNN architecture can see different hierarchical patterns at deep depths, seemingly learning the greater complexities of a image representation and understanding?

[zwaps]

The proposed use cases (folding, social networks) correspond to what was previously termed "Geometric Deep Learning". Here, things like Graph Neural Networks were understood in terms of (pairwise?) relations to be learned under the assumption of symmetry/equivariance. It can be shown that not all relations that fit in a graph can be learned by such GNNs. Further, not all relations can be modeled with a (symmetric) graph.

Hence, these people moved on the using Category Theory, which may or may not lead to the use of GNNs.

Reading the first part of the present paper, the Topological DL would instead move beyond the idea of "pairwise relation".

[dpflan]

Thank you. "Reading the first part of the present paper, the Topological DL would instead move beyond the idea of "pairwise relation". -- interesting. It seems to make sense intuitively, given difficulties in GNNs, to look at a higher order representations -- which may lead to understanding the underlying graph properties too. But I had assumed that such higher orders of graph would be part of the GNN learning process, even representations that are not really "human mathematically understood yet".

That is also interesting me regarding Geometric Deep Learning, which got some hype and interest recently, and seemed like a good start for more formal representation of different deep learning models (finding connections and mathematical steps between the model zoo). Something more mathematically rigorous does seem needed to truly make informed engineering improvements and scientific understanding.

[zwaps]

I agree that it's exciting to move toward a mathematical approach to characterize what can, and what can not be learned by a DNN!

Maybe someone else has the knowledge to give you some more precise answers.

As far as I understand we need to differentiate between higher order relations that can be learned by message parsing and those that cannot. Some higher order relationships can be represented in a graph (of pairwise relations) and, furthermore, can be learned by iterating out from a focal node through the paths (think centrality). GNNs can learn those.

But GNNs can't learn all Graphs because not all such relations are amendable to message parsing. So, not all higher order information contained in a graph are learned by GNNs.

In the present case, the authors seem to make the point that there are higher-order relations (say, node-set) which are also not representable by a graph (or a collection of graphs).

I actually do not know whether that is obviously true. For instance, hypergraphs can be modeled as a collection of graphs. But if the claim stands, then we'd need topological deep learning to move further. Otherwise, like you say, it is maybe a more general framework to understand information processing in DL - or more efficient to learn node-node or node-set information.

Either way, I have not yet quite figured out how Category Theory and Topological DL relate to each other (each generalising Geometric Deep Learning).

[lmeyerov]

Agreed, the definitions here of what a GNN is (=> cannot do) seem pedantically narrow & extreme, and then throws the baby out w the bath water

See last ~2 years of work by Michael Bronstein's teams (who moonlights as Twitter's ex? GNN r&d lead) that address 2+ of the points here

It's true they are not the end-all, but our issues in practice aren't the above, but more like how to best combine temporal deep learning techniques w graph ones (it is not just another continuous dimension)

Worth noting: TDA methods & UMAP are largely the same under-the-hood & in practice, and we find UMAP (incl neural) highly effective, and typically reach for it before GNNs. UMAP has been an admirably rare mix of theory & practicality..

[dpflan]

Interesting. Could one graph be the topological representation, let's say like SCC? So there are heterogenous graphs with meta-graphs connected to them...?

[lmeyerov]

There are a lot of cool experiments happening like this, both principled and from intuition

Ex:Label prop following non-local jumps ("metapaths") based on community detection (network-of-network, ...) or other shapes. Same thing for what constitutes a feature.

Michael Bronstein's papers (and talks!) are often in this vein, in the mainstream, leading, and quite accessible, so would recommend starting there..

[chaxor]

Yeah this is being done by several groups across the field Check out the work by Gunnar Carlsson and Bastien Reick. If those are enjoyable also check out Micheal Bronstein and Petar Velolickovic, although not directly related to your question.

[mplemay]

Do you mind linking the homepage, google scholar, etc. of "Petar Velolickovic"? The only reference to the name on google search appears to be this post.

[dpflan]

- https://geometricdeeplearning.com/ -- group of researchers presenting the paradigm

- https://petar-v.com/ - Petar's homepage

- https://scholar.google.co.uk/citations?user=kcTK_FAAAAAJ&hl=... - Petar's Google Scholar

- https://www.linkedin.com/in/petarvelickovic/ - Petar's LI

[dpflan]

GAT is particularly interesting at the moment: https://scholar.google.co.uk/citations?view_op=view_citation...