Neural Manifolds
Andy and I were talking about this.
What if instead of having a contrastive loss that operates between points, what if they operate between a point and an approximation of a manifold?
Define a function $f(x): \mathbb{R}^n \rightarrow \mathbb{R}$
This is a function that approximates a metric from a point to a surface. The surface is implicitly defined by the function so that sampling from this function gives the distance from any point in the original dimension of the data to the closest point on the manifold. So it is basically a distance function.
Then we place constraints on the values of $f(x)$ like smoothness and sparsity and minimize $\frac{1}{N} \sum_{i=0}^N f(x_i)$ subject to those constraints.
Then you can view the full shape of the manifold by seeding a bunch of points randomly in the grid and using gradient descent to find the closest point to those that is on the manifold.
Update: Yea this is just an energy based model.