[numpad0]

> The main problem with mesh generation from stuff like this is that usually the topology is a mess and needs a lot of cleanup to be useuable. It's not quite so bad for static non deforming objects but anything that needs to be animated deforming or that is organic looking would likely need retopologizing by hand. > > That's one of the worst parts of 3D modeling so it's like you're getting the AI to do the fun part and leaving you to do all the boring cleanup process.

From [1]. Seems like there is a pattern of "AI asked to generate final results with only final results to learn from, immediately asked for the apple in the picture" in AI generators. I suppose lack of specialization in application domains of NNs is a deliberate design choice for these high-profile projects, in a vague hope of simulating emergent behaviors as seen in the nature and avoiding to be another expert system(while being one!), but that attitude seems limiting usefulness, here and again.

[nl]

This is a misdiagnosis of where the problem is.

People developing these models are very aware of what 3D workflow is like.

The issue is that image->point cloud training data is very easy to get, whereas image or point cloud -> clean 3d mesh training data is very hard to get in unconstrained domains.

Generating point clouds is where the state of the art is now. That doesn't mean that the whole field isn't entirely aware that text->3d mesh unlocks many more capabilities.

[elcritch]

Seems like video game engines and the like would be useful ways to get lots of 3d models to corresponding point cloud data. What's the blocker to doing that? The models shown on that page look like 3d graphics circa 2000's or earlier.

[JayStavis]

I agree that random sampling surfaces of 3D meshes seems like a reasonable way to generate synthetic data for mesh > point cloud.

Without knowing a dang thing about AI, it feels like the problem moreso lies in:

1. Math related to topology: vertices, faces, edges, tri vs quad etc

2. Different topologies for the same object are better for different use cases. Rendering, skinning, morphing, physics etc all have different optimal topologies, and the definition of optimal varies based on workflow and scene specifics or even the human who has skills based on certain topological preferences. In other words, I'm not sure how much of 3D workflows are standardized even -- getting the topological data for workflows is no easy task, and it's not super usable until the model output can plug right into a workflow and the existing DCC ecosystem.

text2img generates a static asset, text2mesh is far more interesting beyond just the static rendering part which is where mesh topology becomes a big sticking point.

[nl]

Yes, I agree this is a potential way forward.

I believe there are three problems:

* There isn't software that generates point clouds from video games. This should be solvable but AFAIK hasn't been done yet.

* The diversity of models in video games is much lower than the real world

* Games use a bunch of techniques to reduce the poly count while making assets look like they are high poly (eg texture mapping). It's unclear what should be generated here.

Take a look at the field of NeRFs (Neural Radiance Fields: https://datagen.tech/guides/synthetic-data/neural-radiance-f...) for more on this subject.

[numpad0]

Or ask CG designers, under consent and with credits, for data recordings of intermediate steps. Same for illustrations. It almost seems like circumventing experts is the point.

[nl]

This isn't what is needed. The intermediate steps aren't useful.

[numpad0]

Don't human designers do image or point cloud -> clean 3D mesh in an iterative manner? I see it will be significantly more computationally expensive to iteratively deform a cube to a tree by NN, but I don't see why it isn't a solution.

[make3]

the thing is that it's been shown times and times again (with chatGPT for example) that you can get really pretty good results by giving massive amounts of final results to the model. This approach is better by far than anything we've ever had in either text AI or image generation AI