•   almost 2 years ago

Linking Project Building Information Models and Specifications (expirement)

The idea is to create an infrastructure for a deep learning based program which will take a text document (let's say project specs)
and link it to components in the BIM model. This is kind of a thing I have heard people mention on occasion, and DL is probably the best way to go.
The way I thought about the thing was that handcoding is out of the question, since there are too many standards and that number squared of ways for people
to ignore the standards in their models and specifications. Between a model and a document, there are also too many ways to encode the same thing.
However, when humans look at the model and the document, we can more or less figure out what's what. Deep nets are able to do the same thing (see examples below).
The idea is there will be some common patterns of meaning between what is in a model, and in a natural language document, and there are plenty of pretrained nets out there,
so it should be possible to test on models and documents to see if they can detect the same structure.
And we will not have to handcode features or input stuff into models for sensical linking of project data?

Since there are practically no examples of spec to model links in the world (unless anyone has some), the training has to be unsupervised, so we can take advantage of pretrained algorithms.

The type of deep network probably best suited for the document side and ready-ish out of the box is a deep belief net (DBN), or autoencoder/autodecoder, or one of the proven open source models.

Some resources:

Exploration: The basic pipeline
1.1) Parse PDF specs for text.
1.2) Send to pretrained neural net for unsupervised representation training.
1.1) Do a dumb extract of revit model data, and dump it to text.
1.3) Send revit data to (second version?) of neural net for unsupervised representation training.
2) Run tests to find nearest topic pairs on model and document versions of neural net output(TSNE, PCA?)
User View
On a parallel track, we can think of the UI part of such a program.
Perhaps the revit items get a parameter to link to the section(s) of the PDF partaining to it?
Perhaps a modeless form in revit (or window in BIM360 online?) that pulls up snippets of text or links?

-sample specifications and models for the same project.
-any of python,numpy, revit api (python or c#), revit UI API, lua(?)
-gpu (I can bring one)
-If there is some interest, I can add links to pretrained models from github so we can hit the ground running

-Either a cool UI or mockup for a linked up model to document set
-Perhaps the neural net does horribly, but close enough at times - maybe a concept for visualizing the document and possible model elements together?
-Some benchmark on what can be done with a simple, stupid unsupervised feature embedding of specifications and what lives in a model.

-this would be the "simplest, fastest" thing to test how possible it would be to get an unsupervised topic based matcher of revit to text documents.
-expose team members to some deep learning ideas, and approaches to things which do not rely on handcoding.


Comments are closed.