psavine42 -- • about 2 years ago
Deep Neural Net for semi-supervised revit model prediction (experiment)
"If you can dodge a wrench you can dodge a ball" -Patches O'Houlihan (dodgeball)
Since Deep learning (DL) has been taking over almost every field of machine learning - from computer vision (segmentation, detection, scene description), language translation, scene understanding, etc, etc., why not put put a deep net in revit?
One possible application of DL in BIM-land is to predict elements in a revit model given other elements from the model.
I think the problem can be implemented as a deep recurent neural network (RNN) which takes in data from revit (for example some walls, with their parameters and geometry) and predicts missing data, which in this case, would be missing because we randomly removed it.
Why would something like this be useful?
If you can predict stuff in a general fashion, you can tune what you are predicting latter on. In other words, if we can do something simple like predict locations of walls from other walls,
by expirience from deep learning, one can use that to fill in missing detail by training on a model with the detail and without the detail latter on.
By detail I mean something like given a model with a partially completed floorplan, complete the plan, or take a plumbing model and fill in fittings, couplings, etc without handcoding all the rules.
The big idea that should be focused on is the generalization capabilites of deep nets proven in other fields, and their ability to surpass handcoded system accuracy when dealing with high dimensional data.
The task is to think about the problem in this way, and try hacking an approach.
Why think about it this way as opposed to some other bigger and badder deep architecture?
1) This does not require prepared training data, and can be trained online if an infrastructure to revit can be built. This is the biggest problem I have encountered with deep learning in aec - the absence of datatets (it is also acknowledged by the field), so we avoid it!
2) It is a generic enough idea that if someone wanted to reuse it latter, they would just need to modify the deep net, or transfer to another model, so we avoid all the hairy theory.
3) Since most architectures do not support big 3d models, and it may not even be necessary, we can avoid this by using the parameter and coordinate level data directly from BIM.
Will it work?
Being an expirement, This may totally fail - perhaps voxel data is the way to go, but one has to start somewhere with BIM and DL.
The network architecture I am thinking about is something like this: https://github.com/jcjohnson/torch-rnn, which is generic to lanugage learning (and therefore its inputs/outputs are english tokens)
but maybe the answer is to represent the revit data on a graph and use one of these: http://cvgl.stanford.edu/papers/jain_cvpr16.pdf .
Here's the most basic version of a character level rnn trained on some tv script data it did for practice awhile back: https://github.com/psavine42/parctice-rnn.
If there is interest, I can better spec things out on the neural net side beforehand.
Heres some resources on deep learning.
To implement this, we would need:
1: DL Agent containing the neural network This has to be a program running outside of revit, because most of the DL libraries are in python and are probably too finecky to import into revit python shell, and it should be able to use its own gpu.
2: A tokenizer which takes data from revit queries and converts it into a vector to be fed into the RNN.
3: A trainer (class?) which generates training data by masking some tokenized data, and passinf it to the neural net input, and the full data
4: A little server to pass messages between revit and the Agent. (my thought is the DL agent and revit agent should be on seperate machines, since revit
does not work on linux, and I think tensorflow is the only DL framework that can sanely run on windows. It is probably most flexible for them to just message pass via REST - maybe this
can take data from several models queued up at once for faster training).
5: A revit addin which will wait for commands, and execute whatever query is sent to it.
-tracking the agent's position in the model.
-Since the agent will be taking in a chunk of data from its surroundings, my first thought is to have a function that gets element in a radius,
but maybe raycasting.
-some processing to flatten the element data to be sent back to the agent.
-A mechanism to detokenize the deep net's predictions, and visualize them in back in revit. They will be garbage at start of training, so a way to deal with this may be necessary.
The details of the above would need to be determined.
-revit api in c# or python
-deep learning (concepts and/or practice).
-Some revit models.
-GPU w/ cuda (I can bring spare box).
[DELIVERABLES / PROOF OF CONCEPT]
-Infrastructure such that an software agent can query the revit model from outside.
-A deep net that has some reasonable success or promise to predice model elements.
-Create an infrastructure such that an software agent can query the revit model from outside.
-Think about automation in BIM not as handcoding rules, but as presenting data to an algorithm can be trained on the task.
-Get a baseline on which to do further expirements - run a few training epochs at least.
-Expose team members to deep learning
Comments are closed.