In a recent post, I described the first pass at hand pose detection in VR using tensorflow. I recently used this work as the starting point for this year’s TMCS hackathon. Each year we spend 2 days with students from the Theory and Modelling in Chemical Sciences (TMCS) course and give them a programming challenge. Last year I had students predict VR avatar positions, and this year we spent the two day’s improving the neural network and making a VR experience that used the gesture detection in real time.

To improve the classifier, we collected more data, implemented cross-validation and performed hyper-parameter optimizations on the number of nodes in the hidden layers, the learning rate and the optimizer.  With these optimizations, we reached 99.5% accuracy on the labelled test set.

My main focus was to figure out how to make the DNNClassifier provide predictions responsively. If we were…