Stéphane Bersot has developed a Situated Virtual Reality for Telerobotic Control system which allows an operator to control a piece of machinery in a remote location by viewing a 3D representation of the remote scene.  The cool thing about this is since you are looking at a representation of the remote location you are able to see everything at any angle you want including positions that the physical remote cameras are unable to see. There is a bit of lag between the remote condition and the 3D model but if you aren’t performing anything that needs very fast visual feedback I can see this working very well. If that wasn’t enough the system also has some stereo cameras which are watching the operator to judge where the operator is looking and manipulates the shown image accordingly. The only enhancement I can think of would be to add some haptic feedback to the manipulator that the operator has in his hands.

Be sure to have a look here for more information.

“The objective of this project is to experiment with a new way to teleoperate mining operations based on 3D models. The idea is to substitute completely the view of the physical model by the 3D model and to teleoperate machinery without direct video feedback.

Two methods are available to control the virtual camera which represents the view location in Unity3D: The Logitech DualAction gamepad and/or the head and eye tracking via Facelab software and stereoscopic infrared cameras. We can choose either to use our eye gaze to tilt and pan the camera or a tip tracking method which places the tip of the 3D robot in the centre of the screen with or without delay to smooth the motions. The camera position can be changed using the simple joystick on the gamepad or moving our head right and left.”