Three.js - billboard effect, maintain orientation after camera pans

I have a plane geometry that is always set to face the camera via: plane.lookAt(camera.position); in the update loop. I am using OrbitControls to control the camera. When rotating or zooming the scene, the plane continues to face the camera as expected. However, after panning the scene, while the plane continues to face the camera, rotating the camera appears to rotate the plane as well, so that f