Revert D71141352 (#2864)

Summary:
Pull Request resolved: #2864

CI tests are failing because the ops are not part of nightly build. We need to move them to FBGEMM instead

Reviewed By: PaulZhang12

Differential Revision: D72244609

fbshipit-source-id: cf90e45aab512dcb861fbdcda76374d01ee21b1a

Author: kausv