We present the strongest constraints to date on anisotropies of cosmic microwave background (CMB) polarization rotation derived from 150 GHz data taken by the BICEP2 & Keck Array CMB experiments up to and including the 2014 observing season (BK14). The definition of the polarization angle in BK14 maps has gone through self-calibration in which the overall angle is adjusted to minimize the observed TB and EB power spectra. After this procedure, the QU maps lose sensitivity to a uniform polarization rotation but are still sensitive to anisotropies of polarization rotation. This analysis places constraints on the anisotropies of polarization rotation, which could be generated by CMB photons interacting with axionlike pseudoscalar fields or Faraday rotation induced by primordial magnetic fields. The sensitivity of BK14 maps (∼3 μK-arc min) makes it possible to reconstruct anisotropies of the polarization rotation angle and measure their angular power spectrum much more precisely than previous attempts. Our data are found to be consistent with no polarization rotation anisotropies, improving the upper bound on the amplitude of the rotation angle spectrum by roughly an order of magnitude compared to the previous best constraints. Our results lead to an order of magnitude better constraint on the coupling constant of the Chern-Simons electromagnetic term gaγ≤7.2×10-2/HI (95% confidence) than the constraint derived from the B-mode spectrum, where HI is the inflationary Hubble scale. This constraint leads to a limit on the decay constant of 10-6fa/Mpl at mass range of 10-33≤ma≤10-28 eV for r=0.01, assuming gaγ∼α/(2πfa) with α denoting the fine structure constant. The upper bound on the amplitude of the primordial magnetic fields is 30 nG (95% confidence) from the polarization rotation anisotropies.