It is generally expected that aggregates of metal nanoparticles are more efficient surface-enhanced Raman scattering (SERS) probes than individual nanoparticles, due to the enhancement of the electric field in the interparticle gaps. We show that, for asymmetric nanoparticles, such as gold nanorods (NRs), this is not always the case: the plasmonic behavior of NRs depends on the mutual orientation of the NRs in the ensemble. We report the results of experimental studies and theoretical analysis of the optical properties of clusters of side-by-side assembled gold NRs. Ensemble-averaged SERS spectroscopy showed a reduction in SERS intensity. Comprehensive finite-difference time-domain simulations showed a reduction of electric field intensity as the number of NRs per cluster increased. This is due to destructive interference as the radial component of the surface plasmon modes of the NRs in the cluster interact with each other. The present work expands our understanding of the configuration-specific optical behavior of asymmetric gold nanoparticles. Furthermore, it offers guidance toward the "design rules" for the development of colloidal NR systems for sensing applications.