The organization of plasmonic nanoparticles (NPs) determines the strength and polarization dependence of coupling of their surface plasmons. In this study, plasmon coupling of spherical Au NPs with an average diameter of 15 nm was investigated in shape-memory polymer films before and after mechanical stretching and then after thermally driving shape recovery. Clusters of Au NPs form when preparing the films that exhibit strong plasmon coupling. During stretching, a significant polarization-dependent response develops, where the optical extinction maximum corresponding to the surface plasmon resonance is redshifted by 19 nm and blueshifted by 7 nm for polarization parallel and perpendicular to the stretching direction, respectively. This result can be explained by non-uniform stretching on the nanoscale, where plasmon coupling increases parallel to the shear direction as Au NPs are pulled into each other during stretching. The polarization dependence vanishes after shape recovery, and structural characterization confirms the return of isotropy consistent with complete nanoscale recovery of the initial arrangement of Au NPs. Simulations of the polarized optical responses of Au NP dimers at different interparticle spacings establish a plasmon ruler for estimating the average interparticle spacings within the experimental samples. An investigation of the temperature-dependent recovery behavior demonstrates an application of these materials as optical thermal history sensors.