Small perturbations in the dielectric environment around resonant dielectric structures usually lead to a frequency shift of the resonator modes directly proportional to the polarizability of the perturbation. Here, we report experimental observations of strong frequency shifts that can oppose and even exceed the contribution of the perturbations’ polarizability. We show in particular how the mode frequencies of a lithium niobate whispering-gallery-mode resonator are shifted by planar substrates—of refractive indices ranging from 1.50 to 4.22—contacting the resonator rim. Both blue- and redshifts are observed, as well as an increase in mode linewidth, when substrates are moved into the evanescent field of the whispering gallery mode. We compare the experimental results to a theoretical model by Foreman et al. [J. Opt. Soc. Am. B33, 2177 (2016)JOBPDE0740-322410.1364/JOSAB.33.002177] and provide an additional intuitive explanation based on the Goos–Hänchen shift for the optical domain, with applications to dielectric structures ranging from meta-surfaces to photonic crystal cavities.