We report on prism coupling of high-quality (high-Q) terahertz (THz) whispering-gallery modes (WGMs) in spherical high resistivity float zone grown silicon (HRFZ-Si) resonators over two octaves from 0.2 THz to 1.1 THz. The WGMs are excited using a HRFZ-Si prism and show unprecedented quality factors of up to 2.2 × 104. A detailed discussion of the phase-and mode-matching criteria of the prism coupling scheme implemented in the continuous wave THz spectroscopy system is presented. The results provide numerous opportunities for passive ultra-broadband high-Q devices operating in the THz frequency range.