This paper discusses the use of ground penetrating radar (GPR) to rapidly, effectively, and continuously assess railroad track substructure conditions, especially ballast. To overcome the limited electromagnetic waves penetration for high-frequency antennae and the low resolution of low-frequency antennae, this study uses a multiple-frequency GPR system to assess railroad substructure conditions. High-frequency antennae were used to detect the scattering pattern, which is related to air void volume in railroad ballast, and low-frequency antennae are used to assess deeper substructure conditions. Considering the scattering energy attenuation is highly frequency and material dependent, a time– frequency method based on tracking the frequency spectrum and energy change over depth can be used to extract ballast fouling conditions. From GPR field collected data, ground-truth observation, and ballast gradation analysis, the multiple-frequency GPR system demonstrates a promising capability to assess railroad track substructure condition.