Ballast fouling in the railroad substructure is detrimental to the effectiveness of the railroad track and its structural capacity. The early detection of ballast fouling is of utmost importance to the safety of the rail system and its life-cycle cost-effectiveness. Ground-penetrating radar (GPR), a nondestructive evaluation tool, has shown its potential as a means of assessing the condition of the railroad substructure rapidly, effectively, and continuously. However, an unknown ballast dielectric constant and an unclear interface between clean and fouled ballast limit the accuracy of GPR assessment. In the present study, controlled laboratory testing was conducted to measure accurately the dielectric constants of two common ballast types, granite and limestone, under various fouling and moisture conditions. In addition, a time–frequency method, short-time Fourier transform (STFT), was used to demonstrate graphically the frequency energy variation with the depth of the ballast under various conditions. This method indirectly reflects the ballast fouling condition. To validate the effectiveness of STFT for ballast fouling assessment, field GPR data from the Orin Subdivision in Wyoming were analyzed with the STFT technique and laboratory-measured dielectric constants. Comparison of the ground truth data and GPR measurements proved that the STFT method is effective and that its results are reasonably accurate when it is used to locate fouling and trapped water within ballast, especially when the laboratory-predicted ballast dielectric constants are used in the analysis.