Defects in track geometry have a notable impact on the safety of rail transportation. In order to make the optimal maintenance decisions to ensure the safety and efficiency of railroads, it is necessary to analyze the track geometry defects and develop reliable defect deterioration models. In general, standard deterioration models are typically developed for a segment of track. As a result, these coarse-scale deterioration models may fail to predict whether the isolated defects in a segment will exceed the safety limits after a given time period or not. In this paper, survival analysis is used to model the probability of exceeding the safety limits of the isolated defects. These fine-scale models are then used to calculate the probability of whether each segment of the track will require maintenance after a given time period. The model validation results show that the prediction quality of the coarse-scale segment-based models can be improved by exploiting information from the fine-scale defect-based deterioration models.