Performing indentation tests on rough surfaces at the microscale creates non-negligible scatter of the load–indentation depth curves and leads to an inaccurate computation of mechanical properties. In previous work, the minimization of the error between the shapes of the experimental loading curve and the shapes predicted by Bernhardt’s law enabled the macrohardness of the material to be determined with accuracy and to identify a relationship between the standard deviation of the errors of shapes and the root-mean-square roughness (S q) computed at the scale of the indentation imprint. In this paper, a semi-analytical model applied to roughness measurements is used to understand this relationship. Good agreement is found between the model results and the experimental results. Analysis of the semi-analytical model confirmed that the identified relationship is caused by the topography and not by some experimental bias and that the relevant scale for the computation of S q is the scale of the indentation imprints (15 µm). However, the relationships found with the model and the experimental results show different slopes and y-intercepts. The y-intercept found with the numerical curves is negligible compared with the y-intercept identified with the experimental curve, which is equal to 30 nm. This indicates that even if S q is equal to zero, the zero-point of the curve is not accurately determined by the instrumented indentation device. As for the differences in slope, these may be partly due to experimental noise and the differences of methodology of detections of first-contact for the recording of the load–displacement curves. Overestimation of the resistance of abrasion debris by the semi-analytical model may also explain the differences of slopes. However, further testing is required to confirm this hypothesis.