In this article, we consider gridless source localization based on the spatial covariance matrix of acoustical data collected by an array of microphones. Covariance matrix fitting problems are formulated in infinite-dimensional settings, and solved by the Sliding Frank-Wolfe algorithm. The proposed method does not impose any constraint on the geometry of the array, the propagation model or the domain of interest, and does not necessitate a training phase. It is tested on simulated and experimental measurements for the localization of sources in a three-dimensional domain. Performances are compared to the state of the art, showing in particular a better resolution than MUSIC (MUltiple SIgnal Classification) at low SNR.