We propose a functional gradient descent algorithm (FGD) for estimating volatility and conditional covariances (given the past) for very high-dimensional financial time series of asset price returns. FGD is a kind of hybrid of nonparametric statistical function estimation and numerical optimization. Our FGD algorithm is computationally feasible in multivariate problems with dozens up to thousands of individual return series. Moreover, we demonstrate on some synthetic and real data-sets with dimensions up to 100 that it yields significantly much better predictions than more classical approaches, such as a constant conditional correlation Garch-type model. Since our FGD algorithm is constructed from a generic algorithm, the technique can be adapted to other problems of learning in very high dimensions.