For portfolio optimization under downside risk measures, such as conditional value-at-risk or lower partial moments, we often invoke a scenario approach to approximate the high-dimensional integral involved when calculating risk. Consequently, two types of modeling risk may arise from this procedure: uncertainty in determining the distribution of asset returns and the error caused by approximating a given distribution with scenarios. To handle these two types of modeling risk within a unified framework, we propose a mathematically tractable set-valued scenario approach. More specifically, when short selling is not permitted, the robust portfolio selection problems modeled within a minimum-maximum decision framework using several types of set-valued scenarios can be translated into linear programs or second-order cone programs. These can be efficiently solved by the interior point method. The proposed set-valued scenario approach can be used not only as a methodology to alleviate the modeling risk but also as a useful tool for evaluating the impact of modeling risk. Our simulation analysis and empirical study show that robustness does not necessarily imply conservativeness, portfolio performance is affected by the investment style characterized by the return-risk tradeoff to a large degree and modeling risk only becomes significant when an aggressive strategy is adopted.