Rice farming is increasingly being encouraged in Uganda because of its potential to double as an income as well as food security crop. However, existing data shows that current farmer’ yields are less than half of the expected average, partly due to inadequate soil fertility management interventions. Using a survey of rice farmers in the Lake Victoria Crescent Agro-ecological zone (LVC-AEZ) of Uganda, we assessed socio-economic drivers of soil fertility management interventions and how these relate to soil productivity and rice yield, with particular emphasis on mineral fertilizers. Results showed a wide range of soil fertility management practices being used and these include mineral fertilizers, manure application, crop rotation, soil erosion control and intercropping. Experience with rice farming, participation in farmers groups and land size positively and significantly affect the choice to use mineral fertilizers as a soil fertility management. The amount of mineral fertilizer is likely to increase if farmers use it for top dressing, and if they also practice other soil conservation practices. Similarly, farmers that use organic materials are likely to use less mineral fertilizer. Data also showed that under the archetypal farmers’ conditions, increasing amount of mineral fertilizer explains only 3% of rice yield increase, suggesting that Integrated Soil Fertility Management (ISFM) is inevitable to narrow the yield gap of rice in this region. This study lends support to recommendations that policy actions focus on improving access to quality extension advisory services to farmers in addition to increasing saving and credit schemes and communications infrastructure in order to stimulate fertilizer access. Research and extension on site-specific nutrient constraints is necessary to improve soil fertility management interventions.