Footage broadcast by Al Hadath and Al Jazeera television showed chaotic scenes of people fleeing through streets as sustained bursts of gunfire crackled in the air. People rushed to carry away those who had been hit, the footage showed.
The main protest group accused the ruling military council of trying to break up the camp, calling the action "a massacre". The council said the security forces had targeted "unruly" groups in an adjacent area.
Medics said nine people were killed in Monday's violence and that the number of casualties were still rising.
The military's move comes after a lengthy stand-off with protesters seeking a speedy transition to civilian rule following the overthrow in April of long-time President Omar al-Bashir.
Dura Gambo, an activist, said large numbers of troops besieged the sit-in area outside the military's headquarters and arrested protesters trying to leave in the early morning.
An Associated Press journalist saw buses and soldiers on foot blocking roads leading to the protest site. Civilians were not allowed to walk in the streets, including women and children.
Videos circulating online appeared to show protesters standing at low brick barricades in the street, then being driven back by walls of blue-clad security forces carrying sticks.
Tens of thousands of protesters have been camped for weeks outside the military's headquarters, the epicentre of Sudan's uprising that led to the military overthrow of Mr al-Bashir.
Protesters had vowed to remain in the streets after Mr al-Bashir's ouster, saying an end to his 30-year rule did not go far enough.
Protest leaders and military officials have been negotiating over the make-up of a transitional government, as protesters call for "limited military representation" in a sovereign council that would lead the country as it transitions to civilian rule over three years.
Additional reporting by Press Association.
