Gunshots could still be heard in the morning as troops searched areas near the home of sect leader Mohammed Yusuf in the capital of Borno state.
At least 150 people in northern Nigeria have been killed in the last four days.
The sect, known as Boko Haram, is blamed for launching attacks on police stations and government buildings.
On Tuesday the army shelled Mohammed Yusuf’s compound, and exchanged heavy gunfire with militants through the night.
President Umaru Yar’Adua said the army had acted to nip a potentially dangerous problem in the bud.
Nigeria’s security services have been flooding into Maiduguri, the city worst affected by the violence, the BBC’s Caroline Duffield reports.
Maj Gen Saleh Maina told the Associated Press news agency that troops were hunting for sect members in homes and a mosque and near the railway station. He said the operation was being carried out “to prevent further loss of lives and property”.
Civilians were warned to vacate the district but on Wednesday morning women and children were still leaving the area.
Four states in northern Nigeria have been affected by the violence involving Boko Haram – Borno, Bauchi, Kano and Yobe.
The group is alleged to have shot dead or stabbed civilians at random but many of those killed are young men said to belong to the sect.
Boko Haram is against Western education. It believes Nigeria’s government is being corrupted by Western ideas and wants to see Islamic law imposed across Nigeria.
State of alert
Throughout the night, gunshots could be heard from the Doidamgari area of Maiduguri, where Mohammed Yusuf’s home is situated and the Boko Haram has its spiritual headquarters.
President Umaru Yar’Adua ordered Nigeria’s national security agencies to take all necessary action to contain and repel attacks by the extremists.
“These people have been organising, penetrating our societies, procuring arms, learning how to make explosives and bombs to disturb the peace and force abuse on the rest of Nigerians,” he said before departing on a trip to Brazil.
Maiduguri police said 103 had died in the violence in the city, including 90 members of Boko Haram, eight police officers, three prison officials and two soldiers.
Outside Maiduguri, there is a heightened state of alert across the northern states.
In Bauchi, scene of the first bloodshed on Sunday, 176 people remain under arrest.
At least 39 people were killed in the violence in Bauchi.
Sharia law is in place across northern Nigeria, but there is no history of al-Qaeda-linked violence in the country.
The country’s 150 million people are split almost equally between Muslims in the north and Christians in the south.