It is not clear why Gen Yusuf Ahmed Sarinle was targeted, but correspondents suggest it may be because he backed the deployment of foreign peacekeepers.
Somalia has not had a functioning national government since 1991 with militias dividing it into fiefdoms.
A new Somali government has been named and is due to start relocating to Mogadishu from Kenya on 1 February.
Motive unclear
Gen Sarinle's relatives said four men armed with pistols and AK-47 guns forced entry into the police chief's flat in his home village of Hamar-bile south of Mogadishu at 0800 (0500 GMT) on Sunday.
The general was reportedly shot seven times in the chest and head..
The gunmen then fled in a waiting Toyota pick-up.
The BBC's Mohammed Olad Hassan in the city says Sarinle's had supported the new government's efforts to deploy African peacekeepers and disarm militias.
Insecurity, organised crime and lawlessness have reached alarming proportions in Mogadishu since the formation of the government, our correspondent says.
Islamic courts, which have sought to extend their influence in the absence of a government, have vowed to fight the peacekeepers.
But another reason for his killing has been advanced: Gen Sarinle had also been in charge of investigating the recent destruction of an Italian colonial cemetery by militiamen in Mogadishu.
Some 700 graves in the cemetery were destroyed and their human remains dumped elsewhere.
Mogadishu was under Italian colonial control until World War II and many of the graves belong to Italian soldiers and expatriates.