I think there are two parts to this; first you have to start somewhere so If you take New Zealand for example, you can see startups like Xero, VendHQ, etc who from a small country dominate their country before moving on to the next stop - Australia. Before the world. They test their hypothesis, refine their product, etc before looking elsewhere. That's a reasonable approach that shows Nigerian focused startups makes sense as well.
The point is if you have a reasonable population, educated 18 - 25 yrs old, decent internet, etc, it makes sense to actually start from here before you go upwards.
Where it all breaks down is the second part. Where startups that are 'matured' can't go beyond that initial market. I think this might be for various reasons but in my mind a lot of it will revolve around operations (do you need partnerships, physical presence, regulatory constraints, etc) & technical (internalisation, designs gap, etc). Which of course might make it too much of a burden to do anything about it.
Generally, I would like to think there are some startups that will struggle to go beyond Africa due to the nature of their business. E.g location services like food startups, logistics & delivery, etc might not be able to translate well. On the other hand, pure software e.g. SaaS might stand a better chance.