Growing an olive tree is a bit of a commitment if you’re looking to grow fruit – which will happen at about the four or five year mark. For green olives, pick your fruit when it turns from dark green to light green, or wait for them to turn black for black olives.