It appears you have opened a second thread on basically the same subject. That is generally not a good idea.
To directy answer the title of this thread, the panel switch should always be "Off" unless you detect water in the bilge and the pump is not removing it automatically. And if the pump sensor/float has failed and the pump will not stop running moving the toggle to "Off" should make absolutely no difference.
As I said in reply to the other, more recent, post if you have a conventionally wired bilge pump the toggle switch on the panel should never be in the "On" position unless you are certain the boat is filling and the sensor/float has failed so the pump does not start on its own. Otherwise the pump will run to destruction once the water is gone.
I questioned how you turned it off, and apparently that is because the switch was in the wrong position and you "fixed" it by putting it in the correct position. Again, you should not be able to turn off the pump manually if it started due to the sensor/float detecting water and water is still present. If the water is gone, the pump should shut itself off. If it does not, and the only way to stop it is by using the toggle switch you have a serious installation problem.
I would recommend that you find other service than the ones you have already used. It appears they did not properly diagnose your original problem and did not explain why the pump failed in the first place. I even believe your pump is most likely still wired incorrectly if the toggle must be in the "On" position for the sensor/float to operate the pump.
(Just as a guess, I would say your pump started to pump "automatically", as described in your post, because you turned on the main battery switch and you had the manual bilge pump switch set to "On". Is that a possibility?)