Lolol I got 95 in SPH4U, had around a 10 going into the 1B03 exam. I must've really tanked it because I ended up with a 6, my worst grade by far (2nd lowest was a 10).
Overall, a fairly difficult course that will kill you if you cram. The material wasn't that hard and a lot of it was review but trying to learn it all right before the exam ruined me.
1) Yes, it did bring down my average but a lot of my friends got a grade similar to the grades they got in other subjects. Basically, it varies from person to person.
2) If it wasn't required, I wouldn't have taken it just because I don't exactly love physics. However, I can see the appeal if you are a fan of the subject.
3) Above.
4) Taking a course in the summer is like taking 2 courses during the year (based on what I've heard from numerous people) so taking both Orgos and both Physics would be like taking 4 courses each term. If you think that you can handle a job during the normal school year (5 courses), then you should be able to handle 4 courses and a job. If you think a job would not be manageable during the fall/winter terms, then it likely will be just as hard in the summer.
Hope this helps