Should I worry about losing my job?

Your supervisor sounds very unreasonable. I've worked with s like that before. You shouldn't worry about losing your job, you should only worry about how soon you can find another job and leave this company.