The actor, who played Jesus in the film, explains he hasn't received a lot of offers to play different roles in other projects because many Hollywood executives are a little wary to work with him.
"I made a choice," he tells People magazine. "But what you find out is that some choices are viewed as evil and I don't see it that way."
"If there was a comedy out there, I would do it, but after playing Jesus you don't really get offers like that anymore," he adds. "As soon as I did Passion, other (film offers) stopped coming in. But I'm a big boy and I'm not going to play a victim."
Caviezel returns to faith-based film fare with his hit new movie Paul, Apostle of Christ, in which he plays the disciple Luke, and he explains he wants his projects to "speak the truth" and feels it is his responsibility to "spread the word of God through his work".
"I wasn't out to do a faith-based movie," he says. "I wanted to do something that everybody could go and see and feel something when they walk out. Hopefully, that's the message of love."
And his faith gives him a real purpose in his personal life, which has led him to adopt three orphans from China.
"I've been to places around the world where orphans are and we have three of them," he says. "Two of them had brain tumours and one had cancer sarcoma. Their chance of survival wasn't great, but I wanted to help."
Meanwhile, Jim will reprise his role in Gibson's The Passion of the Christ sequel, The Resurrection.
"My faith makes me strong," he adds. "We all have choices to make and I made mine. When The Resurrection comes, who knows what will happen."