Before contestant Dan Spilo was removed from Survivor: Island of the Idols, producers debated what to do about the 48-year-old talent agent, multiple sources tell PEOPLE.

As early as day 2, contestant Kellee Kim told Spilo that she would prefer that he not touch her. But as the season went on, Spilo continued to touch the other contestants, including Kim. While some contestants were unbothered by the physical contact, Kim brought it up again and was voted out of the game on day 22.

No action was taken while Kim was in the game. During the subsequent tribal council, Spilo apologized, saying, “I work in an industry in which the #MeToo movement was formed and allowed — thank God — to blossom and become powerful and strong. My personal feeling is if anyone ever felt for a second uncomfortable about anything I’ve ever done, I’m horrified about that and I’m terribly sorry.”

Dan Spilo

PEOPLE has spoken to multiple people involved with the show’s production, all of whom say that producers were concerned about how to handle the situation. “This went way beyond the game aspect of the show,” one production source says. “No one has ever been kicked off the show, ever, since 2000. It was unprecedented to even consider removing him.”

The debate involved producers of the show as well as executives at CBS, a second source tells PEOPLE. “When a situation arises, everyone wants to do the right thing,” the second insider explains. “But there was a difference of opinion of what should be done.”

Producers pulled each contestant aside to discuss the issue — but some players now say that the warnings were so unspecific that no one understood the severity of the issue.

“It was just a very vague blanket statement telling me if I ever felt unsafe, I should let production know,” contestant Aaron Meredith told Parade. “There was never any indication that something was going on or that someone had issues.”

Ultimately, Spilo was permitted to stay — on a very short leash. He continued to play the game and was never seriously in danger of being voted out. He seemed destined to make the final three until his surprise ejection on day 36.

At the end of Wednesday’s episode, the screen faded to black with the following text: “Dan was removed from the game after a report of another incident, which happened off-camera and did not involve a player.”

Sources tell PEOPLE that the incident in question — which involved a member of the show’s production team — happened after an immunity challenge as Spilo and other contestants were getting into a boat to transport them back to the camp.

At one point, Spilo allegedly touched the female crew member’s leg. According to multiple sources, he insisted that the contact was inadvertent and accidental as he lost his balance while trying to get into the boat.

But the show’s production team wasn’t convinced that the contact was merely incidental. After consulting with the show’s legal team, producers removed him from the show.

“If it had been his first strike, I don’t think he would have been removed from the game,” the first production source tells PEOPLE. “But the feeling was, ‘it’s time to pull the plug.’ In many ways, this was just the final straw.”

Sources tell PEOPLE that Spilo vehemently disagreed with the decision to boot him from the game.

“When we first told Dan we were pulling him from the game, he was not happy,” host Jeff Probst told Entertainment Weekly. “We talked through everything for quite a while, and by the time he got on the boat to leave he had calmed down and was actually very respectful as he departed.”

After the episode aired, Spilo was told by the CBS legal department that he would not be permitted to appear at the finale.

“It’s ugly and it’s controversial,” a third production source tells PEOPLE. “There was this feeling as the season started that this was a great cast, that they were all playing the game, and it would be really fun. And by the time it got to this point, everyone was just saying this season had become a sh– show. Everyone was really sad that it had gone off the rails like this one, because Survivor is supposed to be fun. The end of this season wasn’t fun at all, and it was time for Dan to go.”