The 41-year-old actor, best known for playing Kayce Dutton across five seasons of Taylor Sheridan’s modern Western epic, says he was fully prepared to hang up his cowboy hat after the series ended last fall.

But as he tells People, saying goodbye to Yellowstone wasn’t as easy as he expected.

“It was seven years of playing a person that I’ll never see again, except for having his hat and jacket in my closet,” Grimes said of the Season 5 finale. “To try to put that show away, it was hard. It was really hard, and I know this is going to sound weird, but it was sort of emotional. It was like losing a family in a way.”

That emotion ran deeper than he anticipated. Grimes recalls the last day of filming as the definitive end of his time as Kayce: “On the last day of shooting, I thought it was my last day as Kayce. It was over to me.”

What Is ‘Y: Marshals’ About?