Woody Harrelson and Matthew McConaughey.Photo: Frank Micelotta/Invision for the Television Academy/AP

Actors Woody Harrelson, left, and Matthew McConaughey in the audience at the 66th Primetime Emmy Awards at the Nokia Theatre L.A. Live on Monday, Aug. 25, 2014, in Los Angeles.

Woody Harrelsonwants to know ifMatthew McConaugheyreally is his brother.

While appearing onKelly Ripa’s SiriusXM podcastLet’s Talk off Camera with Kelly Ripalast week, McConaughey, 53, revealed that he and Harrelson, 61,might be half-siblingsas his mother had told them both that she “knew” Harrelson’s father during a time when she was separated from her husband.

The Oscar winner, who was teasing the pair’s new seriesBrother from Another Mother,also noted how their families often confuse them in photos.

Further explaining McConaughey’s revelation, Harrelson discussed their potential family relation on Tuesday’s episode ofThe Late Show with Stephen Colbertand said he wants them to take a DNA test.

“Well, I’ll just say there is some veracity to that thought because we talked to Ma' Mac, Matthew’s legitimate mother, and she let us know one time … I mean, this is crazy. We were in Greece, we were watching the US team win the World Cup and I don’t know, I mentioned something about regrets,” Harrelson toldColbert.

Harrelson went on to describe the pause as “filled with innuendo,” adding, “The year of his birth, nine months before, she was on a sabbatical from her relationship with his supposed father, Jim.”

ButThe Messengerstar said that McConaughey isn’t keen on doing a DNA test as he could be “losing a father.”

“The thing is, we want to go for a test, but for him, it’s a much more big deal,” Harrelson said. “I mean he feels like he is losing a father. But I’m like, no, you’re gaining a different father and a brother.”

McConaughey expressed his doubts about doing a DNA test during his appearance on Ripa’s podcast. After Ripa asked whether the pair have taken any DNA tests to determine whether they may be biologically related, McConaughey said, “This is what we’re on the precipice of now."'

“It’s a little easier for Wood to say ‘come on, let’s do it,’ because what’s the skin in it for him?” he added. “It’s a little harder for me because he’s asking me to take the chance to go ‘wait a minute, you’re telling me my dad may not be my dad of 53 years’ and believe in it? I got a little more skin in the game.”

RELATED GALLERY:Celebs You Didn’t Know Were Friends

Woody Harrelson and Matthew McConaughey.Charles Sykes/Invision/AP

Woody Harrelson, left, and Matthew McConaughey arrive at the Oscars on Sunday, March 4, 2018, at the Dolby Theatre in Los Angeles.

Never miss a story — sign up forPEOPLE’s free daily newsletterto stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories.

McConaughey, who often refers to Harrelson as a “brother from another mother” in hissocial media posts, said of his mom’s revelation, “In Greece a few years ago, we’re sitting around talking about how close we are and our families and et cetera and my mom is there. And she says, ‘Woody, I knew your dad.’ "

“Then, there’s possible receipts placing them out in West Texas where there might have been a gathering or a meeting or a ‘knew’ moment,” he added.

McConaughey and Harrelson first worked together on the 1999 satire filmEDtvand have since reunited for the 2008 comedy filmSurfer, Dudeand in season one of HBO’sTrue Detective.

source: people.com