Hi RandomGuy95 and welcome,
I know part of what you're asking is whether your penis is too small, so we will start there. The short answer is no. Want a longer answer? Keep reading.
The most common penis length flaccid is about is about 3-4 inches. While erect, it’s 5-6 inches. The average diameter (girth) for an erect penis is about 1.5 inches. But as Heather and Hanne explain in
Innies and Outies: The Penis, The Testes, and More, "Five to six inches is the most common penis length, but that there are penises that are longer and penises that are shorter that are all perfectly normal, healthy, desirable penises."
So, your penis is a common length. But that's not really what is at the heart of what you're asking, is it? Researchers have found men tend to wish their penis was bigger regardless of how big it already is. It's understandable since we are often taught that a bigger penis makes you more of a man or that sex with you is going to automatically be more enjoyable. Neither are true. But that doesn't mean there aren't people who still believe those assumptions, either consciously or unconsciously. It sounds like maybe you've got those beliefs floating around in your head, and you're wondering whether they're floating around in your partners' too?
The only way to answer those questions is to talk about it. You say that you've noticed your penis doesn't seem to "do it for them," but what does that mean? If the sex wasn't pleasurable for one or both of you, I can promise it's not because of your penis.
What Makes Someone Good in Bed has nothing to do with the size or appearance of their genitals. Similarly, if you want an honest opinion about whether your penis size was/is a "dealbreaker" for someone, that's something you have to ask them. But I will tell you that anyone who sees penis size as a dealbreaker is someone who evaluates a person and/or a sexual experience based on one body part. And that's not someone who is going to be a fulfilling sexual or romantic partner.