People love to talk.

There may well be people who will give you suggestions about whether to divorce or not. Sometimes that's the way people are but that does not mean that you have to disrupt or even to destroy your life, your future, your family status, the welfare of your extended family and more, just because people tongues wag too much.

It's your life. You can decide what to do. You can decide whether to remain married or not.

And if you don't think too much then you are likely to want to stay together.

The other people cannot do the proper anaysis for you. They cannot make decisions for you. Only you can decide these issues.

And only you should be the one to decide these issues.

Let their tongues wag. That's their problem. It's a violation of לא תלך רחיל בעמך - rechilut (רחילות) but because they violate a basic principle in the Jewish law does not mean that you have to change your own family status.