Should I not go to my brother's surprise birthday party because of this?

and by doing so play right into her hands? Is that what you really want? You say that his wife is isolating him from all of his friends. By not going YOU are making a choice to isolate yourself from him; not the other way around, kiddo.