I think the whole "screwing up the potentially awesome Phoenix storyline" is what did it for me with X3. Even though it's been years since X3 came out, it still irks me to this day about what could have been with that film. (If only Bryan Singer had passed on Superman Returns and worked on X3 instead. )
With Wolverine, I already came in with lowered expectations and didn't expect too much. I understand why folks are annoyed with the film (the Deadpool crap, mutant cameos in almost every scene, etc.).
But for me, my expectations going into both of these films had a major impact on my final impression of the two: X3 may have met my crappy expectations. But it only added to my separate disappointment about what could have been.
I was expecting Wolverine to be even crappier than X3. And since I really wasn't all that invested in this movie, I could look past the crap I already expected and enjoy those minor parts of the film that were surprisingly good.