No, it wouldn't bother me. I don't want to know, in order to not get my feelings hurt. People unfollow for a whole host of reasons, so I just let it go.
I know someone who Tweeted a controversial opinion touching on abortion and Sarah Palin and got flamed in public on Twitter and then tons of people stopped following her. It was a very unfortunate situation and a lot of feelings were hurt. I think that, if people decide they don't want to be in my Twitter circle (I don't like the term "followers" because it sounds cult-like to me), then it's not a personal thing. People are interested in different things at different points in their lives. Worrying that a few people didn't want to receive your Tweets anymore doesn't seem like a good investment of time. On the other hand, if you notice a mass "unfollowing," then it makes sense to find out why that might be. Other than that, I think that such ebbs and flows are just a natural part of life.
I would only worry about a large number of unfollowers at once. Then I'd want to know if it was something I said or something someone esle said about me. People do follow for different reasons an sometimes those reasons change so I don't get worked up over it. I pay more attention to people I'm following then who is following me.
I've been using Twitterless for a while, and it doesn't bother me when people quit following me. It's not personal, and even if it is, I'm just outspoken enough to not care about that either. ;-) I love it when people enjoy my tweets, but it doesn't bother me if my blurbs aren't their cup of tea.