Individual hobbies attract relatively small percentages of the population and always have. Take chess—a cerebral game that attracts many of the same type of people as philately but with significantly more market penetration. The writers at the chess federation website bemoan the fact that though supposedly 600 million people worldwide know how to play chess, only 83,000 belong to their federation and are considered avid chess players. By that standard, we do pretty well in stamps. The Australia post office estimates that there are 22 million collectors worldwide. Linn's stamp magazine draws an interesting picture, dividing collectors into three concentric circles. The most outer group are people who throw a stamp in a drawer if they like it, never responding to anything more than the visual appeal of a design or subject that they find interesting. In the  United States, Linn's estimates that there are four million of this type of collector. In addition, Linn's surveys and market research estimate that there are about 1.5 million of what they call "casual collectors." Casual collectors have albums and are people who go in and out of interest with the hobby, picking it up for a few months, maybe buying new issues from the Post Office, but never seriously going back and trying to obtain completeness, studying their stamps and postal history, or attending to the concerns of quality.
If you are reading this blog, you are probably in Linn's innermost circle of philatelic interest. Most of us never feel like we are in the innermost circle of anything, so belonging to this group is pretty special. Linn's estimates that there about a quarter of a million of us in this country (out of a population of 300 million). Members of this circle may belong to the American Philatelic Society (though only about 10% of us do), get stamp magazines, and read stamp blogs. Most of all, we are the people who maintain the hobby from generation to generation. We keep up the standards, teach newcomers, write the history, and we pass on philatelic traditions. Philately has been around for over  150 years and has developed protocols and ways of doing things. Just as when we came into the hobby, there were people to show us the way, we do that for the ones who collect after us.
Knowing the estimate of a quarter of a million stamp collectors in the United States, the question that arises is whether there are more or less serious collectors than in past eras. This is hard to tell for sure, but my own sense is that the the percentage of serious collectors is somewhat higher. The number of casual collectors is perhaps less. In the 1930-60 period, there was an enormous philatelic entry complex that encouraged kids to collect stamps. Captain Tim and Boys Life magazine had great market penetration to American boys, and there were millions of collections started. Most of these were of the most elementary kind, usually smaller packets poorly mounted into shabby albums. We see them all the time. The current generation doesn't have this level of casual collector (no H.E. Harris and no school promotions), and so we bemoan the popularity of the hobby. But the inner core, people for who philately is part of their life, has remained a constant, maybe an even increasing number.
Whether this number has increased or decreased over the last twenty years is hard to say. Certainly the number of members of the American Philatelic Society and the number of subscribers to Linn's has declined. But it is so easy to be a collector today without being a joiner or subscriber. If we look at market strength, it is hard to imagine that the internet auction sites of Delcampe, Ebay, and Bid Start could between them support twenty million listings at all times if there was only 250,000 serious collectors. One Ebay based company that I know amassed nearly 200,000 different buyers in a five year period. That would be 80% market penetration if there were only 250,000 serious collectors. As a totally unscientific statistical sample, I spent a month asking every customer of ours that I talked to if they bought from this Ebay company and less than 20% had heard of them. This produces a (totally unscientific) estimate of at least a million serious collectors in this country.
The real answer is, of course, that we don't know how many serious collectors there are, even if we could all agree how to define seriousness. Most collector's interest in the hobby, though it often lasts decades, ebbs and flows over time. It is fair to say, though, that the internet age has made it harder to judge just how many collectors exist and just how serious they are. Again, the market based estimate (about a million) is probably the best. Sellers soon disappear when they have no buyers. So tens of millions of monthly stamp offerings probably means our hobby is in pretty good shape.