The first country to issue Official stamps was Great Britain. It did so at the same time that it issued the first stamp, the Penny Black. But Great Britain was scooped in one major aspect of Official stamps by the United States Post Office, and that was in the issuance of Departmental Officials— Officials for use by individual government departments. The United States issued its first Departmental Official in 1870 and Great Britain followed some ten years later.
Official stamps are issued by postal agencies more as a form of accounting than because of any postal need. Most countries (including France, Belgium, Japan, indeed the vast majority of postal entities) have never issued Official Stamps. It's not that their government departments don't use the mails or require postage. It is just that they either buy their postage or are allotted postal use through an accounting exchange for the amount of postage or postal service that they need. Since all Postal Agencies until a few years ago were government subsidized services, it was felt that carrying Official Mail for free or at a reduced price was an acceptable intra-governmental transfer.
Great Britain shares with the United States and South Australia (and a very few others) the fact that they issued Official stamps for each department of government that was authorized to use stamps. Most countries just provided free stamps or post paid envelopes to various government departments. Other countries issued generic Official stamps for use by all agencies of government. But the United States began the concept of Departmental Officials, that is Officials for use only in specific departments. The purpose for this was accounting and keeping track of which agency used what postage. Accounting seemed to be a major preoccupation with the early British Post Office, maybe because concern over misappropriated postage stamps was one of the major objections to Rowland Hill's stamp issuing plan in the first place. Departmental Officials, however, proved to be more of an accounting solution in search of a problem than a real issue, and by the beginning of the 20th century were only being issued in any serious way by Argentina.