The first to note a connection seems to have been the Principal of King's College, Dr. Wace, who is quoted by Huxley (1889) as follows:—
"But if this be so, for a man to urge, as an escape from this article of belief, that he has no means of a scientific knowledge of the unseen world, or of the future, is irrelevant. His difference from Christians lies not in the fact that he has no knowledge of these things, but that he does not believe the authority on which they are stated. He may prefer to call himself an Agnostic; but his real name is an older one—he is an infidel; that is to say, an unbeliever. The word infidel, perhaps, carries an unpleasant significance. Perhaps it is right that it should. It is, and it ought to be, an unpleasant thing for a man to have to say plainly that he does not believe in Jesus Christ." (quoted in Huxley 1889)Huxley then goes to great lengths, in my limited understanding of these issues, to clarify the difference between not believing what Scripture or the Church claims and not believing in a historical Jesus or a personal God. Before that, however, he even has to distinguish the negative connotation (unpleasant significance) of an infidel/agnostic being a heretic who claims offensive things about Jesus or God.
"If I venture to doubt that the Duke of Wellington gave the command "Up, Guards, and at 'em!" at Waterloo, I do not think that even Dr. Wace would accuse me of disbelieving the Duke."Likewise, agnostics cannot be accused of thinking Jesus was a liar just because they doubt the testimony given in the bible.
Robert Blatchford (1903) picks up discussing the invective, by quoting Huxley quoting Wace. He's been called an infidel, too, but has instead come to regard it as a batch of honour as he makes clear in the preface. He also defines the term infidel quite clearly.
"Be it pleasant or unpleasant to be an unbeliever, one thing is quite clear: religious people intend the word Infidel to carry "an unpleasant significance" when they apply to it one. It is in their minds a term of reproach. Because they think it is wicked to deny what they believe.
To call a man an Infidel, then, is tacitly to accuse him of a kind of moral turpitude. [...]
The root of the idea that it is wicked to reject the popular religion—a wickedness of which Christ and Socrates and Buddha are all represented to have been guilty—thrives in the belief that the Scriptures are the actual words of God, and that to deny the truth of the Scriptures is to deny and to affront God.In my opinion infidels come in various stripes, one being believers in a personal god (but not in scripture or churches), one being atheists, and a third one being agnostics. Maybe agnostics want to reject scripture as evidence and churches as authorities, but still reserve the freedom of belief as a freedom of choice in the absence of evidence. If so, then agnostics and atheists will be different subsets of infidels.
But the difficulty of the unbeliever lies in the fact that he cannot believe the Scriptures to be the actual words of God.
The Infidel, therefore, is not denying God's words, nor disobeying God's commands: he is denying the words and disobeying the commands of men.
No man who knew that there was a good and wise God would be so foolish as to deny that God. No man would reject the words of God if he knew that God spoke those words.
But the doctrine of the divine origin of the Scriptures rests upon the authority of the Church; and the difference between the Infidel and the Christian is that the Infidel rejects and the Christian accepts the authority of the Church.
Belief and unbelief are not matters of moral excellence or depravity: they are questions of evidence.
The Christian believes the Scriptures because they are the words of God. But he believes they are the words of God because some other man has told him so.
Let him probe the matter to the bottom, and he will inevitably find that his authority is human, and not, as he supposes, divine."
Robert Blatchford (1903)