What is the context of this hadith? when did the prophet (saw) say it and why? Also, what is the root of the word kafir? All of this matters. I'm not a fan of translation for many reasons, one being twisted understanding of things, which is why one should strive to comprehend the arabic language well(not saying I'm an expert by any means, still learning here). I'm also not a fan of using hadith or Quran without its context because it can easily be misunderstood and misused.
I do Agree with some of your post, no layman, in fact no man period can decide who is a kafir or who isn't. This doesn't take away the fact that this word is used very freely among muslims and it is a very bad thing for the reason just stated. I only spoke out because I want to see it STOP. It is disturbing.
- cOsMiC
Sis it's not just one hadeeth but multiple ayaat and ahadeeth have explicitly mentioned salaah as being a fundamental pillar of belief in Allah and an absence of it as disbelief.
Allah says about those who neglected the salaah:
فَخَلَفَ مِنۢ بَعۡدِهِمۡ خَلۡفٌ أَضَاعُواْ ٱلصَّلَوٰةَ وَٱتَّبَعُواْ ٱلشَّہَوَٲتِ*ۖ فَسَوۡفَ يَلۡقَوۡنَ غَيًّا (٥٩) إِلَّا مَن تَابَ وَءَامَنَ وَعَمِلَ صَـٰلِحً۬ا فَأُوْلَـٰٓٮِٕكَ يَدۡخُلُونَ ٱلۡجَنَّةَ وَلَا يُظۡلَمُونَ شَيۡـًٔ۬ا
Then there has succeeded them a posterity who have given up As-Salaat (the prayers) and have followed lusts. So they will be thrown in Hell. Except those who repent and believe (in Islamic monotheism), and work righteousness. Such will enter Paradise and they will not be wronged in aught. (Qur'an 19:59-60)
The Messenger :saws1: said:
Between a man and shirk (polytheism) and kufr (disbelief) there stands his neglect of the prayer. (Narrated in Muslim).
He :saws1: also said:
Buraydah ibn al-Husayb (may Allah be pleased with him) said: "I heard the Messenger of Allah (pbuh) say: 'The covenant that distinguishes between us and them is the prayer and whoever neglects it has disbelieved (become a kaafir).'" (Narrated in Ahmad, Abu Dawood, al-Tirmidhi, al-Nisaa’i and Ibn Maajah).
Ibn Taymiyyah said about missing the prayer:
As for the one who does not pray sometimes and does not make them up and does not intend to make them up, or he misses some of the obligatory parts of the prayer and does not make them up or intend to make them up, many of our companions ruled that he becomes a kaafir thereby. Then when he prays again he becomes a believer, as is indicated by the words of the Prophet

saws1

:
"Whoever misses ‘Asr prayer deliberately, his good deeds will be lost."
And he (peace and blessings of Allaah be upon him) said:
"Whoever misses a prayer deliberately is no longer under the protection of Allah."
But again I want to emphasise that a person may have committed kufr but the layman cannot decide who is and isn't a kafir. This is up to the scholars, and if one notices one hardly ever hears the scholars making takfeer.