Well yeah, welcome to public school. It's actually quite interesting, the social norms of different cliques and race/gender groups. Most are subtle, however the common Black teenager, based off of overwhelming personal and external observation, tend to have the most pretentious showing of habits for group behavior.
On average, there are two kinds of blacks, the black who is least influenced by the American black culture (the ones that are hard to tell from other races because they don't retain a very significant cultural signature) and those that are heavily influenced by the common motifs of American Black culture such as being "gangsta" or the "ghetto" kind of blacks. There are several kinds of ghetto sub cultures but the black ghetto subculture is the most relevant in this case.
Hanging in large numbers, talking and never being one of the first in the class is one tell tale sign of this particular group. This can be deduced due to the psychological effects of the group mentality, if your friends are in other classes than you'd be considered a social pariah for not wanting to stay in the hall and chatting with them, it would be you putting education over socializing and that just doesn't fucking happen in High school, especially amongst black teenagers who have a sense of "education don't mean shit" kind of attitude, even if they know better, the mentality of the group is that it is just so, therefore one must conform.
You maybe thinking "well that doesn't happen with JUST black kids" and you're right, it doesn't, however hanging in large numbers is a characteristic of them and because of that they are noticed more. It is also because of they're large group that moving from one spot would be troublesome, therefore, they select a spot to "meet up" and thus you have your clogged halls.
This can all be deduced from logic and prior knowledge of behavioral patterns which we ALL possess. So no, you aren't being "racist" by observing something so common, in fact, calling it "racist" is A. A blatant misuse of the word "racist" and B. Down right ignorant of something that has been witnessed by hundreds of others, and I'm sure if they did a social data collection of "High School Student behavioral patterns amongst races", it would be consistent with what I have described above.