I suggest XCALL GETUSN,usrnam$,console$
For telnet connections, console$ (aka machine$ per the doc) will be something like:
pts/10:18234
while serial connections will be, well, different. (I don't have a handy serial connection to test.)
You'll need to be flexible on your syntax analysis of the two types, because both types will start with the name of the device, which can vary depending on several factors. In particular, I think "pts" may switch one of its letters when you hit some kind of maximum number of connections. (You might want to log on to your largest system and do a finger to see what the range is.)
Note: the AGF_TNT flag in AUI_ENVIRONMENT, which sounds like it might tell this information, is only useful under Windows for distinguishing between local/file-sharing and telnet(ATSD). (I think I ran into the same problem just described - not knowing a simple conclusive way to distinguish between TCP/serial.)