bitcoin-cli -rpcport=yourrpcport -rpcuser=youruser-rpcpassword=yourpassword getpeerinfo
TBH on my local node I see no difference to your output (perhaps I am looking at the wrong thing):
root@MuffinStation:~# docker exec -it bitcoin-node bitcoin-cli -conf=/bitcoin/bitcoin.conf -datadir=/bitcoin getpeerinfo | grep -i pingtime
"pingtime": 0.190681,
"pingtime": 0.128512,
"pingtime": 0.19279,
"pingtime": 0.312987,
"pingtime": 0.188246,
"pingtime": 0.031604,
"pingtime": 4.321106,
"pingtime": 0.027684,
"pingtime": 0.138654,
"pingtime": 0.144322,