*BSD News Article 61745

Path: euryale.cc.adfa.oz.au!newshost.anu.edu.au!harbinger.cc.monash.edu.au!news.mira.net.au!inquo!nntp.uio.no!nntp-oslo.UNINETT.no!nntp-trd.UNINETT.no!due.unit.no!sthaug
From: sthaug@nethelp.no (Steinar Haug)
Newsgroups: comp.unix.bsd.freebsd.misc
Subject: Glaring anomaly in lmbench results: socket bandwidth using localhost = 0.21 MB/s
Date: 11 Feb 1996 14:35:48 GMT
Organization: Nethelp Consulting, Trondheim, Norway
Lines: 1039
Message-ID: <4fkus7$kab@stork.runit.sintef.no>
NNTP-Posting-Host: trane.uninett.no

I have a P133 running FreeBSD-2.1.0 release version, and I've been running
some lmbench tests to see what kind of performance I have. Most results are
really good (which is what I would expect!), but there is one glaring anomaly
that I'd like to find out more about: Socket bandwidth using localhost (ie.
a TCP connection to localhost) only gives 0.21 MBytes/s. As a comparison,
pipe bandwidth is around 22 MBytes/s. The socket bandwidth result is quite
reproducible. I have included the results from a full lmbench run below.

Some further info: I've tried to vary the socket buffer size, and I've also
tried playing around with the rfc1323 and rfc1644 variables. The result is
still the same extremely low number.

Hardware is a P133, 32 MByte EDO RAM, 512 kB PLB cache. OS is 2.1.0 Release
version, no patches.

Any good ideas about what's happing here?

Steinar Haug, Nethelp consulting, sthaug@nethelp.no
----------------------------------------------------------------------
[lmbench1.0 results for FreeBSD verdi.nethelp.no 2.1.0-RELEASE FreeBSD 2.1.0-RELEASE #2: Mon Feb 5 22:26:22 MET 1996 root@:/usr/src/sys/compile/VERDI i386]
[DISK: /dev/rwd0a]
[FILE: /tmp/XXX]
[FSDIR: /tmp]
[REMOTE: ]
[NETWORKS: ]
[MB: 8]
[Sun Feb 11 12:44:08 MET 1996]
[12:44PM up 3 mins, 0 users, load averages: 0.01, 0.02, 0.00]
[net: Name  Mtu   Network     Address            Ipkts Ierrs    Opkts Oerrs  Coll]
[net: lp0*  1500  <Link>                             0     0        0     0     0]
[if: lp0*: flags=8810<POINTOPOINT,SIMPLEX,MULTICAST> mtu 1500]
[net: de0   1500  <Link>00.00.c0.0e.69.cd            0     0        3     0     0]
[if: de0: flags=8863<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST> mtu 1500]
[if: inet 192.168.1.2 netmask 0xffffff00 broadcast 192.168.1.255]
[if: ether 00:00:c0:0e:69:cd ]
[net: de0   1500  192.168.1   verdi                  0     0        3     0     0]
[if: de0: flags=8863<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST> mtu 1500]
[if: inet 192.168.1.2 netmask 0xffffff00 broadcast 192.168.1.255]
[if: ether 00:00:c0:0e:69:cd ]
[net: lo0   16384 <Link>                             2     0        2     0     0]
[if: lo0: flags=8009<UP,LOOPBACK,MULTICAST> mtu 16384]
[if: inet 127.0.0.1 netmask 0xff000000 ]
[net: lo0   16384 your-net    localhost              2     0        2     0     0]
[if: lo0: flags=8009<UP,LOOPBACK,MULTICAST> mtu 16384]
[if: inet 127.0.0.1 netmask 0xff000000 ]
[net: sl0*  552   <Link>                             0     0        0     0     0]
[if: sl0*: flags=c010<POINTOPOINT,LINK2,MULTICAST> mtu 552]
[net: tun0* 1500  <Link>                             0     0        0     0     0]
[if: tun0*: flags=8010<POINTOPOINT,MULTICAST> mtu 1500]
[mount: /dev/wd0a on / (local)]
[mount: /dev/wd0s2f on /usr (local)]
[mount: /dev/wd0s2e on /var (local)]
[mount: /dev/wd2s2c on /local (local)]
[mount: procfs on /proc (local)]
[df: Filesystem  1K-blocks     Used    Avail Capacity  Mounted on]
[df: /dev/wd0a       32255    12793    16881    43%    /]
[df: /dev/wd0s2f    887983   585973   230971    72%    /usr]
[df: /dev/wd0s2e     30207     1677    26113     6%    /var]
[df: /dev/wd2s2c   1313231   343560   864612    28%    /local]
[df: procfs              4        4        0   100%    /proc]
133.29 Mhz, 8 nanosec clock
Null syscall: 8 microseconds
Pipe latency: 107 microseconds
UDP latency using localhost: 214 microseconds
TCP latency using localhost: 259 microseconds
RPC/udp latency using localhost: 355 microseconds
RPC/tcp latency using localhost: 454 microseconds
TCP/IP connection cost to localhost: 346 microseconds
Socket bandwidth using localhost: 0.21 MB/sec
Disk /dev/rwd0a latency: 563 microseconds
Process fork+exit: 2446 microseconds
Process fork+execve: 11033 microseconds
Process fork+/bin/sh -c: 18157 microseconds
Pipe bandwidth: 22.15 MB/sec
Disk /dev/rwd0a bandwidth1711 KB/sec
File /tmp/XXX write bandwidth: 1657 KB/sec
Pagefaults on /tmp/XXX: 23 usecs
"mappings
0.01 51
0.25 109
0.50 148
1.00 249
2.00 507
3.00 96
4.00 112
5.00 109
6.00 108
7.00 107
8.00 108

"read bandwidth
0.02 5.91
0.03 8.63
0.06 23.67
0.12 20.17
0.19 28.89
0.25 36.40
0.38 24.48
0.50 25.19
0.75 25.58
1.00 28.67
1.50 30.02
2.00 32.95
2.50 34.73
3.00 33.47
3.50 38.84
4.00 38.90
5.00 31.02
6.00 31.02
7.00 31.01
8.00 31.05

"Mmap read bandwidth
0.02 85.85
0.03 100.16
0.06 95.42
0.12 95.27
0.19 94.32
0.25 90.19
0.38 85.79
0.50 83.49
0.75 79.17
1.00 78.31
1.50 76.17
2.00 76.12
2.50 55.39
3.00 55.99
3.50 56.16
4.00 56.83
5.00 56.83
6.00 56.42
7.00 56.74
8.00 55.42

"libc bcopy aligned
0.0625 55.16
0.1250 52.65
0.5000 47.71
1.0000 42.56
2.0000 41.07
4.0000 40.92
8.0000 40.92

"libc bcopy unaligned
0.0625 61.76
0.1250 60.07
0.5000 49.83
1.0000 45.36
2.0000 43.59
4.0000 43.42
8.0000 43.44

"unrolled bcopy aligned
0.0625 53.83
0.1250 53.74
0.5000 44.81
1.0000 41.89
2.0000 40.17
4.0000 40.14
8.0000 40.15

"unrolled bcopy unaligned
0.0625 54.02
0.1250 52.13
0.5000 45.93
1.0000 41.78
2.0000 40.36
4.0000 40.14
8.0000 40.15

Memory read bandwidth
1.0000 82.18
2.0000 78.97
4.0000 78.88
8.0000 78.87

Memory write bandwidth
1.0000 83.22
2.0000 83.16
4.0000 83.21
8.0000 83.20

"File system latency
0k	1000	26	59
1k	1000	29	28
4k	1000	23	25
10k	1000	21	25


"size=0 ovr=30
2 26
4 28
8 30
16 31
20 32

"size=4 ovr=64
2 31
4 36
8 36
16 37
20 38

"size=16 ovr=236
2 31
4 33
8 46
16 59
20 63

"size=32 ovr=485
2 2
4 3
8 48
16 91
20 101

"size=64 ovr=915
2 29
4 90
8 172
16 247
20 274
Memory load latency
"stride=8
0.00049 7
0.00098 7
0.00195 7
0.00293 7
0.00391 7
0.00586 7
0.00781 8
0.00977 23
0.01172 33
0.01367 33
0.01562 33
0.01758 33
0.01953 33
0.02148 33
0.02344 33
0.02539 33
0.02734 33
0.02930 33
0.03125 33
0.03516 33
0.03906 41
0.04297 40
0.04688 40
0.05078 39
0.05469 44
0.05859 44
0.06250 43
0.07031 42
0.07812 41
0.08594 40
0.09375 43
0.10156 42
0.10938 42
0.11719 41
0.12500 41
0.14062 40
0.15625 41
0.17188 40
0.18750 41
0.20312 43
0.21875 44
0.23438 44
0.25000 46
0.28125 49
0.31250 50
0.34375 53
0.37500 54
0.40625 56
0.43750 56
0.46875 57
0.50000 58
1.00000 66
1.50000 70
2.00000 71
2.50000 71
3.00000 72
3.50000 71
4.00000 72
4.50000 72
5.00000 72
5.50000 72
6.00000 72
6.50000 71
7.00000 71
7.50000 71
8.00000 72

"stride=16
0.00049 7
0.00098 7
0.00195 7
0.00293 7
0.00391 7
0.00586 8
0.00781 9
0.00977 40
0.01172 60
0.01367 60
0.01562 60
0.01758 60
0.01953 60
0.02148 60
0.02344 60
0.02539 60
0.02734 60
0.02930 60
0.03125 60
0.03516 60
0.03906 75
0.04297 74
0.04688 72
0.05078 71
0.05469 82
0.05859 80
0.06250 79
0.07031 77
0.07812 75
0.08594 74
0.09375 79
0.10156 77
0.10938 76
0.11719 75
0.12500 74
0.14062 73
0.15625 75
0.17188 74
0.18750 76
0.20312 79
0.21875 80
0.23438 82
0.25000 85
0.28125 90
0.31250 94
0.34375 99
0.37500 101
0.40625 105
0.43750 105
0.46875 107
0.50000 109
1.00000 125
1.50000 133
2.00000 136
2.50000 136
3.00000 136
3.50000 136
4.00000 137
4.50000 137
5.00000 136
5.50000 137
6.00000 137
6.50000 137
7.00000 136
7.50000 136
8.00000 137

"stride=32
0.00049 7
0.00098 7
0.00195 7
0.00293 7
0.00391 7
0.00586 8
0.00781 9
0.00977 44
0.01172 67
0.01367 67
0.01562 67
0.01758 67
0.01953 67
0.02148 67
0.02344 67
0.02539 67
0.02734 67
0.02930 67
0.03125 67
0.03516 67
0.03906 80
0.04297 78
0.04688 78
0.05078 77
0.05469 85
0.05859 84
0.06250 83
0.07031 81
0.07812 80
0.08594 78
0.09375 83
0.10156 81
0.10938 80
0.11719 80
0.12500 79
0.14062 78
0.15625 80
0.17188 79
0.18750 80
0.20312 83
0.21875 84
0.23438 85
0.25000 88
0.28125 92
0.31250 96
0.34375 99
0.37500 101
0.40625 105
0.43750 104
0.46875 107
0.50000 108
1.00000 121
1.50000 127
2.00000 129
2.50000 130
3.00000 130
3.50000 130
4.00000 131
4.50000 131
5.00000 130
5.50000 130
6.00000 130
6.50000 131
7.00000 130
7.50000 130
8.00000 130

"stride=64
0.00049 7
0.00098 7
0.00195 7
0.00293 7
0.00391 7
0.00586 8
0.00781 10
0.00977 44
0.01172 67
0.01367 67
0.01562 67
0.01758 67
0.01953 67
0.02148 67
0.02344 67
0.02539 67
0.02734 67
0.02930 67
0.03125 67
0.03516 67
0.03906 80
0.04297 78
0.04688 78
0.05078 77
0.05469 85
0.05859 84
0.06250 83
0.07031 81
0.07812 80
0.08594 79
0.09375 83
0.10156 82
0.10938 81
0.11719 80
0.12500 79
0.14062 78
0.15625 80
0.17188 79
0.18750 80
0.20312 83
0.21875 84
0.23438 85
0.25000 88
0.28125 93
0.31250 97
0.34375 101
0.37500 103
0.40625 106
0.43750 106
0.46875 108
0.50000 109
1.00000 123
1.50000 129
2.00000 131
2.50000 132
3.00000 132
3.50000 132
4.00000 132
4.50000 132
5.00000 132
5.50000 132
6.00000 132
6.50000 132
7.00000 132
7.50000 132
8.00000 132

"stride=128
0.00049 7
0.00098 7
0.00195 7
0.00293 7
0.00391 7
0.00586 10
0.00781 11
0.00977 45
0.01172 67
0.01367 67
0.01562 67
0.01758 67
0.01953 67
0.02148 67
0.02344 67
0.02539 67
0.02734 67
0.02930 67
0.03125 67
0.03516 67
0.03906 80
0.04297 79
0.04688 78
0.05078 77
0.05469 85
0.05859 84
0.06250 83
0.07031 81
0.07812 80
0.08594 79
0.09375 83
0.10156 82
0.10938 81
0.11719 80
0.12500 79
0.14062 78
0.15625 80
0.17188 79
0.18750 81
0.20312 83
0.21875 84
0.23438 85
0.25000 88
0.28125 95
0.31250 101
0.34375 105
0.37500 107
0.40625 110
0.43750 110
0.46875 112
0.50000 113
1.00000 127
1.50000 133
2.00000 136
2.50000 136
3.00000 136
3.50000 136
4.00000 136
4.50000 136
5.00000 136
5.50000 136
6.00000 136
6.50000 136
7.00000 136
7.50000 136
8.00000 136

"stride=512
0.00049 7
0.00098 7
0.00195 7
0.00293 7
0.00391 7
0.00586 17
0.00781 15
0.00977 43
0.01172 67
0.01367 67
0.01562 67
0.01758 67
0.01953 67
0.02148 67
0.02344 67
0.02539 67
0.02734 67
0.02930 67
0.03125 67
0.03516 67
0.03906 82
0.04297 81
0.04688 79
0.05078 78
0.05469 88
0.05859 87
0.06250 86
0.07031 84
0.07812 82
0.08594 81
0.09375 85
0.10156 84
0.10938 83
0.11719 82
0.12500 81
0.14062 79
0.15625 82
0.17188 80
0.18750 82
0.20312 86
0.21875 87
0.23438 88
0.25000 91
0.28125 109
0.31250 122
0.34375 126
0.37500 128
0.40625 132
0.43750 132
0.46875 134
0.50000 135
1.00000 151
1.50000 159
2.00000 161
2.50000 162
3.00000 162
3.50000 162
4.00000 162
4.50000 162
5.00000 162
5.50000 162
6.00000 162
6.50000 162
7.00000 162
7.50000 162
8.00000 162

"stride=1024
0.00049 7
0.00098 7
0.00195 7
0.00293 7
0.00391 7
0.00586 32
0.00781 22
0.00977 43
0.01172 67
0.01367 67
0.01562 67
0.01758 67
0.01953 67
0.02148 67
0.02344 67
0.02539 67
0.02734 67
0.02930 67
0.03125 67
0.03516 67
0.03906 84
0.04297 83
0.04688 82
0.05078 81
0.05469 92
0.05859 90
0.06250 89
0.07031 86
0.07812 85
0.08594 83
0.09375 89
0.10156 87
0.10938 86
0.11719 84
0.12500 83
0.14062 82
0.15625 84
0.17188 83
0.18750 85
0.20312 89
0.21875 90
0.23438 92
0.25000 96
0.28125 126
0.31250 150
0.34375 155
0.37500 157
0.40625 162
0.43750 161
0.46875 164
0.50000 165
1.00000 183
1.50000 192
2.00000 195
2.50000 196
3.00000 196
3.50000 196
4.00000 196
4.50000 196
5.00000 196
5.50000 196
6.00000 196
6.50000 196
7.00000 196
7.50000 196
8.00000 196

"stride=2048
0.00049 7
0.00098 7
0.00195 7
0.00293 7
0.00391 7
0.00586 72
0.00781 52
0.00977 64
0.01172 67
0.01367 74
0.01562 67
0.01758 72
0.01953 67
0.02148 71
0.02344 67
0.02539 71
0.02734 67
0.02930 70
0.03125 67
0.03516 67
0.03906 90
0.04297 88
0.04688 86
0.05078 85
0.05469 99
0.05859 97
0.06250 96
0.07031 93
0.07812 90
0.08594 88
0.09375 95
0.10156 93
0.10938 91
0.11719 90
0.12500 88
0.14062 86
0.15625 89
0.17188 87
0.18750 90
0.20312 95
0.21875 97
0.23438 99
0.25000 104
0.28125 162
0.31250 206
0.34375 212
0.37500 216
0.40625 221
0.43750 220
0.46875 224
0.50000 226
1.00000 248
1.50000 259
2.00000 263
2.50000 264
3.00000 264
3.50000 264
4.00000 264
4.50000 264
5.00000 264
5.50000 264
6.00000 264
6.50000 264
7.00000 264
7.50000 264
8.00000 264

"stride=4096
0.00049 7
0.00098 7
0.00195 7
0.00293 7
0.00391 7
0.00586 112
0.00781 112
0.00977 112
0.01172 112
0.01367 112
0.01562 112
0.01758 112
0.01953 112
0.02148 112
0.02344 112
0.02539 112
0.02734 112
0.02930 112
0.03125 112
0.03516 112
0.03906 142
0.04297 140
0.04688 137
0.05078 136
0.05469 155
0.05859 153
0.06250 150
0.07031 146
0.07812 142
0.08594 140
0.09375 150
0.10156 147
0.10938 145
0.11719 143
0.12500 141
0.14062 138
0.15625 143
0.17188 140
0.18750 144
0.20312 150
0.21875 153
0.23438 156
0.25000 162
0.28125 251
0.31250 318
0.34375 327
0.37500 331
0.40625 340
0.43750 338
0.46875 344
0.50000 346
1.00000 378
1.50000 394
2.00000 399
2.50000 400
3.00000 400
3.50000 400
4.00000 400
4.50000 400
5.00000 400
5.50000 399
6.00000 399
6.50000 399
7.00000 400
7.50000 400
8.00000 400

"stride=8192
0.00049 7
0.00098 7
0.00195 7
0.00293 7
0.00391 7
0.00586 7
0.00781 7
0.00977 112
0.01172 112
0.01367 112
0.01562 112
0.01758 112
0.01953 112
0.02148 112
0.02344 112
0.02539 112
0.02734 112
0.02930 112
0.03125 112
0.03516 112
0.03906 112
0.04297 112
0.04688 112
0.05078 112
0.05469 112
0.05859 112
0.06250 112
0.07031 112
0.07812 112
0.08594 112
0.09375 137
0.10156 136
0.10938 134
0.11719 132
0.12500 131
0.14062 129
0.15625 127
0.17188 126
0.18750 125
0.20312 124
0.21875 134
0.23438 143
0.25000 141
0.28125 238
0.31250 294
0.34375 303
0.37500 313
0.40625 306
0.43750 308
0.46875 313
0.50000 319
1.00000 353
1.50000 372
2.00000 388
2.50000 394
3.00000 396
3.50000 397
4.00000 398
4.50000 399
5.00000 400
5.50000 400
6.00000 400
6.50000 400
7.00000 400
7.50000 400
8.00000 400

"stride=16384
0.00049 7
0.00098 7
0.00195 7
0.00293 7
0.00391 7
0.00586 7
0.00781 7
0.00977 7
0.01172 7
0.01367 7
0.01562 7
0.01758 112
0.01953 112
0.02148 112
0.02344 112
0.02539 112
0.02734 112
0.02930 112
0.03125 112
0.03516 112
0.03906 112
0.04297 112
0.04688 112
0.05078 112
0.05469 112
0.05859 112
0.06250 112
0.07031 112
0.07812 112
0.08594 112
0.09375 112
0.10156 112
0.10938 112
0.11719 112
0.12500 112
0.14062 112
0.15625 112
0.17188 112
0.18750 112
0.20312 112
0.21875 112
0.23438 112
0.25000 112
0.28125 222
0.31250 279
0.34375 276
0.37500 274
0.40625 272
0.43750 281
0.46875 278
0.50000 277
1.00000 314
1.50000 321
2.00000 341
2.50000 353
3.00000 360
3.50000 371
4.00000 381
4.50000 387
5.00000 393
5.50000 395
6.00000 397
6.50000 396
7.00000 398
7.50000 398
8.00000 400

"stride=32768
0.00049 7
0.00098 7
0.00195 7
0.00293 7
0.00391 7
0.00586 7
0.00781 7
0.00977 7
0.01172 7
0.01367 7
0.01562 7
0.01758 7
0.01953 7
0.02148 7
0.02344 7
0.02539 7
0.02734 7
0.02930 7
0.03125 7
0.03516 112
0.03906 112
0.04297 112
0.04688 112
0.05078 112
0.05469 112
0.05859 112
0.06250 112
0.07031 112
0.07812 112
0.08594 112
0.09375 112
0.10156 112
0.10938 112
0.11719 112
0.12500 112
0.14062 112
0.15625 112
0.17188 112
0.18750 112
0.20312 112
0.21875 112
0.23438 112
0.25000 112
0.28125 221
0.31250 278
0.34375 275
0.37500 273
0.40625 271
0.43750 269
0.46875 268
0.50000 267
1.00000 300
1.50000 299
2.00000 300
2.50000 309
3.00000 313
3.50000 328
4.00000 338
4.50000 350
5.00000 357
5.50000 363
6.00000 368
6.50000 371
7.00000 373
7.50000 377
8.00000 381