NGINX vs nghttpx
Recently in version 1.9.5, NGINX introduced their experimental HTTP/2 support, purposefully dropping SPDY protocol support in favour of HTTP/2. Having no SPDY support in NGINX would leave about 60% of users capable of SPDY/3.1 having to fall back to HTTP/1.1 connections, thus lowering performance.
So being interested in webserver performance, I decided to do some small benchmarks, as a rough guide to how NGINX and nghttp2 may perform in a deployed environment. Of course, my tests are an inaccurate representation of real life performance: I'm only serving a single 100kb text file through NGINX.
Setup
The tests are done on my Ubuntu machine, the server this very site runs on. It's an i7 4790 desktop with 20GB of RAM, running Ubuntu 15.10 Desktop.
I'm running three tests:
- NGINX 1.9.7 with HTTP/2
- nghttpx using the above as a backend over HTTP
- NGINX 1.9.7 with SPDY/3.1
And here are my compile flags:
nginx version: nginx/1.9.7
built by gcc 5.2.1 20151010 (Ubuntu 5.2.1-22ubuntu2)
built with OpenSSL 1.0.2d 9 Jul 2015
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-http_ssl_module --with-threads --with-file-aio --with-cc-opt='-g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,--as-needed' --with-<strong>http_v2_module</strong>
nginx version: nginx/1.9.4
built by gcc 5.2.1 20151010 (Ubuntu 5.2.1-22ubuntu2)
built with OpenSSL 1.0.2d 9 Jul 2015
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-http_ssl_module --with-threads --with-file-aio --with-cc-opt='-g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,--as-needed' --with-<strong>http_spdy_module</strong>
nghttp2 was compiled with the default flags.
The config files:
Keep in mind that the NGINX configuration I used is not optimum for high performance. I'll probably follow up with a blog post on better comparison across more configurations and tests.
Results
h2load -n 100000 -c 160 -t 8
NGINX 1.9.7 with HTTP/2
./nghttp2/src/h2load -n 100000 -c 160 -t 8 https://localhost:8443
...
finished in 6.42s, 15571.91 req/s, 1.45GB/s
requests: 100000 total, 100000 started, 100000 done, 100000 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 100000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 10026507840 bytes total, 13900000 bytes headers (space savings 23.20%), 10000000000 bytes data
min max mean sd +/- sd
time for request: 245us 98.86ms 7.93ms 6.33ms 84.34%
time for connect: 24.08ms 1.32s 374.68ms 429.36ms 83.13%
time to 1st byte: 46.91ms 1.33s 392.37ms 424.10ms 83.13%
nghttpx
./nghttp2/src/h2load -n 100000 -c 160 -t 8 https://localhost:8444
...
finished in 6.18s, 16172.23 req/s, 1.51GB/s
requests: 100000 total, 100000 started, 100000 done, 100000 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 100000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 10010249408 bytes total, 3035428 bytes headers (space savings 85.19%), 10000000000 bytes data
min max mean sd +/- sd
time for request: 180us 229.37ms 9.17ms 6.34ms 82.40%
time for connect: 130.16ms 328.52ms 250.65ms 44.00ms 70.63%
time to 1st byte: 221.39ms 407.43ms 305.03ms 52.53ms 46.25%
NGINX 1.9.4 with SPDY/3.1
./nghttp2/src/h2load -n 100000 -c 160 -t 8 https://localhost:8445
...
finished in 7.24s, 13821.13 req/s, 1.29GB/s
requests: 100000 total, 100000 started, 100000 done, 100000 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 100000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 10039635360 bytes total, 28300960 bytes headers (space savings -43.66%), 10000000000 bytes data
min max mean sd +/- sd
time for request: 141us 143.19ms 8.21ms 6.62ms 82.22%
time for connect: 117.32ms 1.05s 345.05ms 205.46ms 78.75%
time to 1st byte: 136.24ms 1.05s 363.70ms 203.06ms 77.50%
h2load -n 100000 -c 16 -t 8
NGINX 1.9.7 with HTTP/2
./nghttp2/src/h2load -n 100000 -c 16 -t 8 https://localhost:8443
...
finished in 9.00s, 11107.09 req/s, 1.04GB/s
requests: 100000 total, 100000 started, 100000 done, 100000 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 100000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 10026500784 bytes total, 13900000 bytes headers (space savings 23.20%), 10000000000 bytes data
min max mean sd +/- sd
time for request: 149us 36.99ms 1.28ms 564us 83.56%
time for connect: 31.02ms 92.15ms 63.49ms 21.68ms 56.25%
time to 1st byte: 43.50ms 95.21ms 72.90ms 17.52ms 50.00%
nghttpx
./nghttp2/src/h2load -n 100000 -c 16 -t 8 https://localhost:8444
...
finished in 6.11s, 16378.77 req/s, 1.53GB/s
requests: 100000 total, 100000 started, 100000 done, 100000 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 100000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 10010237791 bytes total, 3003552 bytes headers (space savings 85.35%), 10000000000 bytes data
min max mean sd +/- sd
time for request: 158us 21.15ms 948us 884us 89.33%
time for connect: 16.70ms 36.73ms 28.63ms 5.23ms 75.00%
time to 1st byte: 25.16ms 40.98ms 32.16ms 3.84ms 75.00%
NGINX 1.9.4 with SPDY/3.1
./nghttp2/src/h2load -n 100000 -c 16 -t 8 https://localhost:8445
...
finished in 6.29s, 15908.56 req/s, 1.49GB/s
requests: 100000 total, 100000 started, 100000 done, 100000 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 100000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 10039873536 bytes total, 28300096 bytes headers (space savings -43.66%), 10000000000 bytes data
min max mean sd +/- sd
time for request: 149us 18.86ms 849us 360us 80.72%
time for connect: 21.72ms 47.04ms 40.27ms 7.42ms 87.50%
time to 1st byte: 37.78ms 48.13ms 43.15ms 4.19ms 37.50%
So far, HTTP/2 seems to perform better when dealing with more connections. However, it seems to degrade significantly when dealing with less requests.
Oops. This is just performance for a single file. Now let's check concurrent streams.
h2load -n 100000 -c 16 -m 32 -t 8
NGINX 1.9.7 with HTTP/2
./nghttp2/src/h2load -n 100000 -c 16 -m 32 -t 8 https://localhost:8443
...
finished in 10.72s, 9332.13 req/s, 892.34MB/s
requests: 100000 total, 100000 started, 100000 done, 100000 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 100000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 10026500784 bytes total, 13900000 bytes headers (space savings 23.20%), 10000000000 bytes data
min max mean sd +/- sd
time for request: 2.45ms 72.64ms 54.21ms 2.85ms 94.57%
time for connect: 26.05ms 184.67ms 91.70ms 52.14ms 56.25%
time to 1st byte: 48.44ms 202.94ms 114.07ms 53.54ms 50.00%
nghttpx
./nghttp2/src/h2load -n 100000 -c 16 -m 32 -t 8 https://localhost:8444
...
finished in 4.69s, 21334.53 req/s, 1.99GB/s
requests: 100000 total, 100000 started, 100000 done, 100000 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 100000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 10010265942 bytes total, 3003092 bytes headers (space savings 85.35%), 10000000000 bytes data
min max mean sd +/- sd
time for request: 281us 90.58ms 21.87ms 10.60ms 70.12%
time for connect: 23.81ms 68.39ms 39.73ms 12.54ms 68.75%
time to 1st byte: 31.49ms 99.03ms 54.43ms 17.97ms 81.25%
NGINX 1.9.4 with SPDY/3.1
./nghttp2/src/h2load -n 100000 -c 16 -m 32 -t 8 https://localhost:8445
...
finished in 6.15s, 16268.07 req/s, 1.52GB/s
requests: 100000 total, 100000 started, 100000 done, 100000 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 100000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 10039873536 bytes total, 28300096 bytes headers (space savings -43.66%), 10000000000 bytes data
min max mean sd +/- sd
time for request: 3.09ms 48.01ms 31.06ms 2.26ms 83.73%
time for connect: 23.49ms 69.30ms 44.28ms 14.69ms 56.25%
time to 1st byte: 41.55ms 81.49ms 61.28ms 12.84ms 43.75%
h2load -n 100000 -c 160 -m 32 -t 8
NGINX 1.9.7 with HTTP/2
./nghttp2/src/h2load -n 100000 -c 160 -m 32 -t 8 https://localhost:8443
...
finished in 5.45s, 18350.02 req/s, 1.65GB/s
requests: 100000 total, 100000 started, 100000 done, 96299 succeeded, 3701 failed, 0 errored, 0 timeout
status codes: 96299 2xx, 0 3xx, 0 4xx, 3701 5xx
traffic: 9656430046 bytes total, 13611322 bytes headers (space savings 23.50%), 9630610592 bytes data
min max mean sd +/- sd
time for request: 257us 797.10ms 201.45ms 130.78ms 76.85%
time for connect: 175.21ms 1.06s 498.01ms 273.99ms 48.13%
time to 1st byte: 193.74ms 1.24s 561.38ms 302.62ms 56.88%
nghttpx
./nghttp2/src/h2load -n 100000 -c 160 -m 32 -t 8 https://localhost:8444
...
finished in 5.22s, 19158.39 req/s, 1.75GB/s
requests: 100000 total, 100000 started, 100000 done, 98246 succeeded, 1754 failed, 0 errored, 0 timeout
status codes: 98246 2xx, 0 3xx, 0 4xx, 1754 5xx
traffic: 9835038326 bytes total, 2988756 bytes headers (space savings 85.32%), 9824936768 bytes data
min max mean sd +/- sd
time for request: 253us 3.64s 185.32ms 325.70ms 95.90%
time for connect: 32.78ms 383.88ms 249.10ms 64.89ms 65.00%
time to 1st byte: 259.03ms 3.81s 1.09s 884.18ms 92.50%
Note the 5xx errors.
NGINX 1.9.4 with SPDY/3.1
./nghttp2/src/h2load -n 100000 -c 160 -m 32 -t 8 https://localhost:8445
...
finished in 5.78s, 17312.89 req/s, 1.62GB/s
requests: 100000 total, 100000 started, 100000 done, 100000 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 100000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 10039635360 bytes total, 28300960 bytes headers (space savings -43.66%), 10000000000 bytes data
min max mean sd +/- sd
time for request: 3.64ms 919.23ms 215.67ms 141.88ms 81.66%
time for connect: 31.48ms 1.24s 424.69ms 271.96ms 66.88%
time to 1st byte: 50.11ms 1.39s 513.96ms 281.44ms 73.13%
Seems okay. I think it must be because the HTTP/2 implementation in NGINX is less refined than nghttp2.
Now let's compare it with a HTTP/1.1 test:
NGINX 1.9.7 with HTTP/2 using HTTP/1.1
./nghttp2/src/h2load --h1 -n 100000 -c 160 -m 32 -t 8 https://localhost:8443
...
finished in 4.32s, 18522.10 req/s, 1.73GB/s
requests: 100000 total, 100000 started, 100000 done, 80000 succeeded, 20000 failed, 20000 errored, 0 timeout
status codes: 80000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 8017809600 bytes total, 15276000 bytes headers (space savings 0.00%), 8000000000 bytes data
min max mean sd +/- sd
time for request: 349us 894.19ms 145.46ms 199.32ms 88.71%
time for connect: 282.14ms 824.93ms 496.51ms 73.88ms 80.00%
time to 1st byte: 328.01ms 867.15ms 523.29ms 85.33ms 80.00%
I strongly believe it's because of the fact that I'm testing on my loopback interface, and real-life performance would be much lower due to network conditions.
Also, the drop in transfer rate when using NGINX HTTP/2 at a low number of connections is probably a limitation in NGINX, that also causes CPU utilisation to drop. Interesting. I'll look into that.
Other than that, as expected, nghttpx as a SSL terminator would probably give you the benefit of consistent performance and also SPDY/3.1 support alongside HTTP/2 (of course, you have to compile nghttp2 with SPDY support).