NGINX vs nghttpx

Recently in version 1.9.5, NGINX introduced their experimental HTTP/2 support, purposefully dropping SPDY protocol support in favour of HTTP/2. Having no SPDY support in NGINX would leave about 60% of users capable of SPDY/3.1 having to fall back to HTTP/1.1 connections, thus lowering performance.

So being interested in webserver performance, I decided to do some small benchmarks, as a rough guide to how NGINX and nghttp2 may perform in a deployed environment. Of course, my tests are an inaccurate representation of real life performance: I'm only serving a single 100kb text file through NGINX.

Setup

The tests are done on my Ubuntu machine, the server this very site runs on. It's an i7 4790 desktop with 20GB of RAM, running Ubuntu 15.10 Desktop.

I'm running three tests:

  1. NGINX 1.9.7 with HTTP/2
  2. nghttpx using the above as a backend over HTTP
  3. NGINX 1.9.7 with SPDY/3.1

And here are my compile flags:

nginx version: nginx/1.9.7
built by gcc 5.2.1 20151010 (Ubuntu 5.2.1-22ubuntu2) 
built with OpenSSL 1.0.2d 9 Jul 2015
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-http_ssl_module --with-threads --with-file-aio --with-cc-opt='-g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,--as-needed' --with-http_v2_module
nginx version: nginx/1.9.4
built by gcc 5.2.1 20151010 (Ubuntu 5.2.1-22ubuntu2) 
built with OpenSSL 1.0.2d 9 Jul 2015
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-http_ssl_module --with-threads --with-file-aio --with-cc-opt='-g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,--as-needed' --with-http_spdy_module

nghttp2 was compiled with the default flags.

The config files:

Keep in mind that the NGINX configuration I used is not optimum for high performance. I'll probably follow up with a blog post on better comparison across more configurations and tests.

Results

h2load -n 100000 -c 160 -t 8

NGINX 1.9.7 with HTTP/2

./nghttp2/src/h2load -n 100000 -c 160 -t 8 https://localhost:8443
...

finished in 6.42s, 15571.91 req/s, 1.45GB/s  
requests: 100000 total, 100000 started, 100000 done, 100000 succeeded, 0 failed, 0 errored, 0 timeout  
status codes: 100000 2xx, 0 3xx, 0 4xx, 0 5xx  
traffic: 10026507840 bytes total, 13900000 bytes headers (space savings 23.20%), 10000000000 bytes data  
                     min         max         mean         sd        +/- sd
time for request:      245us     98.86ms      7.93ms      6.33ms    84.34%  
time for connect:    24.08ms       1.32s    374.68ms    429.36ms    83.13%  
time to 1st byte:    46.91ms       1.33s    392.37ms    424.10ms    83.13%  

nghttpx

./nghttp2/src/h2load -n 100000 -c 160 -t 8 https://localhost:8444
...

finished in 6.18s, 16172.23 req/s, 1.51GB/s  
requests: 100000 total, 100000 started, 100000 done, 100000 succeeded, 0 failed, 0 errored, 0 timeout  
status codes: 100000 2xx, 0 3xx, 0 4xx, 0 5xx  
traffic: 10010249408 bytes total, 3035428 bytes headers (space savings 85.19%), 10000000000 bytes data  
                     min         max         mean         sd        +/- sd
time for request:      180us    229.37ms      9.17ms      6.34ms    82.40%  
time for connect:   130.16ms    328.52ms    250.65ms     44.00ms    70.63%  
time to 1st byte:   221.39ms    407.43ms    305.03ms     52.53ms    46.25%  

NGINX 1.9.4 with SPDY/3.1

./nghttp2/src/h2load -n 100000 -c 160 -t 8 https://localhost:8445
...

finished in 7.24s, 13821.13 req/s, 1.29GB/s  
requests: 100000 total, 100000 started, 100000 done, 100000 succeeded, 0 failed, 0 errored, 0 timeout  
status codes: 100000 2xx, 0 3xx, 0 4xx, 0 5xx  
traffic: 10039635360 bytes total, 28300960 bytes headers (space savings -43.66%), 10000000000 bytes data  
                     min         max         mean         sd        +/- sd
time for request:      141us    143.19ms      8.21ms      6.62ms    82.22%  
time for connect:   117.32ms       1.05s    345.05ms    205.46ms    78.75%  
time to 1st byte:   136.24ms       1.05s    363.70ms    203.06ms    77.50%  

h2load -n 100000 -c 16 -t 8

NGINX 1.9.7 with HTTP/2

./nghttp2/src/h2load -n 100000 -c 16 -t 8 https://localhost:8443
...

finished in 9.00s, 11107.09 req/s, 1.04GB/s  
requests: 100000 total, 100000 started, 100000 done, 100000 succeeded, 0 failed, 0 errored, 0 timeout  
status codes: 100000 2xx, 0 3xx, 0 4xx, 0 5xx  
traffic: 10026500784 bytes total, 13900000 bytes headers (space savings 23.20%), 10000000000 bytes data  
                     min         max         mean         sd        +/- sd
time for request:      149us     36.99ms      1.28ms       564us    83.56%  
time for connect:    31.02ms     92.15ms     63.49ms     21.68ms    56.25%  
time to 1st byte:    43.50ms     95.21ms     72.90ms     17.52ms    50.00%  

nghttpx

./nghttp2/src/h2load -n 100000 -c 16 -t 8 https://localhost:8444
...

finished in 6.11s, 16378.77 req/s, 1.53GB/s  
requests: 100000 total, 100000 started, 100000 done, 100000 succeeded, 0 failed, 0 errored, 0 timeout  
status codes: 100000 2xx, 0 3xx, 0 4xx, 0 5xx  
traffic: 10010237791 bytes total, 3003552 bytes headers (space savings 85.35%), 10000000000 bytes data  
                     min         max         mean         sd        +/- sd
time for request:      158us     21.15ms       948us       884us    89.33%  
time for connect:    16.70ms     36.73ms     28.63ms      5.23ms    75.00%  
time to 1st byte:    25.16ms     40.98ms     32.16ms      3.84ms    75.00%  

NGINX 1.9.4 with SPDY/3.1

./nghttp2/src/h2load -n 100000 -c 16 -t 8 https://localhost:8445
...

finished in 6.29s, 15908.56 req/s, 1.49GB/s  
requests: 100000 total, 100000 started, 100000 done, 100000 succeeded, 0 failed, 0 errored, 0 timeout  
status codes: 100000 2xx, 0 3xx, 0 4xx, 0 5xx  
traffic: 10039873536 bytes total, 28300096 bytes headers (space savings -43.66%), 10000000000 bytes data  
                     min         max         mean         sd        +/- sd
time for request:      149us     18.86ms       849us       360us    80.72%  
time for connect:    21.72ms     47.04ms     40.27ms      7.42ms    87.50%  
time to 1st byte:    37.78ms     48.13ms     43.15ms      4.19ms    37.50%  

So far, HTTP/2 seems to perform better when dealing with more connections. However, it seems to degrade significantly when dealing with less requests.

Oops. This is just performance for a single file. Now let's check concurrent streams.

h2load -n 100000 -c 16 -m 32 -t 8

NGINX 1.9.7 with HTTP/2

./nghttp2/src/h2load -n 100000 -c 16 -m 32 -t 8 https://localhost:8443
...

finished in 10.72s, 9332.13 req/s, 892.34MB/s  
requests: 100000 total, 100000 started, 100000 done, 100000 succeeded, 0 failed, 0 errored, 0 timeout  
status codes: 100000 2xx, 0 3xx, 0 4xx, 0 5xx  
traffic: 10026500784 bytes total, 13900000 bytes headers (space savings 23.20%), 10000000000 bytes data  
                     min         max         mean         sd        +/- sd
time for request:     2.45ms     72.64ms     54.21ms      2.85ms    94.57%  
time for connect:    26.05ms    184.67ms     91.70ms     52.14ms    56.25%  
time to 1st byte:    48.44ms    202.94ms    114.07ms     53.54ms    50.00%  

nghttpx

./nghttp2/src/h2load -n 100000 -c 16 -m 32 -t 8 https://localhost:8444
...

finished in 4.69s, 21334.53 req/s, 1.99GB/s  
requests: 100000 total, 100000 started, 100000 done, 100000 succeeded, 0 failed, 0 errored, 0 timeout  
status codes: 100000 2xx, 0 3xx, 0 4xx, 0 5xx  
traffic: 10010265942 bytes total, 3003092 bytes headers (space savings 85.35%), 10000000000 bytes data  
                     min         max         mean         sd        +/- sd
time for request:      281us     90.58ms     21.87ms     10.60ms    70.12%  
time for connect:    23.81ms     68.39ms     39.73ms     12.54ms    68.75%  
time to 1st byte:    31.49ms     99.03ms     54.43ms     17.97ms    81.25%  

NGINX 1.9.4 with SPDY/3.1

./nghttp2/src/h2load -n 100000 -c 16 -m 32 -t 8 https://localhost:8445
...

finished in 6.15s, 16268.07 req/s, 1.52GB/s  
requests: 100000 total, 100000 started, 100000 done, 100000 succeeded, 0 failed, 0 errored, 0 timeout  
status codes: 100000 2xx, 0 3xx, 0 4xx, 0 5xx  
traffic: 10039873536 bytes total, 28300096 bytes headers (space savings -43.66%), 10000000000 bytes data  
                     min         max         mean         sd        +/- sd
time for request:     3.09ms     48.01ms     31.06ms      2.26ms    83.73%  
time for connect:    23.49ms     69.30ms     44.28ms     14.69ms    56.25%  
time to 1st byte:    41.55ms     81.49ms     61.28ms     12.84ms    43.75%  

h2load -n 100000 -c 160 -m 32 -t 8

NGINX 1.9.7 with HTTP/2

./nghttp2/src/h2load -n 100000 -c 160 -m 32 -t 8 https://localhost:8443
...

finished in 5.45s, 18350.02 req/s, 1.65GB/s  
requests: 100000 total, 100000 started, 100000 done, 96299 succeeded, 3701 failed, 0 errored, 0 timeout  
status codes: 96299 2xx, 0 3xx, 0 4xx, 3701 5xx  
traffic: 9656430046 bytes total, 13611322 bytes headers (space savings 23.50%), 9630610592 bytes data  
                     min         max         mean         sd        +/- sd
time for request:      257us    797.10ms    201.45ms    130.78ms    76.85%  
time for connect:   175.21ms       1.06s    498.01ms    273.99ms    48.13%  
time to 1st byte:   193.74ms       1.24s    561.38ms    302.62ms    56.88%  

nghttpx

./nghttp2/src/h2load -n 100000 -c 160 -m 32 -t 8 https://localhost:8444
...

finished in 5.22s, 19158.39 req/s, 1.75GB/s  
requests: 100000 total, 100000 started, 100000 done, 98246 succeeded, 1754 failed, 0 errored, 0 timeout  
status codes: 98246 2xx, 0 3xx, 0 4xx, 1754 5xx  
traffic: 9835038326 bytes total, 2988756 bytes headers (space savings 85.32%), 9824936768 bytes data  
                     min         max         mean         sd        +/- sd
time for request:      253us       3.64s    185.32ms    325.70ms    95.90%  
time for connect:    32.78ms    383.88ms    249.10ms     64.89ms    65.00%  
time to 1st byte:   259.03ms       3.81s       1.09s    884.18ms    92.50%  

Note the 5xx errors.

NGINX 1.9.4 with SPDY/3.1

./nghttp2/src/h2load -n 100000 -c 160 -m 32 -t 8 https://localhost:8445
...

finished in 5.78s, 17312.89 req/s, 1.62GB/s  
requests: 100000 total, 100000 started, 100000 done, 100000 succeeded, 0 failed, 0 errored, 0 timeout  
status codes: 100000 2xx, 0 3xx, 0 4xx, 0 5xx  
traffic: 10039635360 bytes total, 28300960 bytes headers (space savings -43.66%), 10000000000 bytes data  
                     min         max         mean         sd        +/- sd
time for request:     3.64ms    919.23ms    215.67ms    141.88ms    81.66%  
time for connect:    31.48ms       1.24s    424.69ms    271.96ms    66.88%  
time to 1st byte:    50.11ms       1.39s    513.96ms    281.44ms    73.13%  

Seems okay. I think it must be because the HTTP/2 implementation in NGINX is less refined than nghttp2.

Now let's compare it with a HTTP/1.1 test:

NGINX 1.9.7 with HTTP/2 using HTTP/1.1

./nghttp2/src/h2load --h1 -n 100000 -c 160 -m 32 -t 8 https://localhost:8443
...

finished in 4.32s, 18522.10 req/s, 1.73GB/s  
requests: 100000 total, 100000 started, 100000 done, 80000 succeeded, 20000 failed, 20000 errored, 0 timeout  
status codes: 80000 2xx, 0 3xx, 0 4xx, 0 5xx  
traffic: 8017809600 bytes total, 15276000 bytes headers (space savings 0.00%), 8000000000 bytes data  
                     min         max         mean         sd        +/- sd
time for request:      349us    894.19ms    145.46ms    199.32ms    88.71%  
time for connect:   282.14ms    824.93ms    496.51ms     73.88ms    80.00%  
time to 1st byte:   328.01ms    867.15ms    523.29ms     85.33ms    80.00%  

I strongly believe it's because of the fact that I'm testing on my loopback interface, and real-life performance would be much lower due to network conditions.

Also, the drop in transfer rate when using NGINX HTTP/2 at a low number of connections is probably a limitation in NGINX, that also causes CPU utilisation to drop. Interesting. I'll look into that.

Other than that, as expected, nghttpx as a SSL terminator would probably give you the benefit of consistent performance and also SPDY/3.1 support alongside HTTP/2 (of course, you have to compile nghttp2 with SPDY support).