There’s a myth in the industry that performance penalties apply as soon as you virtualise. This may have been true some time ago but we’d argue that modern technology implementations are eroding this out-of-date view!
As part of the work we are doing with a stock exchange facility, we have been investigating the implications of moving towards a software defined virtualised environment. The organisation in question was interested in combining their development and production environment into a single on-premise cloud to ease management and enable fast internal development of features.
Their existing setup had two separate platforms which meant migrating applications between environments was a little more time-consuming than perhaps it should have been. The performance in their test environment was also significantly different to the production environment as a result of how the existing virtualised infrastructure was implemented.
Our goal was to explore SR-IOV and physical function passthrough on the customer preferred NIC vendor (which happened to be Solarflare).
The test environment consisted of two servers each directly connected to one another (without a switch). We investigated the performance on multiple generations of CPUs with ranging frequencies to observe the impact of frequency on low-latency, small-packet traffic. We were also fortunate enough to get our hands on some overclocked HFT servers which we populated with the Solarflare XtremeScale SFC9250. The tests would investigate the bare metal performance of this environment and then we would reproduce the test running virtualised on top of OpenStack (KVM) on the same hardware.
To begin we setup the Solarflare drivers and set the base environment (note: environment was Centos7 minimal to begin with)
# Install dependancies and SF modules
yum -y groupinstall 'Development tools'
yum -y install kernel-devel wget numactl
# install openonload
wget https://openonload.org/download/openonload-201811-u1.tgz
tar zxvf openonload-201811-u1.tgz
./openonload-201811-u1/scripts/onload_install
# get the sfntest too (Solarflare net test)
wget https://www.openonload.org/download/sfnettest/sfnettest-1.5.0.tgz
cd sfnettest-1.5.0/src
make
# note to check and see what firmware is running just: sfboot
# to factory reset: sfboot -i enp139s0f0 --factory-reset (x2 card)
# change the firmware to full feature (need this when using passthrough for VMs)
[root@solarflare01 Solarflare]# sfboot --adapter=enp139s0f0 firmware-variant=full-feature
# set the low latency firmware (use this when doing perf tests)
[root@solarflare01 ~]# sfboot --adapter=enp139s0f0 firmware-variant=ultra-low-latency
# note: cold power cycle just run
ipmitool power cycle
With the above in place on both servers we can start to look at the tuning. As the servers we are using are dual socket (2 physical CPUs, each with multiple local cores and PCI connectivity) we need to be careful with numa placement of the workloads. For latency tests we want to run the benchmarks on the cores with the closest proximity to the NIC. There’s a few tools we can use to help us identify the cores and devices per numa node.
These initial tests were also on some older hardware where the CPU used was Intel E5-2620v4 which runs at 2.1Ghz (so these are worst case results for the SF NIC)
# what numa node is the interface connected to
cat /sys/class/net/enp139s0f0/device/numa_node
# what cores are in that numa node
cat /sys/devices/system/node/node1/cpulist
# run latency tests (based on ultra-low-latency profile) in this instance core 8 was connected directly to the SF NIC.
# on one node run
[zippy@solarflare02 src]# onload --profile=latency taskset -c 8 ./sfnt-pingpong
# then on the other node
[zippy@solarflare02 src]# onload --profile=latency taskset -c 8 ./sfnt-pingpong --affinity "1;1" tcp 192.168.10.5
oo:sfnt-pingpong[21133]: Using OpenOnload 201811-u1 Copyright 2006-2019 Solarflare Communications, 2002-2005 Level 5 Networks [0]
# cmdline: ./sfnt-pingpong --affinity 1;1 tcp 192.168.10.5
# version: 1.5.0
# src: 8dc3b027d85b28bedf9fd731362e4968
# date: Fri 19 Jul 14:27:10 BST 2019
# uname: Linux solarflare02.testing 3.10.0-957.21.3.el7.x86_64 #1 SMP Tue Jun 18 16:35:19 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
# cpu: model name : Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz
# lspci: 05:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
# lspci: 05:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
# lspci: 8b:00.0 Ethernet controller: Solarflare Communications XtremeScale SFC9250 10/25/40/50/100G Ethernet Controller (rev 01)
# lspci: 8b:00.1 Ethernet controller: Solarflare Communications XtremeScale SFC9250 10/25/40/50/100G Ethernet Controller (rev 01)
# enp139s0f0: driver: sfc
# enp139s0f0: version: 4.15.3.1011
# enp139s0f0: bus-info: 0000:8b:00.0
# enp139s0f1: driver: sfc
# enp139s0f1: version: 4.15.3.1011
# enp139s0f1: bus-info: 0000:8b:00.1
# enp5s0f0: driver: igb
# enp5s0f0: version: 5.4.0-k
# enp5s0f0: bus-info: 0000:05:00.0
# enp5s0f1: driver: igb
# enp5s0f1: version: 5.4.0-k
# enp5s0f1: bus-info: 0000:05:00.1
# ram: MemTotal: 65693408 kB
# tsc_hz: 2099969430
# LD_PRELOAD=libonload.so
# onload_version=201811-u1
# EF_TCP_FASTSTART_INIT=0
# EF_POLL_USEC=100000
# EF_TCP_FASTSTART_IDLE=0
# server LD_PRELOAD=libonload.so
# percentile=99
#
# size mean min median max %ile stddev iter
1 1768 1676 1719 50581 2166 158 842000
2 1729 1680 1720 21333 1839 84 862000
4 1730 1681 1720 20046 1844 92 861000
8 1731 1682 1722 20967 1850 92 861000
16 1739 1684 1730 19944 1867 89 857000
32 1752 1694 1740 10220 1877 90 851000
64 1801 1745 1791 21847 1931 97 828000
128 1893 1824 1875 14862 2029 98 788000
256 2074 1930 2069 27097 2236 111 719000
512 2300 2176 2289 11293 2469 103 649000
1024 2606 2475 2585 23790 2919 141 573000
2048 3857 3521 3829 13734 4398 191 388000
4096 4664 4401 4622 22394 5181 186 321000
8192 6547 6214 6491 15511 7508 240 229000
16384 9997 9540 9946 28038 11050 293 150000
32768 16476 15478 16419 34062 18176 432 91000
65536 29156 27571 29151 54446 31152 647 52000
These are our baseline figures to start with. The numbers above confirm that our systems are capable of around 1.7usec (microseconds) These are the bare metal servers and the numbers to beat when we compare our virtualised setup.
Out of curiosity, we wanted to understand what sort of impact the Solarflare firmware type on performance. Let’s switch the full feature firmware back on and take a look…
# redo with the standard full feature firmware
$ sfboot firmware-variant=full-feature
$ ipmitool power cycle
[root@solarflare01 src]# sfboot | grep -i variant
Firmware variant Full feature / virtualization
Firmware variant Full feature / virtualization
# size mean min median max %ile stddev iter
1 2248 2022 2071 4200847 3239 6349 663000
2 2209 2022 2075 889552 3158 3298 675000
4 2207 2018 2078 865345 3132 3096 676000
8 2199 2035 2093 4827169 2692 6848 678000
16 2241 2045 2092 219934 3027 3539 666000
32 2230 2038 2102 4414673 3319 6302 669000
64 2244 2064 2128 4589940 3179 6427 665000
128 2269 2105 2170 185278 3305 1587 658000
256 2522 2220 2366 4359659 4037 6899 592000
512 2588 2329 2460 1322388 3893 3480 577000
1024 2822 2529 2614 4219115 8031 6869 530000
2048 3973 3566 3742 4108063 8391 7989 377000
4096 5064 4480 4745 270765 11188 4253 296000
8192 7103 6283 6700 4363507 13410 11234 211000
16384 11054 9628 10243 1432292 18087 10285 136000
32768 18228 15645 16971 4225292 34181 16490 83000
65536 30909 27770 29450 4078629 66407 22086 49000
You’ll see that’s quite a jump from 1.7usec up to 2.2usec. Certainly worth keeping in mind for latency sensitive applications.
That’s the baseline numbers recorded – we now move on to the testing in our OpenStack environment. For this we are leveraging SR-IOV, PCI passthrough and the support for physical functions in the SF NIC. Let’s run through the steps to set this up.
# enable 4 physical functions per single SF NIC
sfboot switch-mode=pfiov pf-count=4
# we need to update the kernel to support intel_iommu
grubby --update-kernel=ALL --args="intel_iommu=on"
grubby --info=ALL
# reboot
Now let’s check the PCI entries on the server and collect the PCI IDs as we’ll need these to pass to the nova configuration in OpenStack for PCIPassthrough.
# lets get the ID's of the devices to pass-through
[root@solarflare02 ~]# lspci -nn | grep -i Solarflare | head -n 1
8b:00.0 Ethernet controller [0200]: Solarflare Communications XtremeScale SFC9250 10/25/40/50/100G Ethernet Controller [1924:0b03] (rev 01)
There’s a few parameters we need to input in to OpenStack to instruct it to use these devices rather than the standard virtual NIC.
# nova.conf
[filter_scheduler]
enabled_filters = RetryFilter, AvailabilityZoneFilter, RamFilter, ComputeFilter, ComputeCapabilitiesFilter, ImagePropertiesFilter, ServerGroupAntiAffinityFilter, ServerGroupAffinityFilter, PciPassthroughFilter
available_filters=nova.scheduler.filters.all_filters
# these IDs are taken from the lspci command above
[pci]
passthrough_whitelist={"vendor_id":"1924","product_id":"0b03"}
alias={"name":"SolarFlareNIC","vendor_id":"1924","product_id":"0b03"}
# reconfigure / restart services as required
# flavor setup
openstack flavor create --public --ram 4096 --disk 10 --vcpus 4 m1.large.sfn
openstack flavor set m1.large.sfn --property pci_passthrough:alias='SolarFlareNIC:1'
# spin up VM
openstack server create --image centos7 --flavor m1.large.sfn --key-name mykey --network demo-net sfn1
# repeat for second virtual server placing it on the other physical system using availability zones
Ok we are now ready to perform a repeat of the tests from earlier. We re-install all the drivers and benchmark tools as documented above and run sfnt again.
# results within a VM
[zippy@sfn1 src]# onload --profile=latency ./sfnt-pingpong --affinity "1;1" tcp 192.168.10.1
onload: Note: Disabling CTPIO cut-through because only adapters running at 10GbE benefit from it
oo:sfnt-pingpong[20195]: Using OpenOnload 201811-u1 Copyright 2006-2019 Solarflare Communications, 2002-2005 Level 5 Networks [0]
# cmdline: ./sfnt-pingpong --affinity 1;1 tcp 192.168.10.1
# version: 1.5.0
# src: 8dc3b027d85b28bedf9fd731362e4968
# date: Tue Jul 30 23:26:52 UTC 2019
# uname: Linux sfn8.novalocal 3.10.0-957.21.3.el7.x86_64 #1 SMP Tue Jun 18 16:35:19 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
# cpu: model name : Intel Core Processor =191aa(Broadwell)
sh: /sbin/lspci: No such file or directory
# eth0: driver: virtio_net
# eth0: version: 1.0.0
# eth0: bus-info: 0000:00:03.0
# eth1: driver: sfc
# eth1: version: 4.15.3.1011
# eth1: bus-info: 0000:00:05.0
# ram: MemTotal: 3880088 kB
# tsc_hz: 2099966200
# LD_PRELOAD=libonload.so
# onload_version=201811-u1
# EF_TCP_FASTSTART_INIT=0
# EF_CTPIO_SWITCH_BYPASS=1
# EF_POLL_USEC=100000
# EF_TCP_FASTSTART_IDLE=0
# EF_EPOLL_MT_SAFE=1
# server LD_PRELOAD=libonload.so
# percentile=99
#
# size mean min median max %ile stddev iter
1 1942 1725 1833 273704 4923 802 766000
2 1900 1727 1816 657833 4651 1035 783000
4 1891 1735 1817 402257 4585 849 787000
8 1860 1666 1803 223816 4601 775 800000
16 1785 1638 1718 773823 4202 1102 834000
32 1836 1661 1745 157788 4279 649 811000
64 1903 1717 1789 425864 4582 897 782000
128 2052 1860 1954 537661 5154 978 726000
256 2247 1994 2147 75286 5532 771 663000
512 2541 2240 2405 307319 6814 939 587000
1024 2642 2335 2545 24229 5824 701 565000
2048 3539 3210 3433 2107848 6482 3312 422000
4096 4558 4139 4400 443807 8669 1140 328000
8192 7038 6009 6652 2194215 13286 4967 213000
16384 10645 9503 10182 291544 18314 1944 141000
32768 15752 14786 15096 502919 22343 2251 96000
65536 29551 26566 28286 352787 40923 3319 51000
And bingo! Reasonable latency figures from a virtualised environment. When you look at the 16 byte packets sizes the results are virtually identical to the bare metal environment, although on average there is a small penalty for virtualisation but that’s a penalty many would accept for some of the function benefits that virtualisation can bring to environments (snapshots, backup, migration, rapid deployment etc.)
Now we had the process down we wanted to check and see how this scaled and performed on some higher end hardware ie. The Supermicro-based HFT boxes running Cascade Lake CPUs (6246 @ 3.3Ghz).
# updated on the LL2 cascade lake box
[root@solarflare02-ll2 src]# onload --profile=latency taskset -c 12 ./sfnt-pingpong --affinity "1;1" tcp 192.168.10.5
oo:sfnt-pingpong[27963]: Using OpenOnload 201811-u1 Copyright 2006-2019 Solarflare Communications, 2002-2005 Level 5 Networks [0]
# cmdline: ./sfnt-pingpong --affinity 1;1 tcp 192.168.10.5
# version: 1.5.0
# src: 8dc3b027d85b28bedf9fd731362e4968
# date: Thu 1 Aug 16:00:30 BST 2019
# uname: Linux solarflare02-ll2.internal 3.10.0-957.el7.x86_64 #1 SMP Thu Nov 8 23:39:32 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
# cpu: model name : Intel(R) Xeon(R) Gold 6246 CPU @ 3.30GHz
# lspci: 3b:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
# lspci: 3b:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
# lspci: 3b:00.2 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
# lspci: 3b:00.3 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
# lspci: 5e:00.0 Ethernet controller: Solarflare Communications SFC9220 10/40G Ethernet Controller (rev 02)
# lspci: 5e:00.1 Ethernet controller: Solarflare Communications SFC9220 10/40G Ethernet Controller (rev 02)
# lspci: 86:00.0 Ethernet controller: Solarflare Communications XtremeScale SFC9250 10/25/40/50/100G Ethernet Controller (rev 01)
# lspci: 86:00.1 Ethernet controller: Solarflare Communications XtremeScale SFC9250 10/25/40/50/100G Ethernet Controller (rev 01)
# enp134s0f0: driver: sfc
# enp134s0f0: version: 4.15.3.1011
# enp134s0f0: bus-info: 0000:86:00.0
# enp134s0f1: driver: sfc
# enp134s0f1: version: 4.15.3.1011
# enp134s0f1: bus-info: 0000:86:00.1
# enp59s0f0: driver: igb
# enp59s0f0: version: 5.4.0-k
# enp59s0f0: bus-info: 0000:3b:00.0
# enp59s0f1: driver: igb
# enp59s0f1: version: 5.4.0-k
# enp59s0f1: bus-info: 0000:3b:00.1
# enp59s0f2: driver: igb
# enp59s0f2: version: 5.4.0-k
# enp59s0f2: bus-info: 0000:3b:00.2
# enp59s0f3: driver: igb
# enp59s0f3: version: 5.4.0-k
# enp59s0f3: bus-info: 0000:3b:00.3
# enp94s0f0: driver: sfc
# enp94s0f0: version: 4.15.3.1011
# enp94s0f0: bus-info: 0000:5e:00.0
# enp94s0f1: driver: sfc
# enp94s0f1: version: 4.15.3.1011
# enp94s0f1: bus-info: 0000:5e:00.1
# ram: MemTotal: 196463288 kB
# tsc_hz: 3366144400
# LD_PRELOAD=libonload.so
# onload_version=201811-u1
# EF_TCP_FASTSTART_INIT=0
# EF_POLL_USEC=100000
# EF_TCP_FASTSTART_IDLE=0
# server LD_PRELOAD=libonload.so
# percentile=99
#
# size mean min median max %ile stddev iter
1 1426 1398 1421 39777 1582 67 1000000
2 1428 1396 1422 18082 1584 45 1000000
4 1428 1398 1422 18566 1582 47 1000000
8 1433 1405 1427 5724 1592 43 1000000
16 1439 1412 1434 17545 1581 42 1000000
32 1455 1428 1450 6005 1601 35 1000000
64 1497 1466 1492 17158 1603 41 996000
128 1574 1541 1570 18502 1655 38 947000
256 1716 1668 1714 8920 1806 35 869000
512 1849 1800 1846 15524 1910 38 807000
1024 2441 2355 2433 22335 2626 76 612000
2048 3829 3688 3818 26569 4055 79 391000
4096 5664 5554 5650 14221 5865 60 265000
8192 9451 9321 9439 17686 9679 73 159000
16384 16701 16540 16684 19869 16972 84 90000
32768 30656 30489 30644 44469 30973 111 49000
65536 58556 58389 58540 60230 58895 93 26000
# ll2 with XtremeScale SFC9250 (however on a 10GB cable, not 25GB)
[root@solarflare02-ll2 src]# onload --profile=latency taskset -c 12 ./sfnt-pingpong --affinity "1;1" tcp 192.168.25.5
oo:sfnt-pingpong[28262]: Using OpenOnload 201811-u1 Copyright 2006-2019 Solarflare Communications, 2002-2005 Level 5 Networks [4]
# cmdline: ./sfnt-pingpong --affinity 1;1 tcp 192.168.25.5
# version: 1.5.0
# src: 8dc3b027d85b28bedf9fd731362e4968
# date: Thu 1 Aug 16:23:39 BST 2019
# uname: Linux solarflare02-ll2.internal 3.10.0-957.el7.x86_64 #1 SMP Thu Nov 8 23:39:32 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
# cpu: model name : Intel(R) Xeon(R) Gold 6246 CPU @ 3.30GHz
# lspci: 3b:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
# lspci: 3b:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
# lspci: 3b:00.2 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
# lspci: 3b:00.3 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
# lspci: 5e:00.0 Ethernet controller: Solarflare Communications SFC9220 10/40G Ethernet Controller (rev 02)
# lspci: 5e:00.1 Ethernet controller: Solarflare Communications SFC9220 10/40G Ethernet Controller (rev 02)
# lspci: 86:00.0 Ethernet controller: Solarflare Communications XtremeScale SFC9250 10/25/40/50/100G Ethernet Controller (rev 01)
# lspci: 86:00.1 Ethernet controller: Solarflare Communications XtremeScale SFC9250 10/25/40/50/100G Ethernet Controller (rev 01)
# enp134s0f0: driver: sfc
# enp134s0f0: version: 4.15.3.1011
# enp134s0f0: bus-info: 0000:86:00.0
# enp134s0f1: driver: sfc
# enp134s0f1: version: 4.15.3.1011
# enp134s0f1: bus-info: 0000:86:00.1
# enp59s0f0: driver: igb
# enp59s0f0: version: 5.4.0-k
# enp59s0f0: bus-info: 0000:3b:00.0
# enp59s0f1: driver: igb
# enp59s0f1: version: 5.4.0-k
# enp59s0f1: bus-info: 0000:3b:00.1
# enp59s0f2: driver: igb
# enp59s0f2: version: 5.4.0-k
# enp59s0f2: bus-info: 0000:3b:00.2
# enp59s0f3: driver: igb
# enp59s0f3: version: 5.4.0-k
# enp59s0f3: bus-info: 0000:3b:00.3
# enp94s0f0: driver: sfc
# enp94s0f0: version: 4.15.3.1011
# enp94s0f0: bus-info: 0000:5e:00.0
# enp94s0f1: driver: sfc
# enp94s0f1: version: 4.15.3.1011
# enp94s0f1: bus-info: 0000:5e:00.1
# ram: MemTotal: 196463288 kB
# tsc_hz: 3366141000
# LD_PRELOAD=libonload.so
# onload_version=201811-u1
# EF_TCP_FASTSTART_INIT=0
# EF_POLL_USEC=100000
# EF_TCP_FASTSTART_IDLE=0
# server LD_PRELOAD=libonload.so
# percentile=99
#
# size mean min median max %ile stddev iter
1 1384 1314 1376 30745 1551 68 1000000
2 1386 1319 1377 16738 1550 54 1000000
4 1389 1325 1381 14021 1554 51 1000000
8 1395 1334 1388 5473 1560 50 1000000
16 1403 1340 1395 25656 1571 61 1000000
32 1421 1361 1412 8571 1594 53 1000000
64 1473 1412 1465 8125 1648 48 1000000
128 1581 1516 1572 17270 1771 55 943000
256 1736 1626 1728 15769 1943 63 860000
512 2111 1974 2103 5844 2332 65 707000
1024 2762 2636 2748 16447 3018 84 541000
2048 4118 3941 4098 17379 4424 95 364000
4096 6006 5837 5990 19417 6298 92 250000
8192 9835 9589 9817 20922 10207 125 153000
16384 16972 16732 16952 31324 17381 136 89000
32768 31007 30723 30990 41647 31396 139 49000
65536 58915 58605 58891 71188 59351 160 26000
Latency plummets to 1.3usec and even more impressive is the standard deviation (stddev col) which shows very little jitter or variance on the results. Consistent low latency performance. How will the VMs stack up against this uptake in performance?
# Results of two HFT nodes VM (same flavor as above)
[root@sfn10 src]# onload --profile=latency taskset -c 0 ./sfnt-pingpong --affinity "1;1" tcp 192.168.10.2
oo:sfnt-pingpong[20567]: Using OpenOnload 201811-u1 Copyright 2006-2019 Solarflare Communications, 2002-2005 Level 5 Networks [5]
# cmdline: ./sfnt-pingpong --affinity 1;1 tcp 192.168.10.2
# version: 1.5.0
# src: 8dc3b027d85b28bedf9fd731362e4968
# date: Tue 20 Aug 23:00:19 UTC 2019
# uname: Linux sfn10 3.10.0-957.27.2.el7.x86_64 #1 SMP Mon Jul 29 17:46:05 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
# cpu: model name : Intel Xeon Processor (Skylake, IBRS)
# lspci: 00:03.0 Ethernet controller: Red Hat, Inc. Virtio network device
# lspci: 00:05.0 Ethernet controller: Solarflare Communications XtremeScale SFC9250 10/25/40/50/100G Ethernet Controller (rev 01)
# ens5: driver: sfc
# ens5: version: 4.15.3.1011
# ens5: bus-info: 0000:00:05.0
# eth1: driver: virtio_net
# eth1: version: 1.0.0
# eth1: bus-info: 0000:00:03.0
# ram: MemTotal: 3880072 kB
# tsc_hz: 3366133080
# LD_PRELOAD=libonload.so
# onload_version=201811-u1
# EF_TCP_FASTSTART_INIT=0
# EF_POLL_USEC=100000
# EF_TCP_FASTSTART_IDLE=0
# server LD_PRELOAD=libonload.so
# percentile=99
#
# size mean min median max %ile stddev iter
1 1572 1444 1503 291943 2593 388 947000
2 1663 1577 1630 292976 2911 415 896000
4 1661 1578 1631 194620 2859 321 897000
8 1668 1577 1637 271314 2890 366 893000
16 1533 1453 1486 64385 2538 207 971000
32 1526 1468 1499 240736 2461 380 976000
64 1569 1508 1543 238612 2468 335 949000
128 1651 1586 1623 52374 2899 229 903000
256 1794 1717 1771 293475 2779 434 831000
512 1923 1827 1903 316119 2793 456 776000
1024 2417 2346 2394 52744 3193 197 618000
2048 3798 3650 3766 291841 4781 617 394000
4096 5677 5519 5643 70912 6694 335 264000
8192 9431 9249 9384 826605 10561 2068 159000
16384 16706 16502 16665 269437 17901 939 90000
32768 30649 30414 30603 181373 32231 793 49000
65536 58755 58383 58683 296877 60411 1645 26000
The latency numbers again are slightly higher but 1.6usec from a VM across the network is not a bad result at all. We were set a target of sub 2 usec and comfortably passed that with high end hardware and even managed to creep in under that with previously generation hardware as well.
For those environments that need bare metal performance (ie. trading/production), we can still treat servers as consumable resources (our OpenStack environment supports bare metal provisioning) and as demonstrated above, if you need to virtualise (test/dev) you can still achieve very high levels of performance.
If you need performance and flexibility in your private cloud environment, talk to Define Tech!
Pingback: HFT Infrastructure Management – Define Tech