If I understand this correctly then the huge improvement in latency (from 200ms to 3ms) comes from not having to deal with slow clients directly. Traffic to your front-end server are now only from ELB, and ELB is "spoon-feeding" the web-clients. This is true if you are using ELB in "http-mode".
This also explains why you can cut the front-end servers by 20% - as each request is handled more efficiently (lower latency equals higher throughput). Also, connection-reuse is more efficient as the set of servers in the ELB-pool is more limited that the set of web-clients.
ELBs have terrible TLS support... Cipher suite choice and ordering support is abysmal, and they only recently started supporting newer TLS versions. OCSP stapling isn't supported either.<p><a href="https://wiki.mozilla.org/Security/Server_Side_TLS#Amazon_Web_Services_Elastic_Load_Balancer_.28AWS_ELB.29" rel="nofollow">https://wiki.mozilla.org/Security/Server_Side_TLS#Amazon_Web...</a> gives more information if you're curious.