Load testing is hard and the tools are not great

135 点作者 ntietz超过 4 年前

24 条评论

imiric超过 4 年前

I agree that the popular load testing tools leave plenty to be desired, but have you given k6[1] a try? (Full disclosure: I'm one of the maintainers.)Tests are written in JavaScript and there's support for HTTP, WebSockets and (unary) gRPC. You can easily script a combination of these protocols to mimic real world traffic.Furthermore you can record a user flow with a browser extension[2] and convert the generated HAR file to a k6 script[3], which would give you an even closer real world scenario. The conversion is not perfect and depending on the service you might need to manually modify the script, but it would get you 90% of the way there.[1]: <a href="https://github.com/loadimpact/k6" rel="nofollow">https://github.com/loadimpact/k6</a>[2]: <a href="https://k6.io/docs/test-authoring/recording-a-session/browser-recorder" rel="nofollow">https://k6.io/docs/test-authoring/recording-a-session/browse...</a>[3]: <a href="https://github.com/loadimpact/har-to-k6" rel="nofollow">https://github.com/loadimpact/har-to-k6</a>

评论 #25646019 未加载

jillesvangurp超过 4 年前

I've used vegeta (<a href="https://github.com/tsenart/vegeta" rel="nofollow">https://github.com/tsenart/vegeta</a>) on a couple of projects. Easy to setup and use.IMHO how you load test is almost as important as where you load test and understanding why. There are three separate reasons to do load testing:1) Performance testing. Confirm the system does not degrade under a specified load and find out what performance can be expected under these circumstances. This is basically ensuring your system can handle X amount of traffic without issues and knowing your baseline performance. You should get the same kind of response times as you are getting from live server telemetry.2) Stress testing. finding out what happens when the system is stressed beyond its specs. How does it degrade & where.3) Reliability testing. Find out how your system breaks and when. The goal here is to try to break the system and test things like failover and making sure you don't lose or corrupt data. Better to die gracefully then abrubtly.If that's new to you, you've been probably doing it wrong because those 3 require very different approaches.I tend to avoid having to do load testing as it sucks up time without telling me much if interest. I instead opt for having decent telemetry on the live system. It will tell me how it performs and where the bottlenecks are. I can set alerts and take action when things degrade (e.g. because of a bad change). Besides there is no substitute for having real users doing real things with real data. And in any case, having telemetry is crucial to do any meaningful stress or reliability testing. Otherwise you just know it degrades without understanding why.There are still valid reasons to do separate load testing of course but I seem to get away with mostly not doing this. When I do, vegeta is what I reach for first. I find most of the need for load testing is requirements related to SLAs or otherwise nervous product owners that need to be reassured. They tend to be more interested in performance tests (i.e. the happy case) whereas technically stress testing is where you learn stuff about your system.

评论 #25647198 未加载

streamer45超过 4 年前

At Mattermost we went for the do-it-yourself option and wrote a custom tool for the job [1]. After a lot of research on all the existing open-source frameworks we couldn't really find anything that would fit our use-case. We are quite happy with the result although, as the OP mentioned, there's a significant maintenance cost attached. As new features gets implemented and more API calls added you need to go back and make sure your user behaviour defining logic stays in sync with the real world. If I were to do it all over again, I'd probably give k6 [2] a chance but I am still convinced a tailored solution was the best choice.[1] <a href="https://mattermost.com/blog/improving-performance-through-load-testing/" rel="nofollow">https://mattermost.com/blog/improving-performance-through-lo...</a>[2] <a href="https://k6.io/" rel="nofollow">https://k6.io/</a>

swiftcoder超过 4 年前

Last time I had to do load testing in a professional context, we required 10+ million long-lived TCP and websocket connections transacting multiple times a second. There weren't any off-the-shelf solutions at that time that could come within an order of magnitude of that at a reasonable cost - the most viable solution we tested required a thousand EC2 instances to sustain that traffic.Ended up hand-coding a load test framework in a few hundred lines of highly async epoll-based C. Scenarios controlled by a simple configuration syntax in a text file. Trivially generated enough load to break the load balancer in front, our service, and all the services and databases behind it - all this running on just a handful of instances.Pre-existing tools are great, but often don't solve the problem to hand. If load test frameworks aren't doing it for you, don't be afraid to write your own.

评论 #25646072 未加载

yabones超过 4 年前

I've used Jmeter quite a bit over the last year, and what seems to be the largest issue is that it doesn't "break" apps because it works at the protocol level and isn't a full "browser". As you increase load and response times start to increase, since it runs sequentially through the test plan the time between the requests also increases. But, when actual humans with browsers use apps there are loads of AJAX requests being fired which don't necessarily go in order or wait for others to complete first.What I found is to really bring things to the breaking point you actually need two tests at the same time, one to generate static load, and another to spank it. Of course, most people don't need to do this - we had to bring our app to its knees to find a particularly elusive (possibly concurrency related) bug that only appeared with high load & many active sessions.At the end of the day, I think load testing will only get you to ~80% accuracy. I completely agree with the author about the conclusion.

vinay_ys超过 4 年前

Rather than load testing, do red line testing.That is – gradually increase load on your service with concurrent requests until it is saturated.Measure how the latency and errors progress with more concurrent requests and understand at what point and how your service starts to breakdown under heavy load.Based on this you may have to do many things –1. Can you optimize your service or your service's downstream dependencies or the application calling your service.2. Can you build in graceful degradation into your service – functionality reduction to get more useful throughput out of your service – with same resources.3. Build circuit breakers and throttlers before your downstream dependencies so that you don't overload them and cause them to fail or you don't fail totally when they do indeed fail.4. If you do get overloaded for some reason (say your server pool suddenly became half its size), are you able to recover quickly.5. Do you have monitoring and alerts for these load scenarios?If you do this for each of the services in the service call graph, then your distributed service application would be robust.

评论 #25647790 未加载

评论 #25647741 未加载

anw超过 4 年前

Load Testing has been both a blessing and bane in my career.Most managers only want to see how traffic will perform in what-if scenarios (can we handle Black Friday, what will happen if our traffic goes up 10x during a special event, etc). For these, JMeter and whatever service you're using for analytics/metrics work perfectly fine (New Relic, is the current favorite). Then we can compare average latency, Error Rates, etc. with what we get during JMeter.The issue is that most managers or other PHB want more than just a one off for load testing. And this article greatly covers that.A lot of time people want a Swiss Army Knife of tools for "automation", where load testing falls under that category for them.This magical tool should be able to:* Test your API calls from a functional, integration, and a performance (not load testing; just making sure we're under specific latency)* Test your Web-sites from the same testing perspectives, as well as being modular (ie: Selenium's POM)* Integrate with CI so specific tests, test suites/flows can be tested for every new build, and we can run specific workflows by clicking a button* Be used during load testing so we can measure latency and run tests while recreating customer experience from the Web side of thingsFor each of these steps, there is at least one good tool or library to handle things. However, I've found it difficult to find something that handles all of these well and where (most importantly) the other devs are willing to pick it up.Selenium/Web Driver handles the UI tests. Postman and a handful of others can perform the API tests. JMeter is the de facto for load testing at every place I've been to. And you can write any of these into your CI.Is there something that can do all of this well, and has a low enough overhead where devs on your team are willing enough to actually use it as more than just a toy or experiment?

评论 #25653519 未加载

tkahnoski超过 4 年前

In an era before cloud, one of the major project risks was under-provisioning hardware. Load testing made sense then. It makes less sense now.I've given up on load testing all together. Most applications or services quickly grow to complex to maintain in a cost-effective way with the pace of change demanded of them.Instead every team I talk to about load testing or performance I shift it to observability. If the team can't understand current performance and load in production, any sort of load testing results in another environment will be poorly understood and hold little value.This approach positions the team much better to react to regressions in production vs holding up work trying to create or pass a load test.The exception I make for this is load testing for validating technology choices as an effort in risk mitigation that the technology can't perform. (i.e. Will this query work moving from SQL to ElasticSearch? What happens if I have 100x amount of data in that table?) Targetted specific scenarios, to confirm behavior of things too expensive to do in production.I'm sure there are a few performance critical apps that need these tools, but the vast majority of software doesn't. Don't burn 100s of hours like I did to validate performance before release. Start with gaining a deep understanding of your production behavior and push for production experimentation. It is significantly less time-consuming and pays way more dividends.

评论 #25654532 未加载

saberdancer超过 4 年前

You can create user scenarios in JMeter. It is manual work to setup but it works fine. You can add think times, randomization, scaling of number of users, use scripts to generate input data, etc. It's not a simple tool, but it's not a simple problem to solve either.I never needed WebSocket testing in JMeter but my quick Google shows that it exists.

评论 #25644422 未加载

评论 #25644948 未加载

tmikaeld超过 4 年前

I've tested a few of these, even setup multiple VPSs to do load testing at distributed scale, but it's expensive and hard to orchestrate and tests take a lot of time to setup. Plus, we don't test often enough for it to make sense...Now we just use cloud platforms for load testing, there's quite a few and the one we use the most is even free [0][0] <a href="https://loader.io/" rel="nofollow">https://loader.io/</a>

rlonn超过 4 年前

Good article. It brings up the biggest challenges with load testing. I think the "Maybe don't load test everything?" idea is the solution.We have to realise that there is no silver bullet. You can spend 101% of your hours on load testing only, and still not achieve a realistic load test scenario. The trick is to determine how much effort to spend creating (and maintaining) your tests, and/or what parts of your app to focus on. Limit the scope. The 80/20 rule applies here, like in many other situations; you get 80% of the positive results (i.e. insights into the performance of your app) by spending 20% of the effort. Simple, unrealistic load tests are a million times better than no load tests at all.

londons_explore超过 4 年前

I would like to see load testing done in some kind of 100% deterministic environment. Make a system emulator where everything is fully deterministic, including perfectly timed IO and network characteristics.Then your load test won't have any random noise from things like an SSD with inconsistent performance, or a CPU that throttles down depending on room temperature.A zero-noise load test can be much shorter to get good results, and means you won't be investigating spurious performance regressions. You can compare traces between tested software versions down to clock-cycle accuracy to really locate issues, rather than just guestimating.

评论 #25650091 未加载

评论 #25647821 未加载

safog超过 4 年前

I'm actually curious about why it makes sense to load test an entire web UI (which is what this post seems to be suggesting).In my experience, it was enough to load test individual back end APIs to figure out load profile for each API and use that to scale your h/w and serving HTML is rarely a bottleneck. This is complicated if you have a big monolith with a diverse traffic pattern - so knowing your auth API can support up to 1k QPS on your existing h/w is not super helpful for example. There are ways to work around this, but kind of complicated to enumerate all the scenarios in a comment.This still needs some sort of state building, but this should be a lot easier than setting up state using a UI / clicking buttons.

评论 #25647131 未加载

评论 #25647239 未加载

评论 #25647929 未加载

rdhatt超过 4 年前

At my workplace we're using a modified version of ShadowReader - <a href="https://github.com/edmunds/shadowreader;" rel="nofollow">https://github.com/edmunds/shadowreader;</a> note it is AWS-specific. It works by re-running the load balancer logs, either in near-real time or a replay window. We use it test changes to our Solr search engine. It complements our Locust testing which is more about red-lining our servers and isn't as user-specific.

评论 #25655568 未加载

Someone超过 4 年前

“You have a few options for getting a great picture of how your system performs:• Good old analysis.”I’ve been trying to do that (because it seemed the only realistic method for figuring out how to cost optimize a cloud setup (should we have more smaller API servers or a few big ones? One or two databases? (When) should we scale down instances in less busy times? ...), but didn’t get far.Being mathematically schooled, I went for queueing theory. That isn’t simple, but not the main stumbling block. The complexity of cloud infrastructure is. Figuring out how many simultaneous requests AWS load balancers/Kubernetes can handle, what delays they introduce, how they route calls (round-robin/random/...) how large their queues are, how Kubernetes scales down/up instances is quite a challenge.So, has anybody collected ballpark numbers for these kinds of things, or hints on what can be ignored in the setup? How do people handle cost optimization of cloud instances?Or do people just bring up variations on the infrastructure, benchmark them, and, when getting bored/running out of time/having found something that isn’t too expensive, stick to the best configuration found?

评论 #25648187 未加载

alzoid超过 4 年前

I don't think load testing is hard, it just requires time to do properly. Maybe some people think they can get a tool, record a script and hit play. In some cases that might be enough. Once you are concerned about performance testing, you have to do work to get it right.I usually have a test from the 'front' of the app and separate tests developed for each component (services, dbs etc). I do load and stress testing. Stress testing is very valuable as a preventative measure. Depending on what is available I use different hardware monitoring tools to gather data. Sometimes its OS specific or a vendor tool like PRTG. On the software side I run profilers for the language the app is using.I run focused tests - what the 'expected' use will be and general tests that hammer as much as possible.I have found and addressed problems at all levels. Investigation is the hard part. I find it fun and you learn a lot about infrastructure configuration. Most hardware problems that show up are can usually be remedied by throwing more resources at it. Often times the cost is lower than diving into full investigation / software changes. Server configuration is another inexpensive option. It requires more investigation, if the bottleneck is the number of connections you have to determine if its: one of the many webserver settings or one of the many lower level services. I once discovered a cap on TCP connections on a low level OS service was limiting performance. After those you hit software. Modern profilers will catch most issues. I find with code network latency is the biggest offender. I had an app that took 1.5 seconds round trip. The client wanted it 500ms or less. The fix was simply being smarter with database requests and having a local cache of some data. In the end a cold request took 300ms with subsequent requests being approx 70 - 100ms.I could go on but performance testing is a discipline like any other. It's not hard if you put in the time.

评论 #25647267 未加载

INTPenis超过 4 年前

This is one of those areas where I will almost always opt for DIY.And in regards to simulating users, that's not always necessary if you know what type of load users generate you can simply simulate the load that users would generate.

评论 #25644501 未加载

bizzleDawg超过 4 年前

A significant problem with traffic replay in a few cases I've considered in the past is HTTP POSTs and usage of headers. For most, logging that extra info is not realistic and rules out traffic replay as a possibility for simulating realistic load.Quite often, everyone one the team will already know which endpoint(s) are slow or unscalable. In that case, it is possible to set up a much less realistic load test in order to generate some flame charts of the problematic area. It can be more pragmatic than having to maintain what is essentially a new integration test suite as the author mentions.

评论 #25643653 未加载

jefftk超过 4 年前

I've found this to be a significant problem in testing a side project I've been working on for making music together over the Internet (<a href="https://echo.jefftk.com" rel="nofollow">https://echo.jefftk.com</a>). I can load test the backend by simulating requests reasonably easily, but the hard part is all the cases where my simplistic testing is unrealistic.

ascales超过 4 年前

Repeating what a few others here have said, but k6 has been a really easy tool to use, with great open source tools, and a managed service with pretty competitive pricing. Creating "real world traffic" is a little more tricky, but you can easily spin up a load test for just about anything in 15min.

m463超过 4 年前

I remember years and years ago I had a specific application to test, and wrote a small test harness for it....and as the application got better I ran out of tcp port numbers. (because instead of requests from multiple ip addresses, it got requests from the test harness machine)The rabbit hole goes deep.

7357超过 4 年前

Just thought I could add <a href="https://github.com/processone/tsung" rel="nofollow">https://github.com/processone/tsung</a> to the discussion.

gavanm超过 4 年前

I work in the corporate performance test / performance tuning space in a brownfield environment with a lot of co-dependent systems supported by a lot of different teams. My background prior to this was a programmer / DBA.It's all hard - but it's not any harder than building and testing end to end solution working in the equivalent environment - there's a lot of similarities, benefits and also disadvantages. Overall performance testing provides"Performance Testing" often covers quite a few different areas:- User Experience (more difficult with SPA and Mobile applications)- Server Response- Capacity Planning / Utilization- Performance Tuning- Enterprise / Systems Architecture- Production Support for Performance Problems- Monitoring (both for testing purposes and planning monitoring for production)Like programming, most the time actual coding / scripting isn't the challenge. Some of the bigger challenges I deal with are:- Test Data Setup, Availability, Size, Distribution, Consistency, Security, Backup/Restore over multiple systems- Understanding / Confirming Scenarios- Complex Application Authentication (more secure authentication methods can be harder to test or work-around)- Data integrity / security (difficult to use SaaS solutions for regulatory reasons)- Environments, Sizes, Accuracy (Prod Like), Support (Application and Infrastructure), Monitoring, Limitations, Architecture- Load Planning/Estimation/Reporting (both in Performance and Production)- Deployment Planning / Management- Investigating Performance Problems- Working with lots of different development teams.A lot of the recent trends in DevOps / Agile / IaaS / PaaS are helping improve the situation with significant improvements to deployments and scaling.Good commercial tools and OSS tools are available. The move to web / HTTP based systems has actually made testing much simpler because there's really only one protocol to target these days (HTTP) - older applications platforms using non-HTTP based protocols often make things more difficult. There are still some smaller niche areas for industries such as call centres where there are physical requirements (such as generating call traffic), and Desktop Virtulization / Remote Access (due to COVID) that don't fit the more common testing patterns. Mobile App performance testing (which is closer to User Experience testing) is also relatively immature compared to SPA and HTTP testing. There are options out there though.I also view there being (at least) two different traditional paths into performance testing - Functional Testing (Non Technical) and Developers/Tech Admin (Technical) who bring different strengths and weaknesses to large performance testing efforts.I'm a fairly big fan of trying to push most performance testing down into individual application teams as they know the technical details of the applications and often have better technical skills than a lot of performance testers. However, you still need cross-system / whole environment perf testing where you have complex interconnected systems and dependencies.It's difficult trying to balance everything between purely scientific / methodological and pragmatic (just prove it's capable of meeting the volumes) approaches.

mlthoughts2018超过 4 年前

I’ve always found Locust to be incredibly simple for simulating any user behavior. Very well designed task and user behavior APIs and patterns in it.

评论 #25645168 未加载