Enlisting the help of friends and family can only get you so far, so how do you gather quality feedback pre/post-launch, without shelling out for a professional agency to do the testing?
Solo dev for an app with hundreds of thousands of users. These are the things I do that ensure smooth releases:<p>• run a beta test group. These users consist mostly of users that have emailed in with feature suggestions or bugs. I ask them if they’d like to join the beta (e.g. they can check their bug is fixed). I frequently get great feedback from a few of these users.<p>• fiverr.com. Shell out $100 spread across a range of testers, and give them all slightly different briefings. Some feedback is terrible but there’s some gold there that’s really worth far more than $100.<p>• learn how to test your own product. It’s hard to do well, but you can learn. It’s a valuable skill that can really set you apart in a regular engineering job too.<p>• used phased releases. Pause the release at 1%, give it a few days for any minor issues to roll in and fix these before releasing to 100%.
It doesn't matter how big you are, the same cheep method works for informal but fairly accurate quick and dirty testing.<p>Beer.<p>You think I'm kidding but if you go to a bar, there are lots of people who will test whatever you put in front of them for a free drink. If you want to keep your costs down even further then go during happy hour.<p>Is that "user testing" no... not in the sense we think of it today. Is it testing of the usability of your product: yes it is. Im guessing that most apps issues that are more likely the latter and not the former. Basic things that you think are clear that really aren't (your too close to the problem and the solution). If you explain, point and guide through the same aspect of your feature 10 times in 10 tests people might not get it! These are the problems you might want to solve, dumb it down, change it up, make it better.<p>Does this work? Yes, as long as you can moderate. If you can you be neutral and friendly and avoid "traps" that will alter your results. Saying things like "MY app" or reacting personally will alter your results, and learning when to "guide them" through a frustrating point and when to let them puzzle through it is a bit of an art (this too offers clues if your guiding people at the same place). For me personally a drink or two helps ME in the process...<p>I have also discovered that this process works very well with pairs of people. Two friends already have a rapport and will happily talk to each other to try to get through something problematic. One can encourage or discourage the other and their dialog can be enlightening. Letting them play together with your app and drink free drinks can give you a LOT more insight than the next three individuals testing. Anything more than two people becomes problematic so look to avoid large groups.
1. Stick the mailing list sign-up form on a landing page and do a bit of promoting in relevant places. Specifically say it's for beta announcements, preview release testing and picking everyone's brain. This should net you some users to bootstrap the process with, and they <i>will</i> be responsive.<p>2. Release the beta, mark it as such and ask people for feedback after they used the product for a bit. Listen to the feedback, act on what's relevant, keep improving the beta. Give kudos where they are due.<p>Have public forum, actively participate in all threads. Many people look at how active the support forum is before trying the goods or leaving any feedback. So the more active it is, the better. Also, moderate it aggressively to trim junk and trivialities.<p>3. Once out of the beta, don't forget to <i>genereously</i> reward everyone who chipped in. Ideally with a special gift, like a specially tagged lifetime premium account, a handful of most expensive licenses, etc. The worst thing to do is to offer a discount on a production version - this comes across as a cheap and greedy move.
In the past (when I was still a solo developer), after being 90% confident the code won't completely break the environment or do something screwy with user data, sticking the code in production and funneling 50% of the requests to it while watching NewRelic.<p>There are no better user testers than your users, and the risk is only moderate. I could always pull the server down and restore the previous version while trying to fix whatever I broke in the deployment.<p>But if you aren't doing SaaS, user testing as a solo developer is not really feasible. The best user testing I can think of that doesn't "cost" is a Beta/RC Program for your users to volunteer to get potentially broke software for a discount or just early feature releases. For this to work though, you have to have enough users.
I have an app in the windows store with a few thousand users. When anyone emails me with feedback or an issue, I usually ask if they want to try the beta version (after I've resolved their issue of course). I've gotten about 15-20 part time testers this way, they've been a mixed bag. Some provide pages if detailed feedback, some barely anything.<p>When I have a decent, bug free mvp of the next version I'm going to fork out for usertesting.com
Post-launch:<p>What's your goal? Do you want to test whether your update works correctly on many devices? Do you want to get ideas for new features? How many users do you have?<p>I have about 10,000 active users. I don't do much testing. I test myself on an Android device and an iPhone. Since my app is built with Ionic and based on Cordova, it works anyways and real crashes are not to expected.<p>Feedback is easy: I just wait until people write me an email. It's funny: Sometimes, you don't hear about a feature request or some issue for months. Then 2-3 people write you within a day. Then nothing again.<p>Back when I started, I implemented a lot of features. Now I basically reject almost every feature request, unless it comes up extremely often in emails. I sometimes take notes, but most of the time, I type a quick reply, thanking for the suggestion, archive the email and only try to get a feeling for what people want.<p>Obviously, this has its limits, but since I know my core users and what they want, it works for me.<p>If there's a serious bug, people will let you know soon enough.
I wouldn't ever pay for an agency to do user testing, and it has nothing to do with money.<p>Your ideal tester is someone who cares enough about your product to use it.<p>I also wouldn't even start developing the product unless I had a decent sized list of people who really want it.<p>In which case, testing it is easy. Just ask those people to test drive your app during development. If your app solves a real problem they will happily do it for free since it solves their problem, but you could also give them perks like giving them 2 months of your service for free when it launches.<p>Your users will naturally give you feedback if things are broken, so that handles post-launch real world testing. Of course you'll want to supplement that with heaps of automated tests too.
I've used UserTesting.com for previous tests. If your product affects a lowest-common-denominator user or you are testing mainly UX/interaction, I'd recommend it. At that time I paid around $20 per usability test (I bought a packet of multiple sessions). Testers recorded using my product for 5 to 10 minutes and it was great watching them stumble over what I had designed. They were also trained to be over-communicative and think aloud, which really helped. In total I did four or five of them before I had my first release (since it was still somewhat expensive, I tried to ration my tests by fixing all previous UX bugs before issuing another one). It's been a long time, but if I recall correctly, a few times the testers weren't tech-competent enough to actually do the test, but when I asked UserTesting for a credit or refund to re-do the test because the tester was "bad", they provided one without hassle.<p>You need to be make it as easy as possible to run your user test, because a tester will give up on roadblocks that take more than 60 seconds to solve.<p>I was developing a language learning product, and so needed testers who were interested in language X. In the end, the people I found using the filter [Interested in X language, native English] weren't actually interested in language X. Rather, they came across as people who had accepted the filter to make an extra buck.<p>However, it was good enough for a random internet trial. If I wanted better results I had to do it in person (which I also recommend).
If I need to get feedback on a feature, I build a prototype, and then show it to one or two people (friends, family, customers that sent feature requests). I tell them roughly what they should do, and watch if they can figure it out. Then I change things as necessary, and show it to the next person. Usually a handful of people are enough to find the the major issues.<p>- don't "burn" beta testers. show a prototype to one or two people, and don't show it to more people before you fixed the issues that have come up. otherwise you just get duplicate feedback<p>- ask people personally. Emails to a group of people are ignored by 99% of recipients<p>- feel free to ignore advise from testers. everyone always wants to give advise, but that is irrelevant. the important part is: did the tester manage to complete the task?<p>- do not ignore people struggling with your app. the natural reaction to people failing to use your app is to think they are stupid. they aren't. most of the time your ui is stupid and you need to fix it. it doesn't matter how brilliant you think the ui is, if testers can't use it, change it.<p>- sometimes the answer is that what you built doesn't work and you just wasted a lot of time and you need to throw it away and start over. do not ignore this possibility
For web apps I’ve always used Mechanical Turk and paid each user $10 vs. just a dollar or less. I made each tester download a screen record plug in for chrome, hit record and then go through all the steps in the test. Once finished they emailed me the recording.<p>The above is what you get from usertesting.com and other sites but pay up the nose($50 a test) vs. $10 or less dollars per test.<p>If it’s a downloadable app Just spin up a web UI/UX and have the turkers follow same steps but have them change the browser to say iPhone 6/7/8 view.
The most important way to do this is to actually be a user of your product, e.g. "Eating your own dog food".<p>This doesn't mean logging in, trying all the features and playing with it for an hour - you have to <i>really</i> use it with your data on a daily basis over a long period of time. You'll come across a lot of bugs and start logging feature requests yourself.<p>Once <i>you</i> can use it without issues [and you'll need to be careful to split the wishlist features you come up with, against the actual bugs which need to be fixed], then you can get others to try it.<p>Best way to get user testing from others?<p>- email your users and ask them if they have had any issues, or want any new features.<p>- ask people to give your product a quick test - for example, <a href="https://www.lifepim.com" rel="nofollow">https://www.lifepim.com</a> is my new webapp to easily manage your personal information [shamless plug/warning - notes and tasks work well, contacts needs work, and calendar is not pretty]<p>- pay particular attention to requests from users who pay for your product. Everyone wants software to work in a certain way, but when someone <i>pays</i>, it can be a good indicator to listen to what they want. If they say something isn't working, make it a priority to investigate and address this.
I usually hang out in the kitchen at our shared office space and ask people. If that's not enough, then I'll start visiting friends' offices for team breakfast or post-work beers and do the same.
There are user-testing companies that does this for small fee:<p><pre><code> * http://www.trymyui.com/
* https://www.usertesting.com/
* https://Whatusersdo.com/
* https://Userlytics.com/
</code></pre>
Disclaimer: I've also worked on couple of those companies as a tester when I needed money and I was out of job, but thankfully I'm currently back working as developer again. On some of them, the tester also needed to record themselves using webcam.
I'd say it depends on the sort of application you're developing.<p>In the next couple of weeks I'm due to release a boxing game (shameless plug: leatherthegame.com), and other than myself my only tester has been my cousin.<p>Having worked on the project for over 3 years I'm aware that there are areas where I "can't see the wood for the trees", but I am confident in my own knowledge, and only wanted an external viewpoint to see how a noob would see things from afresh (both in terms of not having used the app before and also him being only a lite-casual boxing fan, so he wasn't up-to-speed with some terminology).<p>I know full well that once its released the proper boxing fans will get in touch, and I'll change things post-release from that feedback, so my strategy is very much a post-launch end user feedback one. In an ideal world I would have had more friends who could help me test it, but there's only so many opinions I can factor in pre-launch, and managing their experiences and feedback would have been a mini-project to manage purely by itself. When already time-constrained with launch schedules, app store listings and last minute bug hunts and device optimisation I couldn't spare the time to manage a phalanx of testers.<p>This has been a passion project/scratch-my-own-itch project, which is why I feel more secure not having many outside opinions. If you're building something more strategic, to fill a target niche in a specific market say, then my approach will DEFINITELY NOT WORK. In this case you'd be best off firing out a message on LinkedIn or something like that, asking for would-be best testers.
I used to be a solo dev working in a niche field where I made plug-ins for some creative apps. I found that one thing that really helped was being active in the forums where users were, and (if allowed by the forum) posting that you have a product and you'd like to get some people to test it. I'd then require them to fill out a simple form that gathered some information about how they work.<p>The form served 2 purposes:<p>1) It gave me the ability to balance different types of users. Hobbyists vs. professionals. People in this specialty vs. that specialty. etc.<p>2) It weeded out people who just wanted something free and weren't interested in giving feedback. Seriously, a simple form requiring a name, an email, and the answer to 3-5 questions helped get rid of the freeloaders. But don't make it too onerous or you won't end up with any testers.<p>I'd then usually give a free license for the released product to anyone who provided useful feedback.
Using your product daily is essential not only to help steer the ship on what to build but also to cue you in on what's buggy / inefficient. I have the good fortune of having a live chat interface (product is: <a href="https://www.jqbx.fm" rel="nofollow">https://www.jqbx.fm</a>) so I get pretty quick feedback if something that's been pushed is causing anyone any issues. However to keep bugs to a minimum I keep most of the logic client side and have a dev environment that I can run on production- I usually test a version for a couple days (see: eat your own dogfood) before releasing anything. Also forcing yourself to review your own code is a good skill and has helped catch some issues right before pulling the trigger.
I generally find sellers on fiverr. They are willing to record screen and comment for about 30 mins for $5. Not a bad deal and mostly good insightful feedback.
One thing I have found to be very useful is to ask my existing userbase of other projects if they want to test something new and give me feedback.<p>In fact this worked so well that I am currently building a new project around this. A site that lets startups gather feedback from people who are interested in trying out new sites and giving feedback.<p>It is currently in private beta with about 20 startups. If you want an invite, feel free to send me an email.
Try to make a simple tutorial video of some sort. Even if it isn't useful in a practical sense it forces yourself (or someone else) to walk through the UI stating their intentions and trying to execute them. Some of the best feedback I've gotten was through watching videos like this for an application that I work on as you can see where the stumbling points are throughout the process.
I push the code live and see what the numbers say.<p>I know this isn't the answer most people want to hear, but no one tells the truth like live users.
I do a free beta, then various paid betas. The goal is also to find a price point. A user can either leave feedback in my betas and get that month free, or they can pay. The idea that all they have to do is give feedback for a free service usually works well...<p>Eventually, when it's easier to pay, some users will start leaving money instead if feedback. At some point I'll raise prices a bit and stop the feedback for service system.<p>I've done this for:<p><a href="https://projectpiglet.com/" rel="nofollow">https://projectpiglet.com/</a> and <a href="https://easy-a.net/" rel="nofollow">https://easy-a.net/</a><p>And a few others, it's always fairly successful - <i>assuming you have a decent landing page</i> and you can advertise the site in your target market for cheap / free.
I didn't read all the comments, so don't know if anyone else suggested this.<p>A few times now, I have successfully got my first few hundred users by offering a lifetime discount and positioning it as a limited access beta testing program.<p>Got lots of feedback, and most users kept their accounts for a very long time, even if they weren't using the software, just do they wouldn't lose their discount.
One tool I use is Mouseflow, primarily for session playback. It's super helpful to see how users are interacting with the app, qualitatively, and has helped me find bugs and UX issues that never made it into the error logs.<p><a href="https://mouseflow.com" rel="nofollow">https://mouseflow.com</a>
Automated testing.<p>I start out with manual tests.<p>If I have a bug at any part of the code I add a unit test.<p>When it gets too big and starts taking too long then I start writing e2e tests.
Probably it was already said, but quite crucial in the process would be proper analytics. To see where the user's flow is concentrated or stops at all.
external agency, 1-2 weekly tests, specially critical in the first stages of the product.<p>good price, cheap compared to a dev’s salary, or compared to my time,
paying less than $500 per month.<p>do 5-second tests, usability, card sorting, path testing, all ux and ui testing for greater chance of success