Very good idea, but I don't program that way. I use a competent IDE to tell me syntax errors and obvious programming mistakes while I write. If I can't debug my algorithm with trial & error I write a test first and modify my algorithm until it complies with the test.
To think that a programmer will always come up with the exact and most optimal solution to any given problem is a bit naïve. In my 10 years as a programmer I have never seen one wonder-kid that is able to do that. The real wonder kid assets in a team also tend to be those that are more creative than syntax-anal compared to those that can write syntax-error-free-code with pen and pencil and remember large parts of a phonebook...
This is quite nice, even if the exercise isn't all that difficult.<p>It would be nice if the boundary conditions were better specified (people seem to have been bitten by overflowing integers). This could be inferred if you told people that you're using a 32-bit machine (the sum of elements is at most SIZE_MAX * INT_MAX; if int and size_t are 32-bit, SIZE_MAX * INT_MAX < INT64_MAX.)<p>I was also somewhat surprised to see that my code compiles without #include <stdint.h>. I'm not sure this is correct (it could be solved by putting the code written by the candidate <i>before</i> the test code).<p>Finally, I assume you have proper sandboxing set up?
I think this is cute but I would think hard about the kinds of questions you develop for this. Coming from the perspective of someone working for a big company with a ton of software people that is not a tech company, I could use something like this in general. But, the question you asked is the kind of thing I ask CS undergrads (almost exactly, in fact) because it allows for clever solutions without actually requiring someone to be a particularly good programmer. I can work through it with them and if the basic solution is easy, prompt them to give me something more clever, so I know how they think, which is good when you're looking at an inexperienced developer. However, for an experienced developer that I want to hire to churn out Java code, the difference to me between a correct but simple O(n^2) solution vs a clever O(n) is not terribly meaningful for determining whether they can actually do what I want them to do. I want something that requires people to write more than a handful of lines and demonstrates an ability to keep track of a few more variables correctly, and honestly I don't care all that much about their algorithmic complexity (although I might care a great deal about their space usage, usage of external resources, etc).
So, if this is the audience you are looking to attract, consider thinking about this as a screening tool for people outside of the startup world that actually need "knows how to really do shit in language X" more than "knows how to write clever algorithm,"
This is interesting. It might be worth noting somewhere how many bits is an Int on your test rigs.<p>I got 94 using C in 12min (didn't check for overflows of large numbers, silly haskell making me used to Integers).
That is an amazing piece of software. I hope you make a mint.<p>I'd give people access to their language's output facility for playing around to the solution, but that is a minor nitpick. (And I suppose I could open an irb instance on my machine for exploration and then give you code when I'm good and ready.)<p>My Ruby solution, in case we've got anybody who is struggling: <a href="http://pastie.org/771438" rel="nofollow">http://pastie.org/771438</a> This would be my first cut if you asked me to whiteboard this.
I like how the performance evaluation seems to be fairly realistic - I used Ruby and my algorithm could have been more efficient (O(<i>n</i>) space, and O(<i>n</i>) time but with more traversals than necessary), but I scored 100 because I met all the <i>stated</i> constraints. (<a href="http://codility.com/demo/results/?id=demo8XKE77-H8G" rel="nofollow">http://codility.com/demo/results/?id=demo8XKE77-H8G</a> if anyone wants to refer to and/or mock my code.) It seems to me "ability to solve problems <i>well enough</i> under time pressure" is exactly what you want a simple hands-off test like this to screen for.<p>In face-to-face interviews you'd probably follow up by adding constraints: "your algorithm uses O(<i>n</i>) space in addition to the input, can you do it in constant space?". It would be cool if Codility supported that kind of workflow - following up the submitted solution with refinements on the original question. (Relatedly - do you support the workflow where you get a second try if you didn't get 100, maybe because the evalution revealed a constraint that wasn't clearly stated in the problem outline?)
Is the code editor they're using something publicly available? I'm looking around for a light in-browser code editor with programmer-oriented features (e.g. syntax highlighting, this also seems to auto-indent Python code to some degree).
It gave me a test concerning finding the "equilibrium index" of a sequence. It said to assume that the sequence is very long. (The question is repeated here: <a href="http://forums.sun.com/thread.jspa?threadID=676596&start=0" rel="nofollow">http://forums.sun.com/thread.jspa?threadID=676596&start=...</a>) This problem seems underspecified, because it doesn't state whether you can iterate repeatedly over the sequence in question. If you can, it's easy. If you can't I'm not sure how to do it.
Really nice site. Well done. The example problem that showed up to me (equilibrium point detection) was not trivial, but didn't require advanced stuff which a halfway decent programmer wouldn't use day in and day out.<p>I also like the way you tell me what you scored on and what my result was.<p>I wonder if you should relax the condition that you can submit only once.<p>I gather TopCoder offers something similar. What's your strategy to compete with them?<p>PS: got 94 using C++ in 16 min b/c I didn't check overflow of large numbers. Good thing you check for that.
You know, now that I think about it...The description of the test was pretty incoherent. If I don't know what my test data is, and if the problem description is incoherent, I can't really make it work.<p>It's also taking a metric fuckton of time to evaluate my solution. Which apparently fails. For reasons I can't debug because I can't see the test data.<p>Any company that relies on code snippets, incomplete problem statements and undefined test data to judge coders will get what they deserve.
5 minute version of the demo problem in Ruby - 100%:<p><a href="http://codility.com/demo/results/?id=demoMPC2W9-JDG" rel="nofollow">http://codility.com/demo/results/?id=demoMPC2W9-JDG</a>
As a student this is very interesting since it could also be used to measure self progress. There's a myriad of ways to test your abilities in most subjects, but if I want to find out if I'm any good at programming or to measure my progress, there are very few easy ways to do that. Are you going to sell this to just business's?
For C++ it would be nice to know what compiler is being checked against. At least in the demo question I would use a 64 bit int to hold the sum if the arrays are really "large" but MSVC names the type __int64 vs G++ using long long and other such minor details. Same for the other languages aka python 2.x or 3.x etc.
Got my 94% C solution up to 100% by using long long ints for intermediate sums.<p><a href="http://pastebin.com/m6e742f56" rel="nofollow">http://pastebin.com/m6e742f56</a><p>Edit: And won't this be a problem if interviewees can access the internet? The interviewers will have to make sure they can check the browsing history of the computer being used!
Codility people, if you're reading, here's a bug report.<p>The test window's layout is fubar if you use a small window
size. I was using 800x600 and the language selection buttons were hidden behind the source code text box. Something (couldn't tell what) overlapped the first two lines of the instructions.<p>Firefox 3.5.7/Ubuntu Linux 4.10, FWIW.
I'm not sure the timer is a good idea. For one, I found the seconds counting down <i>very</i> distracting. It also encourages quick and inefficient solutions that are easy to hack out, over more efficient algorithms that may take longer to code.<p>Maybe you should time the programs, instead of the programmers.
Might need some way of making explicit which libraries are available?<p>eg. if the programmer goes off and writes a multithreaded solution, they might only realise too late that pthread isn't included. Admittedly they should have checked first using verify, but you never know with interview pressure.
Seems like a cool idea. If they were smart they'd not only target companies looking for people but people looking for jobs. e.g. Adding a "Allow the codility community to take this test" check box. Then you have the a whole world full of people taking your test.
Very nice product. Much-needed quantification in tech recruiting. Maybe consider marketing it to the stackoverflow people. They have been running a job board with much fanfare since recently. (94 points for me in 7 mins, yay)
Just a heads up:<p>I took a demo quiz and it worked fine. Great idea for a product.<p>However I think I'm too competent for the test because I was able to run something I probably shouldn't have been able too.
also thought this was nice after trying it out.<p>one thing i missed, though, was the ability to write and run my own tests. perhaps a "scratchpad" that evaluates code and returns a result would be a nice addition?