It is hiring time again. The recruitment agents start instinctively calling, smelling the scent of a commission, and pretty soon the resume's start flowing in. The local I.T. market is pretty restricted right now. Initially the client wanted to hire two permanents, (Expectedly to phase me, the final contractor, out) but has had to settle on 1 perm and 1 new contractor because we have a load of work coming down the pipe. Since we're looking to hire people that are expected to hit the ground running in the project these roles were for highly experienced developers that would be familiar with working with things like TDD, IOC/DI, and be able to write loosely coupled, "clean" code. One of my tasks was to put together a coding quiz of sorts to give to successful candidates. Nothing really difficult, but something to give them a taste of what we're looking for and something to give us an idea of how they understand requirements and complete a task. The quiz consisted of a set of requirements with some general instructions on what the sample project already contained, and what they would need to complete. The project they were given consisted of a small number of interfaces for dependencies that would provide much of the additional functionality that they would utilize to meet the requirements. (I.e. data retrieval, sending e-mails etc.) The quiz was expected to take about 2-3 hours to complete including unit tests. (Tested with one of our junior developers.)
Part of the interesting element of this quiz was identified when our developer gave it a test run. There were a few holes in the quiz. For instance a requirement would reference an expiry date on an existing domain object which did not exist. The test implies that elements that their specific requirements were being provided by other team members, so issues like this were left in to see how candidates responded to such problems. (Ignored, noted with ToDos, or brought up when they submitted their response.)
The reason I chose to create a template project with the quiz was to give candidates hints into the way we develop code, and the kinds of things we are looking for. Do they understand how to do dependency injection for their service class based on the interfaces we've provided, or do they use the IOC Container like a global Factory/Registry? Do they try to follow our naming convention and style? Even with the requirements I dropped hints about TDD/BDD and mocking.
The results of the quiz were even better than we expected. Out of a total of 8 candidates, (2 perm applicants and 6 contractors) all with impressive looking resumes, the quiz made it very clear who knew their shiz from those that just continued to write poor quality, unmaintainable code with modern tools and libraries. The quiz probably worked a bit too well because of those 8 candidates, only one really stood out, and he was the one with the least experience. However, he was the only one that understood the concept of dependency injection, and while he had no experience directly with mocking frameworks, he picked up on it from the requirements, researched one, and tried applying it (reasonably well) within a sample unit test. His main failing was the amount of work for the time he spent, but the fact was he hit all of the requirements with clear markers where details still needed to be filled in.
What was really surprising was how bad some of the contractor submissions were. Only one actually met all of the requirements, but while his resume toted .Net 3.5 & 4.0, the code sample he wrote was effectively .Net 2.0. He had provided unit tests, but they were merely auto-generated and code-coverage results for his service was only 70% with much of it single-hit results. Most of the other contractors (these are guys in the same region as me with > 14 years of experience) either missed basic behaviour requirements, or had fairly severe logic errors present in their samples. One big one I looked for was a requirement that each operation in the service had an authorization requirement. Most of the submissions went and did the authorization check in the construction of the service rather than on each call. A risky assumption in any situation with a service, and in cases where this service was Singleton scoped in an IOC container (as something like this would commonly be) this would easily lead to embarrassing bugs. One candidate's project didn't even compile, while another attempted to write unit tests, which all failed when run.
So in the end we'll probably settle for 1 decent intermediate that hopefully has the enthusiasm to step up to the plate, and one contractor who delivered a working sample, but we hope was writing rather dated code out of habbit. (He's done a lot of work for government and their pace of adopting new technology I am well aware of.) The moral of the story is when looking to hire people, don't put too much credit in their resume, or whether they can recite namespaces for commonly used libraries. Build up a code sample project in the style you like with a set of reasonable requirements, and get them to write some code. Seperate the shiny nuggets from the vast slurry that call themselves "Senior Software Developers".