LinkedIn thinks I know PHP

A criticism of LinkedIn Skill Assessments

Tuesday, October 22, 2019 permalink
tags: programming - hiring

Around a month ago, LinkedIn released Skill Assessments, which are basically quizzes on various programming languages and other tools.

According to the announcement:

69% of professionals think their skills are more important than college education when job-seeking, and more than 76% wish there was a way for hiring managers to verify their skills so they could stand out amongst other candidates.

and from a separate post on LinkedIn’s “business” blog:

the speed and scale of these tests encourage recruiters to assess more candidates — including more nontraditional candidates, further diversifying the talent pool.

I wasn’t a business major, but it makes sense. LinkedIn’s main product is its member base; adding quizzes can improve engagement, and maybe more importantly improve quality of recruiter searches. It also ties into LinkedIn Learning, with links to courses upon completion of an assessment.

From the user perspective, there are very low barriers to entry. There is no risk to failure; LinkedIn only posts results on success. Assessments take upwards of 15 minutes, although in practice 10 minutes is more than enough. The only barrier to entry is knowledge of the subject matter… well, more about that shortly. To pass, you need to finish in the top 30% of quiz takers.

I took a bunch of quizzes, at first related to languages I know. The first three I took were Python, C, C++. I don’t remember a lot of details from those quizzes, but they didn’t go too deep. If you know what pointers are you will probably pass C. If you know what const references are you will probably pass C++.

After that the next I took was Java. I did use Java professionally at one point, so I expected I stood a good chance at passing. I ended up failing, and of course I can’t say exactly why I failed (since I don’t know) but I suspect the reason was not knowing about certain Java 8+ language features. That’s fine, time moves on, I stopped using it when Java 7 was modern.

But on a lark I started taking quizzes for things I didn’t know, and I found I actually did pretty well.

The questions aren’t great

I don’t know PHP. I know it is a C-like syntax, it is widely used in the web… and that’s it. I’ve never used it. I’ve seen it, but if you showed me PHP code and asked “is this PHP?” I would have no clue.

So I was pretty surprised when I passed the PHP assessment. I wasn’t shocked, as I kind of thought I was getting questions right, but why should someone who doesn’t know a language be able to get things right?

Well, a lot of the questions weren’t really that specific to the language in question. A programmer should be able to read a for loop in most languages:

for val in vals:
    val.process()

for(auto &val: vals){
    val.process()
}

Only Python and C++ experts can understand the above

Obviously you can find incomprehensible snippets in any language (shout out to the International Obfuscated C Code Contest), but for the most part for loops and if statements look very similar across languages, especially the ones that make it into LinkedIn Skill Assessments.

By remembering snippets from previous questions, as well as studying and carefully eliminating answers, I was able to learn as I went. In other words, I could game the format. I’m good at test taking, but, like I’ve already said, I don’t know PHP.

Test-taking skills are really useful for passing tests, but they’re really useless for doing practical things in professional programming settings. I have to imagine people who are good at taking multiple choice tests are not the untapped non-traditional audience LinkedIn is trying to connect.

I ended up passing a total of five courses for subjects I don’t know. Besides PHP, I also passed AWS, C#, Ruby on Rails, and MongoDB.

One wrinkle from the AWS quiz; some of the questions asked about capabilities of AWS, and I just guessed the one that was the best. I.e. if a question asked “Does AWS support X” and the answers were “no,” “yes, but it’s hard”, and “yes, and it’s easy” I just went with “yes, and it’s easy.” Maybe I was right, maybe not. I don’t know how many questions I got right, but this technique seemed to work.

I largely passed C# because I know it’s vaguely similar to Java and at one point in my life, like I mentioned before, I knew Java fairly well. In case you are skimming: no, I did not pass the Java Assessment. Read into that what you will.

How did I pass Rails and MongoDB? It was mainly the same technique of carefully reading questions, comparing answers, and making educated guesses… again, the kind of thing unlikely to correlate to success on the job but likely to correlate to privilege and college prep tutors.

The MongoDB assessment had a question about what language Mongo uses for its query language; granted, this was only one question, but you’d think that someone who used Mongo would 100% know what language they were writing queries in.

I actually was able to figure out the correct answer, but only after submitting the incorrect answer. I answered Python, but I kind of realized seconds after locking that answer in that it’s probably JavaScript. The quiz itself had several questions earlier that included queries, and while Python and Javascript snippets can look similar, Javascript object key literals don’t have to be quoted, while Python dictionary key literals do. Basically the following is valid Javascript:

{foo: 'bar'}

while the equivalent in Python must include quotes in the key:

{'foo': 'bar'}

I say this to make two points:

a) if you can learn everything you need to answer one question directly from another question, maybe that’s not a great test of knowledge of the material, and

b) if someone doesn’t even know what language Mongo queries are written in, how can that person possibly know Mongo at all? It seems that getting some basic questions wrong should be a large waving red flag that the person doesn’t know the subject matter whatsoever.

The passing standard doesn’t make sense

I mentioned earlier that the top 30% of quiz takers pass. Maybe the actual passing standard is more complicated, but given that anyone can take a quiz, grading on a curve like this doesn’t make sense.

Maybe you didn’t like taking classes on a curve back in school, but in this case it’s possible a large portion of people being graded didn’t even take the class. That’s the equivalent of people who don’t know a language taking the quiz; their scores count towards the curve.

Even if the questions were great, which I don’t think they are, the grading standard is completely unobjective and not based on how competent you are in the material, but far more correlated to how many people who have taken the Skill Assessment are competent.

For example:

If 90k people who don’t know Python at all and 10k people who do know some Python take a quiz, AND the quiz is perfect, there will be 20k people who don’t know Python who are in the top 30%.

Conversely, if 50k people don’t know Python and 50k do, then 20k people who do know Python will fail.

Ideally there would be an objective standard unrelated to how many random people of random skill have taken the quiz before you.

It’s also worth mentioning that who takes the quiz after you doesn’t matter, which is also weird.

In conclusion,

On my LinkedIn profile today, I have eleven passing quizzes (if you’re keeping count, I also passed JavaScript and JSON, both of which I sort of know).

Of those eleven, I am completely unequivocally incompetent in five of them.

I don’t mean to make such a big deal about a couple of bad quizzes. After all, in my life I’ve probably spent more time on quizzes to tell which Spice Girl I am than LinkedIn Skill Assessments.

However, Spice Girl personality quizzes don’t pretend that they’ll help you get a job; in fact, I’ve received feedback that I should omit that I’m Baby Spice from my resume.

From a brief survey of search results, it seems like the public opinion is still waiting and seeing but overall warm on Skill Assessments. I’m confident that passing Skill Assessments in the short term will help job searches, since LinkedIn wants to drive usership to this feature and they control their recruiter search results.

But I’m concerned about a tool whose results so poorly map to reality actually playing a large role in recruiting, which I fear may be inevitable if that’s what LinkedIn wants.

I got my current job via a recruiter on LinkedIn (shout out to Rob Case). It’s an extremely powerful platform with millions of users. As all platforms do, it has baked-in biases and flaws. It’s a great goal to reach out to non-tradional candidates, but even if they worked well, and none of the issues I’ve mentioned are valid, I’m not sure that multiple choice quizzes are the right mechanism to do so.