One of the nice features of coding challenge sites like hackerrank is that when you submit an algorithm it will also be checked against random data. This validates the performance of the algorithm and will test it against different inputs. CMD Challenge is a web application that executes shell commands remotely in a Docker container. When it was initially launched, all the checker did was validate the output of the execution against a list of strings, expressed in YAML.
For example, the "hello world" challenge:
- slug: hello_world description: | Print "hello world". Hint: There are many ways to print text on the command line, one way is with the 'echo' command. expected_output: lines: - 'hello world'
When the user submits "
echo hello world" the command is executed remotely and
the response is validated against the string "
hello world". This worked fine for commands
that only needed to be verified by checking their output, but what about commands that
modify files on disk? For these types of challenges there are a
a set of shell scripts
that also run inside the container and will return non-zero if any of pre-determined tests fail.
This is how the "delete files",
"remove files with extension"
challenges and other similar ones work to ensure the bash one-line command you entered
This solves most of the validation but what about this challenge?
- slug: sum_all_numbers description: | The file sum-me.txt has a list of numbers, one per line. Print the sum of these numbers. expected_output: lines: - "42"
One way you can "trick" the site is to simply enter the command "
echo 42" which
is the correct sum of the numbers in "
In fact, a lot of random internet people did exactly that
as you can see from a small subset of submissions that were previously
marked as "correct":
echo 42 echo '42' echo "42" echo "4""2" expr 41 + 1 echo $(( 29 + 13)) etc. etc.
So the next natural thing to do is to add a way to randomize the data that the command is operating on. Just like scripts added to run tests, now there are bash scripts for every challenge that allow you to manipulate files before running the command again. Only if the output is still correct (and tests pass, if they are present) will then the challenge be marked as correct.
For example, for the "sum all numbers" challenge the script looks like this:
#!/bin/bash echo $(( $RANDOM % 1000 )) >> sum-me.txt paste -sd+ sum-me.txt | bc exit 0
Now if you type the command "
echo 42" the randomizer check will fail since will also verify that
the command can run against the modified "
Currently, there are a randomizer scripts added for a subset of challenges.
If you want to contribute and add more please feel free to send a pull request.
If you would like to
read some more about how the site is built in an AWS free tier account see the earlier post on
building cmdchallenge. If you liked this write up let me know
by following @thecmdchallenge on twitter or drop
me mail at firstname.lastname@example.org.