_____ _____ ____ _____ _____ _____ __ __ _____ _____ _____ _____ | | | \ | | | | _ | | | | | __| | | __| __| | --| | | | | | | --| | | |__| |__| __| | | | | | __| |_____|_|_|_|____/ |_____|__|__|__|__|_____|_____|_____|_|___|_____|_____| ~ about ~
  1. Building cmdchallenge using Lambda and API Gateway in the AWS free-tier with Docker and Go

    Mon 24 April 2017

    Have you ever thought about building a side-project for fun without spending a lot on hosting? This post might be for you. With the most tech-buzz-wordy title I could conjure up here is a quick overview of how cmdchallenge.com is built. The site is a simple web application side-project that executes shell commands remotely in a docker container in AWS. The front-end gives the feeling of a normal terminal but underneath it is sending whatever commands you give it remotely on an EC2 instance inside a Docker container.

    The source code for most of it is located on github including a tiny command executer written in Go, the challenge definitions, and a test harness.

    The following AWS services are used for the site:

    • Cloudfront
    • API Gateway
    • S3 bucket
    • Lambda function
    • DynamoDB
    • t2.micro EC2 Instance running coreos
    • CloudWatch logs

    In addition to this Amazon Certificate Manager and Route53 was used but for everything above you can keep costs close to zero in AWS. There is no free tier for Route53 (sad panda) but it's like 50 cents a month for a single zone.

    Features:

    • Submit commands, execute them in a bash sub-shell.
    • Check the output of the command for different challenges.
    • Run tests for challenges that need them in addition or in place of checking output.

    Deployment tools (simple and boring):

    • Makefiles.
    • Python fabric for running commands and copying files over ssh.
    • Kappa, zips up code, sends it to lambda, also manages Lambda permissions.

    With these tools the following automated steps are taken to deploy the site:

    • Create a Docker image that holds the challenges.
    • Launch a new coreos EC2 instance.
    • Run a fabric script that does the following on the instance over SSH:
      • Configures TLS so that a Lambda function can communicate to Docker on an EC2 instance.
      • Executes some periodic commands to ensure that the host cleans up old containers.
      • Downloads the docker image that has the challenges.
      • Copies the read-only volume that is used on the container for the tests and command runner.
    • Update Lambda with new code.
    • Sync the static assets to S3.
    • Invalidates CF cache for the main site.

    Architecture diagram:

    There are two public entry points for the site, one is the main web-site which is static and served S3. The other is the API gateway at api.cmdchallenge.com which is also fronted by CloudFront so that it can use a certificate from ACM and cache requests.

      api.cmdchallenge.com         cmdchallenge.com
      ********************         ****************
    +---------------------+    +---------------------+ 
    |      Cloudfront     |    |      Cloudfront     |   
    +---------------------+    +---------------------+  
               |                          | 
    +---------------------+         +-----------+
    |    API Gateway      |         | s3 bucket |
    +---------------------+         +-----------+
               |
      +-----------------+
      |                 |
      | Lambda Function |    +----------+
      |                 |--- |          |
      +-----------------+   \| DynamoDB |
               |             |          |
       +--------------+      +----------+
       | EC2 t2.micro |
       |   (coreos)   |
       +--------------+
    

    One nice thing about using AWS server-less components was that a single t2.micro instance ended up being fine for handling all of the load, even at peak.
    See section on caching/performance below.

    Here is what happens when a command is submitted in the cmdchallenge.com:

    • Javascript code sends an HTTP GET to https://api.cmdchallenge.com
    • If it is cached it returns a response immediately. If not, it forwards the request to the API gateway which in turn sends it to a Lambda function.
    • The Lambda function looks up the challenge and the command in DynamoDb and if it already has an answer it returns that. If the challenge doesn't exist in DyamoDB it is forwarded to the EC2 instance as a command using the docker API.
    • The command that the user provides is passed to a Go command runner that executes the command in a bash sub-shell inside a docker container, checks the output and runs the tests.
    • Results are returned to the Lambda function, it writes them to DynamoDb and returns the response.

    The challenges are expressed in a single YAML, here an example of one challenge:

      - slug: hello_world
        version: 4
        author: cmdchallenge
        description: |
          Print "hello world".
          Hint: There are many ways to print text on
          the command line, one way is with the 'echo'
          command.
    
          Try it below and good luck!
        example: echo 'hello world'
        expected_output:
          lines:
            - 'hello world'
    

    Interested in coming up with your own? You can submit your own challenge with a pull request. Your challenge will be added to the user-contributed section of the site.

    Caching:

    You may notice that when you do echo hello world on the hello world challenge it returns almost immediately. As it is shown above there are two layers of cache, one at CloudFront and one at DynamoDb to reduce the number of command executions on the Docker container. API Gateway can provide caching but it costs money, I worked around this by sticking CloudFront in front of it but this is only possible with HTTP GETs. With Cloudfront in front the cache-control header in the response from Lambda is set to a very long cache lifetime with every request. The version of the challenge as well as a global cache buster param is passed in so we never have to worry about returning a response from a stale challenge.

    Performance:

    If you are wondering how well this would scale for a lot of traffic, the Lambda function currently dispatches commands to a random host in a statically configured list of EC2 instances making it pretty easy to add more capacity. So far it seems to be operating fine with a single t2.micro EC2 instance handling all command requests that are not cached.

    • Time to get a echo hello world response from a cached cloudfront command - ~50ms
    • Time to get a echo hello world response from a cached command in dynamoDB - ~2.5s
    • Time to get a echo hello world response, executed in a container - ~4s

    Without caching this wouldn't be possible and also the caching at CloudFront enables most commands to return fairly quickly.

    Wrapping up

    If you like the site please follow @thecmdchallenge on twitter or if you have suggestions drop me a mail at info@cmdchallenge.com.

    Thanks!

  2. User Submitted Solutions

    Sat 04 March 2017

    Adding to the interesting 191 ways to echo hello world I've now added the ability to see user-submitted solutions to cmdchallenge.

    There are some gems if you dig through them including maybe the longest regex I've ever seen for pulling an IP address out of a file:

    (25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)
    

    Also scrolling down the page of solutions to the corrupted text problem is glorious.

    The solutions are not updated regularly right now but would be easy enough to do in the future if people want to see more, let me know on twitter and also Update: Solutions are now generated every five minutes. Feel free to submit suggestions for new challenges on github.

  3. figlet breakout

    Fri 24 February 2017

    I was looking for a cool ending for cmdchallenge and decided to dust off a 2 year old javascript project which created a breakout game from figlet fonts. Not quite a full re-write but fixed a lot of bugs and did away completely with coffee-script. More info on the github page.

    Or you can click here to play.

  4. 191 ways to echo hello world on the command line

    Wed 08 February 2017

    It's been about 12 days since the launch of cmdchallenge, a weekend project to create some common command-line tasks that can be done in a single line of bash. One common request has been to share user-submitted solutions. Or to put it another way, you may be wondering what do random people on the internet and hackernews type if you give them some basic command-line tasks and a shell prompt? Well lucky for me this is no longer a mystery!

    Starting off with Challenge #1:

    CMD Challenge #1: print "hello world" at the bash prompt

    There has been a lot of diverse input for such a simple task. I really love how people do weird stuff even when it is totally unnecessary.

    Here are some of my favorites:

    ( for i in h e l l o \  w o r l d ; do echo "$i" |awk -F, ' {print $NR}'; done ) |tr -d \\n; echo
    echo ifmmp xpsme |tr bcdefghijklmnopqrstuvwxyza abcdefghijklmnopqrstuvwxyz
    touch helloworld && echo "hello world" > helloworld && cat helloworld
    sed 's.\.\...;yhHh\hh;ywWw\ww;2,$d' README
    

    A few things to note that may clarify some of the more interesting submissions:

    • There is a README in each challenge directory, in this case it contains the string "hello world" so some people took advantage of that.
    • The directory itself was named hello world.

    Here are all of the correct submissions for the first challenge as of yesterday:

    a=(d e h l o r w X \");s=(2 1 3 3 4 7 6 4 5 3 0);for i in ${s[@]} ; do echo -n ${a[$i]}|tr X\n ' ' ; done ; echo ""
    ( for i in h e l l o \  w o r l d ; do echo "$i" |awk -F, ' {print $NR}'; done ) |tr -d \\n; echo
    for i in h e l l o; do echo -n $i; done; echo -n " "; for j in w o r l d; do echo -n $j; done
    cat README | head -n 4 | tail -n 1 | awk '{ print $3 " " $4 }' | sed -e 's/\"//g;s/\.//g'
    echo ifmmp xpsme |tr bcdefghijklmnopqrstuvwxyza abcdefghijklmnopqrstuvwxyz
    < README grep hello | tr -d '#' | cut -f3- -d' ' | tr -d '"' | tr -d '.'
    touch helloworld && echo "hello world" > helloworld && cat helloworld
    letters=(h e l l o \  w o r l d); printf '%s' "${letters[@]}"
    touch test.txt ; echo "hello world" > test.txt ; cat test.txt
    grep "hello world" README | sed -E 's/.*(hello world).*/\1/'
    cat README | grep Print | sed -E 's/^.+"([^"]+)".+$/\1/'
    echo "hello world" > ./hiworld.txt && cat ./hiworld.txt
    head -n 1 < README  | sed 's/# //' | tr '[A-Z]' '[a-z]'
    cat $0|cut -d\; -f4;echo ";hello world;" > /dev/null
    touch file && echo "hello world" > file && cat file
    cat README | sed -e 's/.*/hello world/' | head -1
    touch "hello world"; ls *hell*; rm "hello world"
    head -1 README | cut -c3- | tr '[A-Z]' '[a-z]'
    echo "hello world" > hello.txt | cat hello.txt
    awk 'BEGIN {print "hello world"}' < /dev/null
    echo "hello world" > hello.txt; cat hello.txt
    echo "hello world" > testfile; cat testfile
    echo "hello world" > test.txt; cat test.txt
    grep hello README | awk -F\" '{ print $2 }'
    for n in 'hello world' ; do echo ${n}; done
    sed 's.\.\...;yhHh\hh;ywWw\ww;2,$d' README
    echo "hello world" > foo.txt; cat foo.txt
    echo -n "h""e""l""l""o"" ""w""o""r""l""d"
    touch 'hello world' && ls h*| xargs echo
    msg="hello world"; printf '%s\n' "$msg"
    cat README| grep hello | cut -d '"' -f2
    echo " " | awk '{print "hello world"}'
    echo "hello world">hello | tail hello
    awk ' BEGIN { print "hello world" } '
    export HAHA="hello world";echo $HAHA
    echo 'hello world' > henk | cat henk
    echo "hello world" > test & cat test
    awk 'BEGIN { print "hello world"; }'
    echo "$(echo hello)" "$(echo world)"
    perl -e 'printf "%s", "hello world"'
    mkdir "hello world" && ls | tail -1
    awk 'BEGIN { print "hello world" }'
    pwd | cut -d'/' -f4 | sed 's/_/ /g'
    touch "hello world"; ls | grep ello
    printf '%s%s%s%s\n' hel lo\  wor ld
    cat README | grep -o "hello world"
    echo "hello world" | sed 's/b/h/g'
    echo | awk '{print "hello world"}'
    echo "hello world" > tmp; cat tmp
    echo hello world > henk; cat henk
    cat >/dev/stdout <<<"hello world"
    awk 'BEGIN{print "hello world";}'
    awk 'BEGIN{print("hello world")}'
    cat >/dev/stderr <<<"hello world"
    awk 'BEGIN {print "hello world"}'
    echo "hello world">algo;cat algo
    hello="hello world"; echo $hello
    perl -e 'print "hello world\n";'
    GGG="hello world" ; cat <<< $GGG
    echo|awk '{print "hello world"}'
    echo 'hello world' > /dev/stdout
    awk 'BEGIN{print "hello world"}'
    ls .. | grep hel | sed 's/_/ /g'
    printf "%s %s\n" "hello" "world"
    perl -e 'print "hello world\n"'
    echo s | sed 's/s/hello world/'
    echo -n "hello"; echo " world";
    #!/bin/bash\necho "hello world"
    printf -- '%s\n' 'hello world'
    printf '%s %s' 'hello' 'world'
    printf "%s %s" "hello" "world"
    perl -le 'print "hello world"'
    text='hello world'; echo $text
    perl -e 'print "hello world";'
    perl -e "print 'hello world';"
    echo "hello world" > a ; cat a
    echo -n hello ; echo " world"
    perl -e 'print "hello world"'
    perl -e "print 'hello world'"
    STR="hello world"; echo $STR
    perl -E 'say "hello world";'
    grep hello <<< "hello world"
    perl -e'print "hello world"'
    printf "%s\\n" "hello world"
    perl -E 'say q[hello world]'
    touch "hello world"; ls *\ *
    printf '%s %s\n' hello world
    printf "hello world" | cat -
    VAR="hello world";echo $VAR
    cat < <(echo "hello world")
    perl -E 'say "hello world"'
    date|sed s/.*/hello\ world/
    touch 'hello world'; ls he*
    echo  "hello world" | cat -
    printf '%s\n' 'hello world'
    perl -e'print"hello world"'
    printf '%s\n' "hello world"
    printf "%s\n" "hello world"
    printf "hello %s\n" "world"
    printf "%b\n" "hello world"
    echo|sed -n "i hello world"
    touch 'hello world'; ls h*
    echo "hello world" | cat -
    printf %s\\n "hello world"
    printf 'dlrow olleh' | rev
    printf '%s %s' hello world
    true && echo "hello world"
    printf "%s %s" hello world
    i='hello world'; echo $i;
    printf '%s' "hello world"
    cat <<EOF\nhello world\nEOF
    printf -- 'hello world\n'
    cat <(echo "hello world")
    echo `echo "hello world"`
    printf 'hello %s' "world"
    printf "hello %s" "world"
    printf "%s" "hello world"
    printf '%s' 'hello world'
    printf -- "hello world\n"
    echo "dlrow olleh" | rev
    echo 'hello world' | cat
    echo ${PWD##*/}|tr _ ' '
    printf "hello world\013"
    printf '%s ' hello world
    echo "hello world" | cat
    /bin/echo 'hello world'
    /bin/echo "hello world"
    printf %s 'hello world'
    cat - <<< 'hello world'
    echo -e "hello world\n"
    a='hello world';echo $a
    printf -- "hello world"
    printf "hello world" \n
    printf %s "hello world"
    cat - <<< "hello world"
    cat - <<<"hello world"
    printf "hello world\n"
    echo -ne "hello world"
    echo 'hello world'|cat
    printf 'hello world\n'
    echo hello world | cat
    echo """hello world"""
    echo -n  "hello world"
    cat <<< "hello world"
    /bin/echo hello world
    echo -e "hello world"
    printf "hello world";
    echo -n 'hello world'
    echo -e 'hello world'
    echo -n "hello world"
    cat <<< 'hello world'
    echo dlrow olleh|rev
    (echo "hello world")
    cat <<<"hello world"
    echo "hello world" ;
    'echo' 'hello world'
    echo   "hello world"
    printf "hello world"
    echo ''hello world''
    printf 'hello world'
    cat <<<'hello world'
    echo ""hello world""
    echo -n hello world
    echo  "hello world"
    printf hello\ world
    echo 'hello world'\
    echo $"hello world"
    echo 'hello world';
    echo 'hello world '
    echo "hello world "
    echo $'hello world'
    echo  'hello world'
    echo -e hello world
    echo "hello world";
    echo "hello world"\
    echo hello world""
    echo ""hello world
    echo "hello world"
    echo hello\ world;
    echo 'hello world'
    echo hello   world
    'echo' hello world
    echo hello world \
    echo hello' 'world
    echo hello world\
    echo hello world;
    echo hello \world
    echo hello  world
    echo hello\ world
    echo  hello world
    echo hello world
    


    Of course there are a lot of quoting and space variations here as well. I will make a data-dump available soon with the responses for all challenges.

    On the building and hosting side of things this is getting more than the usual tiny trickle of side-project internet traffic. So far I have tried to contain the entire thing in an AWS free tier account, it has worked out OK so far with a few hiccups here and there. Since several people have asked, I will share more details about how the site is put together in a future post, follow cmdchallenge on twitter or use rss.

  5. cmdchallenge - now with badges

    Tue 07 February 2017

    New badges

    Just returned from an extended weekend of traveling and now finally getting to open questions and issues on the cmdchallenge site. Sorry for the slow responses.

    With a few hours to hack on the front-end I did a quick face-lift and also created some fun badges. Local browser storage is used to store state so unfortunately you will lose your progress from one computer to the next. If people want it I could add logins but it seems like it might be more trouble than it is worth. New challenges are coming in via pull requests (thanks!) and I think I will move the user-contributed ones into their own category soon. If you have more ideas please let me know via twitter or mail.

    Tel Aviv

    Now the back to work and back to cold weather...

Page 1 / 1