Posts on Kevin Lui

Programming Language Classifier

Mon, 18 Nov 2019 16:22:11 -0700

In this post, I’ll explain some of the details that went into http://langclass.kevinlui.org/. A fun thing I did was feature importance recovery after feature hashing. I’ll explain that at the end.

Data source

The data was retrieve from the Rosetta Code Project using this git repo: https://github.com/acmeism/RosettaCodeData. The features are code snippets and the labels are programming languages.

Tokenization

We consider 2 different tokenizers.

character-level tokenization.

print('Hello World') ==> ['p', 'r', 'i', 'n' 't', '(', ...]

split on non-alphanumerical.

print('Hello World') ==> ['print', '(', 'Hello', 'World', ')']

Vectorization

After tokenizing, we consider consider 1-grams followed by feature hashing modulo 4098. We also do the same with 2-grams.

Classifier

We use lightgbm with default parameters. There’s a lot of tuning to be done but that wasn’t my main interest so I’m putting it off.

Feature Importance

A fun thing I did was feature importance extraction despite feature hashing. After training a tree-based model on feature-hashed inputs, you can normally only determine the feature importance of the hash values. For example, you might discover that the hash 123 has high importance. But this does not tell you which one of our original tokens has high importance. You can determine which tokens hashes to 123 but it could be the case that many tokens share the same hash value. In the next section, we share an idea for recovering the feature importance of the original tokens.

The top features for character-level 2-grams are:

[';\n', ')\n', ' (', ', ', ' =', '= ', 't(', 't ', 'in', '# ']

The top features for 1-grams where we split on non-alphanumericals are:

[';', '=', '.', ':', 'print', ',', '"', '#', ')', '-']

The most importance feature in both cases contains a semi-colon. This is unsurprising since some programming languages seldomly use semi-colons (for example, Python) while some programming languages use semi-colons extensively (for example, C).

Mathematical Heuristic

This technique is mainly inspired by bloom-filters. For ease of exposition, we’ll focus on the character-level 2-grams case. Let $X$ be the space of character-level 2-grams, $Y$ be the hashed space, and $H_1, H_2, H_3:X \to Y$ be 3 different hash functions. In our implementation, we take $H_i$ to be the murmur3 hash with seed $i$.

For $x\in X$, let $f_x$ be the feature importance of $x$ if we were to train our model using unhashed features. For $y\in Y$ and $t\in \{1,2,3\}$, let $g_y ^t$ be the feature importance of $y$ using the hash function $H_t$. For our heuristic, we assume that $g_y ^t$ is approximately $\sum_{x\in H_t ^{-1}(y)} f_x$.

Our model is trained on the hashed space so we have access to $g_y ^t$. Our goal is to understand $f_x$. We can of course approximate the solution to a massive linear system so try to recover $f_x$ but what we’re really interested is a ranking of $f_x$. So do this, we’ll use the geometric-mean of the hashed values. In particular, let $F_x = (\prod_{t=1} ^3 g_{H_t(x)} ^t)^{1 /3}$. Then we can show that if $f_x > f_y$, then $\mathbb{E}(F_x)>\mathbb{E}(F_y)$.

Alerting long commands in zsh

Sat, 31 Aug 2019 19:35:54 -0700

Update (2019-10-03): I fleshed this out into a plugin! Check it out https://github.com/kevinywlui/zlong_alert.zsh!

I often find myself running commands that take a long time and then coming back to it later. For example, when I’m running a long computation for my research or when I’m building a package. I recently discovered this zbell gist which seems to be almost exactly what I want! I’m trying to learn more shell scripting so I’m going to reimplement this and explain what I’m doing. The final version will be this

# Ignore commands if they start with the follow
zlong_ignore_cmds='vim ssh'

# Define what a long duration is
zlong_duration=60

# Need to set an initial timestamps otherwise, we'll be comparing an empty
# string with an integer.
zlong_timestamp=$EPOCHSECONDS

# Define the alerting function, do the text processing here
zlong_alert_func() {
    local cmd=$1
    local secs=$2
    local ftime=$(printf '%dh:%dm:%ds\n' $(($secs / 3600)) $(($secs % 3600 / 60)) $(($secs % 60)))
    notify-send "Done: $1" "Time: $ftime"
    echo "\a"
}

zlong_alert_pre() {
    zlong_timestamp=$EPOCHSECONDS
    zlong_last_cmd=$1
}

zlong_alert_post() {
    local duration=$(($EPOCHSECONDS - $zlong_timestamp))
    local lasted_long=$(($duration - $zlong_duration))
    local cmd_head=$(echo $zlong_last_cmd | cut -d ' ' -f 1)
    if [[ $lasted_long -gt 0 && ! -z $zlong_last_cmd && ! *"$cmd_head"*==zlong_ignore_cmds ]]; then
        zlong_alert_func $zlong_last_cmd duration
    fi
    zlong_last_cmd=''
}

add-zsh-hook preexec zlong_alert_pre
add-zsh-hook precmd zlong_alert_post

Overview

The goal of this script is to alert me whenever a command, that has taken over 60 seconds, is complete. I will be use notify-send to alert me and also send a bell character.

For my particular desktop setup, notify-send will alert me via dunst and the bell will set my kitty terminal to urgent which i3wm will pick up on and turn the workspace indicator red.

So my alert function will send me the command that was ran, the duration formatted nicely, and a bell character.

zlong_alert_func() {
    local cmd=$1
    local secs=$2
    local ftime=$(printf '%dh:%dm:%ds\n' $(($secs / 3600)) $(($secs % 3600 / 60)) $(($secs % 60)))
    notify-send "Done: $1" "Time: $ftime"
    echo "\a"
}

Timing commands

The zsh/datetime module gives us the EPOCHSECONDS parameter which is the number of seconds since the unix epoch. To determine the duration, we just need to subtract EPOCHSECONDS when the command is complete from EPOCHSECONDS when the command started. We will use zsh hooks for this.

Zsh hooks

One of the advantages of zsh over bash is that you get hooks. For this script, we’re interested in

preexec which ‘’executed just after a command has been read and is about to be executed.’'
precmd which ‘’executed before each prompt.’'

zlong_alert_pre() {
    zlong_timestamp=$EPOCHSECONDS
    zlong_last_cmd=$1
}

zlong_alert_post() {
    local duration=$(( $EPOCHSECONDS - $zlong_timestamp ))
    local lasted_long=$(( $duration - $zlong_duration ))
    if [[ $lasted_long -gt 0 ]]; then
        zlong_alert_func $zlong_last_cmd duration
    fi
}

add-zsh-hook preexec zlong_alert_pre
add-zsh-hook precmd zlong_alert_post

Ignore some commands

I use vim and ssh often and it doesn’t really make sense to count them as a long command. I store the commands I want to ignore in a comma separated list at the top, I can extract the head of a command as follows:

cmd_head=\$(echo $zlong_last_cmd | cut -d ' ' -f 1)

and substring testing is done using the star wildcard

*"$cmd_head"*==zlong_ignore_cmds

Fixing a zbell bug

In the zbell gist, there was a super annoying bug! If you run ls, wait long enough, then press ctrl-c, it’ll alert you! This is because preexec did not get a chance to run again but precmd did so it’ll count the duration as the time since the ls command. This was fix by setting zlong_last_cmd to be the empty string at the end of preexec then checking to see if the string is non-empty before sending it to the alert function.

Alternatives

One alternative to alerting the end of long commands that I should mention is using Pushbullet. Pushbullet is an app that I have installed on my phone that can send me notifications. The best part about Pushbullet is that it has an API. I’ve use https://github.com/Red5d/pushbullet-bash to send me an alert of my phone whenever a long computation I was doing for my research has finished.

Another cool zsh feature is REPORTTIME. This won’t alert you but it’ll tell you how long a command has run for. For example, by setting REPORTTIME=3, zsh will tell me the run duration of any command that takes more than 3 second.

Automatic Vim PlugUpdate

Sun, 04 Aug 2019 21:58:21 -0700

This post was inspired by this Reddit thread.

I use vim-plug to handle my vim plugins and I want to keep my plugins up-to-date. Ideally, this would be done automatically but I don’t want to suddenly have a broken vim setup. Luckily, vim-plug comes with a snapshot feature.

I save the following as ~/bin/vimplug_update.

#! /usr/bin/env bash
vim_cmd="nvim"
BACKUP_DIR="$HOME/.vimplug_snapshots"
BACKUP_FILE="$BACKUP_DIR/snapshot_$(date +"%Y-%m-%d-%H%M%S").vim"

mkdir -p "$BACKUP_DIR"

$vim_cmd -c "PlugSnapshot $BACKUP_FILE" +qa
$vim_cmd +PlugUpdate +qa

Then I set a daily cron job by running crontab -e and writing @daily ~/bin/vimplug_update. If I ever have to restore my settings, I run

vim -S snapshot_2019-08-05-085852.vim

Developing for Sagemath

Thu, 01 Aug 2019 16:22:11 -0700

For my first post, I’ll explain how to begin developing for Sagemath. This post is essentially a gist that I shared with members of the UW Sage Seminar. This is a short version of the official guide.

Official Sagemath guide

https://doc.sagemath.org/html/en/developer/index.html

Setting up local development copy

The first task will be to obtain a local development copy of Sagemath.

Clone Sagemath and checkout the develop branch

git clone https://github.com/sagemath/sage
cd ~/sage
git checkout develop

Build Sagemath. I had access to a 48 thread machine so I used all 48 threads. It was a shared computer so I also assigned the build process a nice level of 19. This will take between 30 minutes to 6 hours so we’ll move on to the next step while this is going.

cd ~/sage
export MAKE='make -j 48'
nice -n 19 make build

Developing via trac

Sagemath development is done through the Sagemath trac server: https://trac.sagemath.org/. We now setup our local git repository to interact with this server easily.

The new way to get a Sagemath-trac account is to use your Github account. Click on “Github Login” on the main Sagemath-trac page.
The Sagemath people made a cool tool called git-trac-command (https://github.com/sagemath/git-trac-command). This tool provides the git trac subcommand helps our local git repository interface with the Sagemath-trac server. Install this package.

cd
git clone https://github.com/sagemath/git-trac-command
cd ~/git-trac-command
python setup.py install --user

We can now authenticate using username/token. The username is list on the Sagemath-trac page. It is also gh.yourgithubname. The token is listed under account preferences.

cd ~/sage
git trac config --user USERNAME --token TOKEN

We also need to authenticate with ssh keys. So first generate the keys.

cd
ssh-keygen # follow the prompts or spam <ENTER>
cat .ssh/id_rsa.pub

The output of the previous command is your ssh public key. Copy this over to your account preferences under ssh keys.

Make a ticket!!

Hit the new ticket button.
Fill in what it asks.

Checkout the ticket

Now checkout the ticket in your local development copy.

cd ~/sage
git trac checkout <ticket number>

Make a commit

Perform some edits.
git add filename to tell git to track these edits.
git commit -m "<one line description>" to make a commit.
git trac push to send to the trac server.
When ready, modify the ticket and set to “Need Review”.