[{"slug":"ml4pe-dayhoff","layout":"presentation","title":"The Dayhoff Atlas: scaling sequence diversity for improved protein generation","author":"alex","subtitle":"Invited talk at ML for Protein Engineering (ML4PE)","description":"My invited talk at the ML4PE seminar series on the Dayhoff Atlas and synthetic data augmentation for protein language models","date":"2025-10-07","published":false,"tags":["ml","protein","paper"],"category":"Presentation","youtube":"https://www.youtube.com/watch?v=ke8IJIOtV_Y","link":"https://doi.org/10.1101/2025.07.21.665991","github":"https://github.com/microsoft/Dayhoff","huggingface":"https://huggingface.co/datasets/microsoft/Dayhoff","md":{"html":"

I was invited to give a talk at the ML for Protein Engineering (ML4PE) seminar series on our work on the Dayhoff Atlas. This was a really fun talk to give — it’s a great community and I got some good questions from the audience.

The talk covers the main ideas behind the Dayhoff Atlas, which started as my internship project at Microsoft Research New England in the BioML group (with Ava Amini and Kevin Yang). The core question was: can we use synthetic protein data to improve protein language models, and does that improvement translate to real wet-lab results?

The answer turned out to be yes. We created BackboneRef, a large-scale synthetic protein dataset (240,811 de novo designed backbones, 73M sequences) using a pipeline of RFDiffusion, ProteinMPNN, OmegaFold, and Foldseek. Training protein language models with this synthetic data augmentation led to an 87.5% increase in expression success rate and 50% increase in novelty of de novo designed proteins compared to baseline.

The paper is currently in revision at Cell Systems.

You can watch the full recording here: YouTube.

"}},{"slug":"plmtalk","layout":"presentation","title":"A survey of sequence models for proteins","author":"alex","subtitle":"pLMs and their history","description":"A talk I gave to a CS-audience on protein sequence modeling","date":"2023-10-19","published":true,"tags":["ml","paper"],"category":"Presentation","md":{"html":"

I gave this talk for a group of undergrads interning in the Abbasi lab for the semester. I had a good time reviewing some of the papers that I think are interesting in this area. To be clear, I am not an expert in protein language modeling or protein biophysics (although I spent a lot of time thinking about the physics of specific peptide drugs while at Nano Precision, for formulation reasons). A lot of the work was done by the Marks lab at Harvard—some really amazing researchers there—really the type of computational biologists I would like to be.

The presentation is probably better viewed in its standalone form at this link: here. You can also fullscreen it if you are in Chrome by pressing the “f” key while interacting with the presentation.

"}},{"slug":"representationlearning","layout":"blog","title":"What is representation learning in biology for?","author":"alex","subtitle":"Unhelpful musings on ML in biology","description":"What's the destination that computational biology is going towards?","date":"2023-10-18","published":true,"tags":["biology"],"category":"blog","md":{"html":"

What does representation learning in biology even mean?

In the last few years, there has been an explosion of interest in applications of machine learning in biology, particularly in (relatively) data-abundant subfields such as single-cell sequencing and protein/DNA sequencing data. Tools like transformers, graph neural networks, diffusion models, and other new and powerful techniques are entering the world of biology, and it’s quite exciting! I myself in various fellowship and grant applications have written in a general way about the promise of computational biology when applied to unraveling the complexities of the rough and messy biological world.

But I’ll also admit that when it comes down to it, a lot of machine learning methods (my own included) seem to only perform marginally better than simpler statistical methods in many cases. And I think it’s reasonable to say, and I also believe, that it’s because in general in computational science (in academia) what we are really doing is overfitting to benchmarks or various contrived scenarios for paper publication and not for what is “useful” to biologists at the bench and to companies trying to develop new therapeutics and so on.

When I think about what is “useful”, however, I’ve realized I don’t really know what useful means, and maybe the field doesn’t either. When I read opinion pieces or grant applications I often see a reference to “data-driven science” or “hypothesis-free science” and in general I buy the idea specifically that if we had a ton of good data we could do something interesting. We could, for instance, gather a huge heaping pile of data and do what companies like insitro and Recursion are doing and create these “maps” of biology that we can use to understand interactions, structure, and ultimately create predictive perturbation models. I’m quite optimistic about these sorts of approaches in general. They are in some sense new (we have not had the ability to do this at large scale until now), but more importantly they have the potential to move us towards more general pictures of biological variation across the genome. Right now, I think something that makes it difficult to transfer inferences across subfields in biology is that every finding is conditioned on a huge number of specific details. Was this a study in immortalized cell lines, organoids, fibroblasts, iPSC, or did they use a combination of systems? What perturbations were used and how were they implemented? What phenotypic measurement technique was used? What were the working definitions of the various outcomes of the study? In general, I would hope that moving towards these large scale surveys has the potential to at least outline what is more general and what might be more specific, and to strucuture any future investigations on what makes up the differences. Certainly I think others would agree that the objective of projects where we generate a ton of data and analyze it is to find new hypotheses to test, and I think these approaches will do nothing if not generate new hypotheses.

However when I talk to cynics and particularly people who have been in systems biology for longer, I can’t really come up with an argument that really says these big data-driven approaches are going to be useful. I think the main argument is that we could use much simpler models (statistical or “good old fashioned machine learning”) to do a lot of what we do, and insofar as the majority of what we do in computational biology is hypothesis generation, I think that’s true. Certainly there are now many examples in the literature of how deep learning models only marginally outperform simple ones like linear regression for specific tasks. I realize that’s not really fair to expect of the “representational learning” community, but I think it’s a fair criticism of the field in general.

On the other side of this is also a criticism that I see folks like Konrad Kording bring up about neuroscience. We can only test so many hypotheses, and one way to see what we’re doing in computational biology (and neuroscience) is generating a ton of hard-to-test ideas that haven’t really helped us build towards specific theories of how these systems work. To say nothing of the huge amount of flawed methodologies and false-positive-generation-rituals we have in the field, I think it’s fair to say that we have a lot of work to do to advance scientific understanding.

I don’t have any answers, and I mostly wrote this just to help myself understand what I find irritating about the state of the field and I suppose what I am really wondering is: what are we actually going to do with this data? This huge, huge amount of data? How really is it going to help us understand biology?

"}},{"slug":"vae","layout":"blog","title":"Diffusion models","author":"alex","description":"Some paper notes on diffusion models","subtitle":"something","date":"2023-06-14","published":false,"tags":["ml","paper"],"category":"Note","md":{"html":"

Understanding Diffusion Models: A Unified Perspective

Background: VAEs, Hierarchical VAEs and the ELBO

A key concept in VAEs is the concept of the ELBO. Mathematically we want to represent our concepts and data with the joint distribution $p(\\mathbf{x}, \\mathbf{z})$ . There are two routes to the marginal:

Explicit marginalization of $\\mathbf{z}$ :
$\\tag{eq-1} p(\\mathbf{x}) = \\int p(\\mathbf{x}, \\mathbf{z}) \\mathrm{d}\\mathbf{z}$
Chain rule of probability: $p(\\mathbf{x}) = \\frac{p(\\mathbf{x}, \\mathbf{z})}{p(\\mathbf{z}\\mid\\mathbf{x})}$

Computing the likelihood $p(\\mathbf{x})$ is hard, so we turn to the ELBO, or evidence lower bound as a proxy for the objective. Formally this is:

\\mathbb{E}_{q_\\phi(\\mathbf{z}\\mid\\mathbf{x})} \\left[ \\mathrm{log} \\frac{p(\\mathbf{x}, \\mathbf{z})}{q_\\phi(\\mathbf{z}\\mid\\mathbf{x})} \\right]

which can be rearranged to:

\\mathrm{log}\\ p(\\mathbf{x}) \\geq \\mathbb{E}_{q_\\phi(\\mathbf{z}\\mid\\mathbf{x})} \\left[ \\mathrm{log} \\frac{p(\\mathbf{x}, \\mathbf{z})}{q_\\phi(\\mathbf{z}\\mid\\mathbf{x})} \\right]

Here parameters of some model to approximate a variational distribution are $\\phi$ . Let’s spell out what we are doing here a bit more:

Keep in mind for this (for me), the reason we can go from the integral to the expectation is because the definition of expectation is: $\\mathbb{E} = \\int \\mathbf{x}f_\\mathbf{x}\\ \\mathrm{d}\\mathbf{x}$ , and since we have set in this case that $q_\\phi (\\mathbf{z} \\mid \\mathbf{x})$ is the PDF of $\\mathbf{z}$ , then we have satisfied this condition; see also LOTUS.

However this is more of a definition than an explanation. To better understand why this is actually a good thing to be modeling, let’s dig in further:

\\begin{align*}\n& \\mathrm{log}\\ p(\\mathbf{x}) &{} = &{}\\ \\mathrm{log} \\int p(\\mathbf{x}) \\int q_\\phi(\\mathbf{z} \\mid \\mathbf{x}) \\mathrm{d}\\mathbf{z} && &{} \\text{Multiply by}\\ 1 \\\\ \n& &{} = &{} \\int q_\\phi (\\mathbf{z} \\mid \\mathbf{x})\\ \\mathrm{log}\\ p(\\mathbf{x}) \\mathrm{d} \\mathbf{z} && &{} \\text { Bring evidence in integral} \\\\ \n& &{} = &{}\\ \\mathbb{E}_{q_\\phi (\\mathbf{z} \\mid \\mathbf{x})} \\left[\\mathrm{log}\\ p(\\mathbf{x}) \\right] && &{} \\text { Definition of expectation} \\\\\n& &{} = &{}\\ \\mathbb{E}_{q\\phi (\\mathbf{z} \\mid \\mathbf{x})} \\left[\\mathrm{log} \\frac{p(\\mathbf{x}, \\mathbf{z})}{p(\\mathbf{z} \\mid \\mathbf{x})} \\right] && &{} \\text { Chain rule of probability} \\\\\n& &{} = &{}\\ \\mathbb{E}_{q\\phi (\\mathbf{z} \\mid \\mathbf{x})} \\left[\\mathrm{log} \\frac{p(\\mathbf{x}, \\mathbf{z}) p_\\phi (\\mathbf{z} \\mid \\mathbf{x})}{p(\\mathbf{z} \\mid \\mathbf{x}) q_\\phi (\\mathbf{z} \\mid \\mathbf{x})} \\right] && &{} \\text{} \\\\\n& &{} = &{}\\ \\mathbb{E}_{q\\phi (\\mathbf{z} \\mid \\mathbf{x})} \\left[\\log \\frac{p(\\mathbf{x}, \\mathbf{z})}{q_\\phi (\\mathbf{z} \\mid \\mathbf{x})} \\right] + && &{} \\text{Split expectation using log rules} \\\\\n& &{} &{} \\quad \\mathbb{E}_{q\\phi (\\mathbf{z} \\mid \\mathbf{x})} \\left[\\log \\frac{q_\\phi(\\mathbf{z} \\mid \\mathbf{x})}{p (\\mathbf{z} \\mid \\mathbf{x})} \\right] \\\\\n& &{} = &{}\\ \\mathbb{E}_{q_\\phi (\\mathbf{z} \\mid \\mathbf{x})} \\left[\\log \\frac{p(\\mathbf{x}, \\mathbf{z})}{q_\\phi (\\mathbf{z} \\mid \\mathbf{x})} \\right] + && &{} \\text{Definition of KL divergence} \\\\\n& &{} &{}\\ \\quad\\mathcal{D}_{KL}(q_\\phi (\\mathbf{z} \\mid \\mathbf{x})\\ \\Vert\\ p(\\mathbf{z} \\mid \\mathbf{x})) \\\\\n& &{} \\geq &{}\\ \\mathbb{E}_{q\\phi (\\mathbf{z} \\mid \\mathbf{x})} \\left[\\log \\frac{p(\\mathbf{x}, \\mathbf{z})}{q_\\phi (\\mathbf{z} \\mid \\mathbf{x})} \\right] && &{} \\text{KL is always greater than 0}\n\\end{align*}

This is better than our previous formulation for several reasons:

The evidence, $\\mathrm{log}\\ p(\\mathbf{x})$ , is always equal to the ELBO $\\mathbb{E}_{q_\\phi (\\mathbf{z} \\mid \\mathbf{x})} \\left[\\log \\frac{p(\\mathbf{x}, \\mathbf{z})}{q_\\phi (\\mathbf{z} \\mid \\mathbf{x})} \\right]$ plus the KL divergence between the approximate posterior $q_\\phi (\\mathbf{z} \\mid \\mathbf{x})$ and the true posterior $p(\\mathbf{z}) \\mid \\mathbf{x}$ . In the first derivation, Jensen’s inequality removed this term.
The sort of theoretical strength of the of this framework is that we can easily see how we want to match the true posterior $p(\\mathbf{z} \\mid \\mathbf{x})$ with our approximate, $q_\\phi(\\mathbf{z} \\mid \\mathbf{x})$ by matching their KL divergence.
However, in practice we cannot access the true distribution—but by noticing that the evidence is constant with respect to $\\phi$ (because we marginalize out all latents from the joint and so it does not really depend on the model itself, or the $\\phi$ ’s), we can see that any maximization of the ELBO is a minimization of the KL divergence.

Variational autoencoders

In the default formulation, VAEs maximize the ELBO according to the approach and ideas above. This is variational because we approximate for the best $q_\\phi$ amongst a family of potential distributions parameterized by $\\phi$ . To see this, let’s split the ELBO term into components to understand them:

\\begin{aligned}\n& \\mathbb{E}_{q_\\phi (\\mathbf{z} \\mid \\mathbf{x})} \\left[\\log \\frac{p(\\mathbf{x}, \\mathbf{z})}{q_\\phi (\\mathbf{z} \\mid \\mathbf{x})} \\right] &{} = &{}\\ \\mathbb{E}_{q_\\phi (\\mathbf{z} \\mid \\mathbf{x})} \\left[\\log \\frac{p_\\theta(\\mathbf{x} \\mid \\mathbf{z}) p(\\mathbf{z})}{q_\\phi (\\mathbf{z} \\mid \\mathbf{x})} \\right] && &{} \\text{Chain rule} \\\\\n& &{} = &{}\\ \\mathbb{E}_{q_\\phi (\\mathbf{z} \\mid \\mathbf{x})} \\left[\\log p_\\theta (\\mathbf{x} \\mid \\mathbf{z}) \\right] + && &{} \\\\\n& &{} &{} \\quad \\mathbb{E}_{q_\\phi (\\mathbf{z} \\mid \\mathbf{x})} \\left[\\log \\frac{p(\\mathbf{z})}{q_\\phi (\\mathbf{z} \\mid \\mathbf{x})} \\right] && &{} \\text{Split expectation} \\\\\n& &{} = &{}\\ \\mathbb{E}_{q_\\phi (\\mathbf{z} \\mid \\mathbf{x})} \\left[\\log p_\\theta (\\mathbf{x} \\mid \\mathbf{z}) \\right] - && &{} \\\\\n& &{} &{} \\quad \\mathcal{D}_{KL}(q_\\phi (\\mathbf{z} \\mid \\mathbf{x})\\ \\Vert\\ p(\\mathbf{z})) && &{} \\text{Definition of KL} \n\\end{aligned}

Where we can see the first term is basically a reconstruction (something implemented as MSE, maybe), and the second term is a prior matching term, which encourages the encoder to learn a distribution rather than collapsing into a delta function (on 0, or perhaps on the mean of the distribution).

Usually we chose a multivariate Gaussian for the prior, and a multivariate Gaussian with diagonal covariance for the latents.

"}},{"slug":"docker-singularity","layout":"blog","title":"A short guide to Docker/Singularity","author":"alex","subtitle":"Docker/Singularity how-to","description":"A quick run-down of basics of containers","date":"2023-02-01","published":true,"tags":["software"],"category":"Software","md":{"html":"

Note: this is a guide I made for a student I was TA’ing for, and I thought it might be useful to others. I tried to make it relatively comprehensive, but it is not meant to be a complete guide and the language is meant to not be really, really technical.

What are Docker and Singularity/Apptainer?

The main idea of these softwares, which are referred to as containerization softwares, is to encapsulate a programming environment in a single file.

What this means is that we are doing something generally analogous to a virtual environment in Python—but what Docker and Singularity (or Apptainer) allow users to do is encapsulate an entire computing environment.

This includes the OS (so, Ubuntu or CentOS or even something like a custom OS) and all of the software. We can install a custom version of Python, or even several versions, and then all of a specific group of libraries.\nWe can also install more complex libraries, like CUDA and GPU compute-related libraries, as well.\nWhat’s even better about this is that we get all of this in one file that we can then pass around or send to someone else.\nBy sending someone a Docker or Singularity image—which are the individual files that encapsulate the computing environments, we can then let them\nrun an entire workflow that we’ve created, down to all of the specific libraries and versions that we used.

This is obviously highly useful for reproducible computing, but\ncan also help make your life easier by allowing you to perform the same computation on the same environment across different places.

What is the difference between Docker and Singularity/Apptainer?

The original containerization software was Docker. It was used to facilitate workflows\nin more computer-science focused and particularly enterprise workflows, for example to allow a user\nto create a cluster of similar compute environments (for example, to have all the same environment to produce some\nproduct recommendation or something else). Scientific computing users thought this might be useful,\nbut realized that there is a significant drawback of Docker—it requires root access to a given machine,\nand particularly in the high performance computing clusters that we typically operate\non in academia, might allow a specific group of users to cause undesirable changes to a clusters’ underlying environment.

Singularity (originally developed by LBNL), which later became Apptainer (although a lot of people still use\nSingularity, and it is generally interoperable with Apptainer), are the standard for academic use.

In particular, you cannot use Docker images on Wynton—but you can use Singularity. One issue is that\nyou cannot create Singularity images on Wynton (with some notable exceptions).

How do I use a Docker or Singularity image?

How to use Docker

Pulling a Docker image from DockerHub

Let’s assume for now we already have a Docker or Singularity (I’m just going to refer to them as Singularity images, but know that here we can use\nApptainer here as the term interchangably.) I will also assume you have already installed Docker, which can be done\nfairly easily on an Ubuntu machine or on OS X—see the installation guide.

Say, for example, we want to use a Docker image equipped with the CUDA libraries and PyTorch.

This is sort of a simple case that might be easy, maybe we want to use a specific CUDA version or something similar.

Fortunately, many Docker images have already been created, and are uploaded to a central repository called the DockerHub (https://hub.docker.com)\nIn fact, PyTorch itself distributes a group of relatively minimal Docker images with PyTorch installed on DockerHub: https://hub.docker.com/r/pytorch/pytorch

What we can do, then, is type in at the command line [see here to see the specific versions of the images provided]:

docker pull pytorch/pytorch:latest # the text after the colon indicates the version; \n# alternatively: \n# docker pull pytorch/pytorch:1.31.1-cuda11.6-cudnn8-runtime

NOTE: often Docker is set up to allow specific users to access it by invoking the docker command.\nFor other situations you may need to prepend sudo to the docker call—this is one of the reasons that\nHPC clusters prefer to use Singularity. For this document I will just save space by not prepending the sudo but note that you may need it in practice.

When we do this, we get an output that looks like:

$\"Output$

And we can then run a new command, docker images, which gives output shown in Figure 2 below.

$\"Output$

Keep in mind that in general if you want to keep track of which images are installed on your computer,\nyou can run this command. There are also other commands to keep track of which images are currently running, so,\nthis is sort of like keeping track of the difference between having Python 3.8 installed versus having an active Python process.

There is a key difference between running docker pull pytorch/pytorch, docker pull pytorch/pytorch:latest and something like docker pull pytorch/pytorch:1.31.1-cuda11.6-cudnn8-runtime. If you run docker images you’ll likely see\nthat if you were to run these three commands in sequence that you would get independent versions. This\nis important for usage reasons, but also practically if you just run docker run pytorch/pytorch it will assume\nyou are looking for the latest one, and in fact if it detects that there is a newer version on DockerHub\nwill pull that one down and use it instead. Since images can be large, (5-10+GB), this can be sort of inconvenient.\nIn practice it is good to fix a version and use that specific one until you need to update.

Using the image and using GPU’s

Then let’s say I actually want to use the image. I need to run something like:

docker run --rm -it pytorch/pytorch python\n# the it basically means "interactive" \n# the rm means "remove the container after I finish"\n# keep in mind you need to prepend a specific other command to allow \n# GPU access to the container

This then gives the output in Figure 3.

$\"Output$

So this is great, because we are clearly running the commands inside the container!\nIf we want to use CUDA in the image, we need to install some new software found at: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html#docker

And if we do this, and follow the command, we can see that we get the right output that CUDA is available (Figure 4).

$\"Output$

We can run a Python program by calling: docker run --rm pytorch/pytorch python [someprogram] (notice I omitted the -it because we are not interested in running anything interactively.)

And in general we can run any program installed in the container using the same syntax. So, for example, if we\nhad installed samtools, a common sequence processing program, we could run: docker run --rm samtools --version or something similar.

One last note is that we can invoke the same container both with:

docker run --rm -it pytorch/pytorch [SOMECOMMAND] # or\ndocker run --rm -it 71eb2d092138 [SOMECOMMAND] # this is the "image id" \n# found in the output of the `docker images` command

An important note is that the Docker images actually located in /var/lib/docker/images by default, but\naccessing the actual files themselves can be sort of complex—although is possible by transmitting a .tar file (ex: docker save -o pytorch.tar pytorch/pytorch and docker load --input pytorch.tar | docker load < pytorch.tar).\nIn general if you want to try to send Docker images from one computer to another you might prefer to use a Singularity image, or to transmit the\nDockerfile, or the “recipe” to build a Docker image (more on that later).

Binding directories using `docker -v`

One important thing to note about Docker is that by default it does not bind any directories on the host machine.\nBinding, here, means something roughly analogous to mounting a drive on a normal Unix filesystem. In practice\nwhat this means is that you can’t access any of the files on your local machine by default in your Docker image.\nSo, if you had a file at /work-files/datafile.txt, then running docker pytorch/pytorch cat /work-files/datafile.txt would yield a file not found error.

So, what Docker allows you to do is mount volumes using the -v flag. Note that this flag is case sensitive: -V will not work (uppercase V).

If we wanted to access the datafile.txt file, what we would do is then run something like:

$\"Output$

The syntax of the -v command is to have a source and destination mount volume path on either side of a colon.\nSo, in this example, what we are saying is to mount /work-files at the location /work-files inside the container.

We don’t have to have the destination path be the same as the source path.\nIf we had changed our previous mount to -v /work-files:/data then we would just have to adjust the cat command correspondingly, so:

docker run --rm -v /work-files:/data pytorch/pytorch cat /data/datafile.txt

Which would produce the same output. What matter is consistency in how the data is accessed. Also note that\nif you want to bind multiple paths, say /harddrive1 and /harddrive2, you would need to issue two -v commands, so something like:

docker run --rm -v /harddrive1:/data1 -v /harddrive2:/data2 ...some other stuff...

This is a super basic introduction to Docker—there is a whole lot more, but for scientific computing this generally covers the very basics.\nThere is a ton online to be found that covers more advanced topics like container organization and management.\nYou can also check the Docker man page or run docker --help to see more subcommands.

How to use Singularity / Apptainer

The good news is that Singularity follows a lot of the same rules as Docker.

The main difference, is that instead of with Docker, where the images “live” somewhere in the /var/lib/docker folder (or somewhere else),\nSingularity images are usually stored in user space directories, with particular endings: .simg, .img or .sif.

There is also an analogous resource to DockerHub, called SingularityHub where images can be stored.\nYou can also pull Docker images directly through Singularity. There are some subtle differences in the way that\npermissions and environment variable mappings are performed with Singularity images vs Docker, so\nin general there can be some inconsistent behavior using a Docker image vs a Singularity-converted version of the same\nimage, but this can be mediated somewhat by building on top of a Docker container.

What this means in practice is that we can operate much the same as we do with Docker images. The main difference is that we first need to “pull” the\ncontainer before we operate on it. This is different from Docker because with Docker we can just run docker run [imagename] [command] and Docker will know it’s supposed to grab it (if it exists) from DockerHub.

With Singularity, we need to run commands in sequence:

singularity pull python-pytorch-image.sif docker://pytorch/pytorch:latest \n\n# this will pull from Docker hub and create file python-pytorch-image.sif\nsingularity shell python-pytorch-image.sif # will drop you into a shell\nsingularity exec python-pytorch-image.sif python # python interpreter

Note there is one command for shell and one command for exec. Most of the time\nyou will probably want to use exec to run generic programs, unless you are explicitly looking\nto get a shell from the image to inspect something. exec is somewhat\nequivalent to run from Docker.

Mounting volumes with `-B`

This functionality is basically the same as Docker, but instead of -v,\nwe need to use -B (note capital). Again you need to have multiple -B commands to mount multiple volumes.

How do I build my own containers?

The easiest thing to do is to find a Docker or Apptainer/Singularity container that already has the software you want.

The next easiest thing is to start from a container that has most of what you want, and then to build on top of that. This is by far the best way to build complex containers that utilize packages like CUDA—there are some sharp edges to be encountered with building these softwares even on a local machine.

The main tool you will use for this type of work is basically a config file called a Dockerfile or a Singularity / Apptainer recipe file.

For example, at the pytorch repository, in the main repository you can find a Dockerfile: https://github.com/pytorch/pytorch/blob/master/Dockerfile.

Let’s go through the sections of this file.

First, we find:

ARG BASE_IMAGE=ubuntu:18.04\nARG PYTHON_VERSION=3.8\n\nFROM ${BASE_IMAGE} as dev-base

So there are a couple important things to see here. Notice that each line has an all-uppercase command (either ARG or FROM) and then something afterwards. Every line in a Dockerfile needs to start (you can have a multiline command that you break up with \\ but in general each line should start with one of several of these commands).

Possibly commands are:

ARG : sets a variable within the Dockerfile
FROM : a special command that allows us to set a “base” image that we want to build off of. I will explain this in the next lines.
RUN : actually run a command, so for example this could be apt-get install SOMEPACKAGE or, pip install numpy. Anything you would want to run on the command-line can be run using RUN.
ENV : sets an environment variable inside the container
COPY : copy a file from the local environment (ie your computer) and the Docker image. Why would you want to do this? In case, for example, you have a particular script file that you want to put into your container, or maybe you have a config file you want to pull in for future usage.
WORKDIR : sets the working directory; sort of like cd-ing into a directory.

This is a subset of the commands, and you can read more on the Docker documentation

Back to the first couple lines of the Dockerfile, this is setting up an argument for the “base” image, so we are saying “I want to start with a fully-built Ubuntu 18.04 image” (this FROM can read from any Dockerhub image, so we can start from any image that has been uploaded to the hub).

Then, we set up a PYTHON_VERSION argument as well to keep track of the Python version—you don’t have to do this, but it can be convenient to keep track of arguments if you have something you refer to often.

The next section actually starts to build the command:

RUN apt-get update && apt-get install -y --no-install-recommends         build-essential         ca-certificates         ccache         cmake         curl         git         libjpeg-dev         libpng-dev &&     rm -rf /var/lib/apt/lists/*\nRUN /usr/sbin/update-ccache-symlinks

And this should start to look somewhat familiar. Here we are just installing a couple of basic packages. Notice that we are actually running two different bash commands in the first RUN that we split up with a &&. Also note that we are splitting the line up using \\ just to make it more readable. The next RUN command is then just to update system symlinks.

RUN mkdir /opt/ccache && ccache --set-config=cache_dir=/opt/ccache\nENV PATH /opt/conda/bin:$PATH

The next line is doing some more abstract commands with the C compiler. Then in the next line we are adding the conda installation path to $PATH.

The rest of the container build is going through some more sophisticated operations to basically make the container smaller.

But let’s skip forward and just try to understand the commands in general. In these next commands, we are installing conda:

RUN case ${TARGETPLATFORM} in          "linux/arm64")  MINICONDA_ARCH=aarch64  ;;          *)              MINICONDA_ARCH=x86_64   ;;     esac &&     curl -fsSL -v -o ~/miniconda.sh -O  "https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-${MINICONDA_ARCH}.sh"\nCOPY requirements.txt .\nRUN chmod +x ~/miniconda.sh &&     bash ~/miniconda.sh -b -p /opt/conda &&     rm ~/miniconda.sh &&     /opt/conda/bin/conda install -y python=${PYTHON_VERSION} cmake conda-build     pyyaml numpy ipython &&     /opt/conda/bin/python -m pip install -r requirements.txt &&     /opt/conda/bin/conda clean -ya

Just to really unpack this, we download the Miniconda3-latest-Linux-{SOME-COMPUTER-ARCHITECTURE}.sh shell script from the Anaconda website, then use it to install conda. We copy the requirements.txt file into the container and then use it to install whatever dependencies we need—we don’t have to do this, and we could actually just run a long command with the specific packages, for example:

RUN /opt/conda/bin/python -m pip install numpy pandas scipy\n\n# or\nRUN pip install numpy pandas scipy\n\n# or even\nconda install numpy pandas scipy

These are the basic operations that you would use to build a container. If you wrapped all of this stuff up into a Dockerfile, then we can actually build the container with this command:

# say we have named the file Dockerfile, which is the standard\n# you don't have to name it Dockerfile and if you don't specify\n# -f and if you don't it will assume you want to use a file called Dockerfile\n\ndocker build --tag SOME_IMAGE_NAME -f Dockerfile .

And if all goes well, you will see the image with that tag (you don’t have to use a tag, but it helps to keep track of things) in the output of docker images.

From there we can use the image like we discussed in the previous sections.

Converting to Singularity / Apptainer for use on Wynton

It’s super convenient to build images this way and then convert to Singularity / Apptainer if necessary. In general, Docker containers are stored as .tar.gz files stored in a specific file (usually /var/lib/docker). These can then be converted directly to Singularity images (which are more or less the same as Apptainer images) using the handy docker2singularity command: https://github.com/singularityhub/docker2singularity.

The example command is here:

$ docker run -v /var/run/docker.sock:/var/run/docker.sock -v /tmp/test:/output --privileged -t --rm quay.io/singularity/docker2singularity ubuntu:14.04

And here we are saying that we want to take the image ubuntu:14.04 (which can also be whatever image you have locally) and convert it to a singularity sif file (you will get a ubuntu_14.04…some…descriptors…sif file).

You can also convert a given Docker image by saving it as a tarball (basically a zip file) with:

docker save IMAGE_ID -o some-name.tar\n# image id is read directly from IMAGE ID in docker images\n# so for the prev example in figure 2 would be:\n\ndocker save 71eb2d092138 -o some-name.tar

Then we convert it using:

singularity build --sandbox my-new-simg.sif docker-archive://some-name.tar

which will have the same output. The --sandbox part we will discuss later.

Building Singularity / Apptainer images directly

A lot of the structure to how you use a Singularity “recipe” file is similar to that of Docker.

Here is a super simple version from the Singularity docs and this other website:

# let's call this image recipe.def\nBootstrap: docker\n\nFrom: tensorflow/tensorflow:latest\n\n%setup\n  touch /some-config-file-on-host\n\n%files\n  /some-config-file /opt\n  /some-other-file /tmp\n\n%post\n  apt-get update\n  apt-get upgrade -y\n  apt-get install -y python3\n  python3 -m pip install jupyter\n\n%runscript\n    exec /usr/bin/python "$@"

Basically now we split up the config process into several “operations”. The first part is fairly similar to what it was before—we say we want to start with a Docker image that are from the tensorflow/tensorflow:latest image on Dockerhub (note we could also set up from Singularity; other software vendors also have pullable sources such as nvidia container toolkit.

The first part, setup just runs things on your local machine. Here you might build some software you can then copy into the image, or just to set up maybe a log file or something similar.

The next part, files, is analogous to COPY in the Dockerfile, and refers to copying files from the host machine to the image. So, for example, here we are copying /some-config-file on the host to /opt.

Then, post is the section we are going to actually do most of the things we want to do. This is analogous to RUN in the previous section—we put stuff here where we would want to install some programs or do some kind of actual operations. For example, pull the Minicoda shell script and install Miniconda like we did in the previous example.

runscript is a new section where we can specify what happens when we run singularity run [MYIMAGE]; which will automatically, in this case, run python on the next thing in the command.

This is different from the previous version of what I showed you, which used singularity exec [MYIMAGE] SOMETHING where SOMETHING could be any command—singularity run is really meant to facilitate using a single program. A potential use-case for this is if we had a container that runs a huge pipeline—we could just package a runscript here that runs the entire thing from the argument and allows the user to skip a bunch of long command type-ins.

Then we can build the image using:

singularity build my-image-name.sif myrecipe.def

"}},{"slug":"transformertheory","layout":"blog","title":"Theories on transformer function","author":"alex","subtitle":"SVMs and rate-reduction","description":"A quick review of two recent papers in transformer research","date":"2023-02-01","published":true,"tags":["ml","paper"],"category":"Paper","md":{"html":"

Theories on transformer function

Transformers are SVMs

I recently saw a new paper called “Transformers are Support Vector Machines” which I thought was very interesting. The authors describe a way to interpret the attention mechanism, a key component of transformers that can be difficult to interpret. The idea is that the action of attention parameterizes an SVM at every layer, which the function of learning to separate (or give very small or zero softmax weight) to the irrelevant parts of the input. This comes in the form of maximizing the distance between sets of tokens (the “good” ones and the “bad” ones), as in the below figure, where the lines illustrate the different trajectories of two token sets (presumably, the red lines refer to the “good” tokens) and the inset shows with stars the “optimal” SVM-solution tokens.

Transformers are coding rate reducers

The idea of creating separability or “distance” between related samples in some arbitrary data matrix reminds me of the principle of rate reduction, as defined in the papers “White-box transformers via sparse rate-reduction and “Learning diverse and discriminative representations via the principle of maximal coding rate reduction”. This second work establishes three basic principles for desirable neural representations:

between-class discriminative: features from different classes should end up in different low-dim and hopefully linear subspaces
within-class compressible: features from similar classes should end up in similar low-dimensional subspaces (i.e. the subspaces should be “close” to each other)
maximially diverse representation: the overall variance of each class should be as large as possible (intuitively this is meant to prevent feature collapse, where for example a class could be mapped to a singular vector or an inappropriately low-rank subspace as can happen in some algorithms)

What this looks like is something like this:

And the idea is to use $\\mathrm{log} \\det(\\mathbf{Z})$ as a way to quantify the “volume” of a data matrix. If $\\mathbf{Z}$ is a subspace with correlated features, then $\\mathrm{log} \\det(\\mathbf{Z})$ will be small (left of figure). If $\\mathbf{Z}$ is a subspace with uncorrelated features, then $\\mathrm{log} \\det(\\mathbf{Z})$ will be large (right of figure). The goal of representation learning might then be posed as a way to discover a feature transformation such that the overall volume of the data matrix is maximized, but where there is high separability and low coherence between the features for each class-specific subspace.

A recent investigation into transformers extends some of this theory to t ransformers, establishing a class of models which interprets the two building blocks of the self-attention layer from this perspective. Self-attention operations are viewed as incremental rate-reduction compression operations. The MLP operations are then interpreted as ISTA-like operations (shrinkage/sparsifying operations) that sparsify the token features with respect to some global basis functions (referred to as a global dictionary). This class of methods is then referred to as CRATE, or Coding Rate Transformer. The performance of these models is not quite as good as, for example, ViT, but the authors suggest that the benefits of interpretability are worth the trade-off and that a number of innovations have gone into training ViT and ViT-like models.

"}},{"slug":"vaetalk","layout":"presentation","title":"Identifiability in VAEs","author":"alex","subtitle":"Recent papers in VAE research","description":"A talk for a journal club on innovations in VAEs","date":"2023-01-31","published":true,"tags":["ml","paper"],"category":"Presentation","md":{"html":"

I gave this talk for a journal club on new methods in VAE research, mostly from reading the paper Provable concept learning for interpretable predictions using variational autoencoders. I think the paper is really interesting and I learned a lot from it. I then read some of the papers (Khemakhem et al. and Locatello et al.) which really were informative—I’ve since quite enjoyed reading papers from the same labs (Scholkopf and Hyvärinen). Particularly Hyvärinen, because of their innovation (which I had not been aware of) in ICA, or independent components analysis, which I used quite a lot during my fMRI analysis projects. Coincidentally I also found that Hyvärinen worked on sliced score matching back in the early 2000s and has since done a lot more research in the foundations of identifiable, (more) interpretable deep learning models.

"}},{"slug":"attention-across-datapoints","layout":"blog","title":"Attention across datapoints","author":"alex","subtitle":"A new neural network architecture for tabular-ish data","description":"A paper notes summary about Kossen et al. 2022.","date":"2022-07-31","published":false,"tags":["ml","paper"],"youtube":"https://www.youtube.com/watch?v=zejXBg-2Vpk&list=PLoROMvodv4rNiJRchCzutFw5ItR_Z27CM&index=7","link":"https://arxiv.org/abs/2106.02584","github":"https://github.com/OATML/Non-Parametric-Transformers","category":"Paper","md":{"html":"

Author abstract:

We challenge a common assumption underlying most supervised deep learning:\n that a model makes a prediction depending only on its parameters and the features\n of a single input. To this end, we introduce a general-purpose deep learning\n architecture that takes as input the entire dataset instead of processing one datapoint\n at a time. Our approach uses self-attention to reason about relationships between\n datapoints explicitly, which can be seen as realizing non-parametric models using\n parametric attention mechanisms. However, unlike conventional non-parametric\n models, we let the model learn end-to-end from the data how to make use of other\n datapoints for prediction. Empirically, our models solve cross-datapoint lookup\n and complex reasoning tasks unsolvable by traditional deep learning models. We\n show highly competitive results on tabular data, early results on CIFAR-10, and\n give insight into how the model makes use of the interactions between points.

Introduction

Supervised DL workflows often utilize some data: $\\mathcal{D}_{train} = \\{(x_1, y_1), ..., (x_n, y)\\}$ to learn a function $f$ that maximizes $p(\\mathbf{y}\\ |\\ \\mathbf{x}; \\theta)$ .

these functions produce test-time predictions of the form $p(\\mathbf{y}^*\\ |\\ \\mathbf{x}^*; \\theta)$ where there is no explicit link between $\\mathcal{D}_{train}$ and $\\theta$ .

This paper introduces an architecture called nonparametric transformers, or NPT, which the authors suggest have the following advantages:

explict learning of interactions between datapoints using self-attention
usage of non-parametric and parametric mechanisms.

The hope is then to learn functions that can produce a test-time prediction of the form $p(\\mathbf{y}^*\\ |\\ \\mathbf{x}^*, \\mathcal{D}_{train}; \\theta)$ , schematized in $(c)$ and $(d)$ below.

NPTs take entire datasets as input

The general framework is this:

We have, again, some $\\mathcal{D}$ , which we provide as the entire data matrix $\\mathbf{X} \\in \\mathbb{R}^{n\\ \\mathrm{x}\\ d}$ .
- Columns of $\\mathbf{x}$ are referred to as attributes $j$ . For classic regression or classification, the paper refers to the target variable as $\\mathbf{X}_{:,\\ d}$ , with other attributes $\\{\\mathbf{X}_{:,j}\\ |\\ j \\neq d\\}$ as input features.
- The paper uses a similar framework as in masked language modeling. They introduce a masking matrix $\\mathbf{M} \\in \\mathbb{R}^{n\\ \\mathrm{x}\\ d}$ to specify masked elements of $\\mathbf{X}$ .
- The model is then trained to predict masked entries: $\\mathbf{X}^{M} =\\ \\{\\mathbf{X}_{i,j}\\ |\\ \\mathbf{M}_{i,j}\\}$ .

The initial setting for the paper is that of single-target classification or regression, where the masking matrix $\\mathbf{M}$ has 1 at every entry of the target column $\\mathbf{X}_{:,\\ d}$ .

Alternating self-attention over datapoint embeddings and datapoint attributes are the central operations in NPT

The interesting part of the method starts to really come out here. What we will do is compute an embedding for each column (feature) of $\\mathbf{X}$ . The reason for this is twofold:

allow for the computation of a per-variable “mask” embedding
allow for embedding of all features into an embedding space of the same dimension (so, if we set some embedding dimension $e$ , we would encode all variables, both categorical and continuous, into $\\mathbb{R}^e$ )
- to spell this out further, if we have a continuous variable $x_j$ then we would provide the model with a an $(n\\ \\mathrm{x}\\ 2)$ matrix, with one column continuous-valued and the other an indicator variable for whether the data point is missing. To see this, note the npt.Trainer object in the npt codebase, specifically the npt.Trainer.run_epoch method, along with the npt.model.NPTModel.in_embeddings definition.
- if we had a categorical variable of cardinality 3, we would provide the model with a one-hot encoded vector of length 4 (“mask” and the three original categorical variables).

Each data “batch” contains some number of datapoints that will be encoded as above, as well as a masking matrix that denotes a given “target” column or a different type of masked-learning procedure.
- the way the data is provided, for example, in the breast cancer dataset (569 observations, with 31 variables each [not counting the ID]):
  - a series of $(569\\ \\mathrm{x}\\ d)$ matrices, with $d = 3$ if categorical (original cardinality is 2), or $d = 2$ if continuous.
- the code then embeds these into a latent dimension of dimension $e$ .

A small review of self-attention

For a nice review of self-attention, see Lillian Weng’s blog post about it: Attention? Attention! I think if I read and absorbed all of Lillian’s blog posts maybe I would get out of grad school faster.
To recap some of the terminology used in the paper here, multi-headed dot product attention is defined as:

$\\mathrm{MHSelfAtt}(\\mathbf{Q}, \\mathbf{K}, \\mathbf{V}) = \\mathrm{concat}_h(\\mathbf{O}_1, ..., \\mathbf{O}_k)\\mathbf{W}^O$ , with:

$\\mathbf{O}_j = \\mathrm{Attention}(\\mathbf{Q}\\mathbf{W}_{j}^{Q}, \\mathbf{K}\\mathbf{W}_{j}^{K}, \\mathbf{V}\\mathbf{W}_{j}^V)$

this paper also combines MHA with LayerNorm and skip connections, for:

$\\mathrm{Res}(H) = \\mathbf{H}\\mathbf{W}^\\mathrm{res} + \\mathrm{MHSelfAtt}(\\mathrm{LN}(H))$

where the residual branch incorporates a weight matrix $\\mathbf{W}^\\mathrm{res} \\in \\mathbb{R}^{h\\ \\mathrm{x}\\ h}$ .
the authors also use a second residual branch, parameterized by a row-wise feed forward network, finally producing:

$\\mathrm{MHSA}(\\mathbf{H}) = \\mathrm{Res}(\\mathbf{H}) + \\mathrm{rFF}(\\mathrm{LN}(\\mathrm{Res}(\\mathbf{H}))) \\in \\mathbb{R}^{n\\ \\mathrm{x}\\ h}$

Modifications of attention to allow for attention between datapoints, and attention between attributes (features)

the innovative advance in this paper is to come up with a way to modify MHSA to allow for explicit comparison across pairs of variables (ABA) and pairs of datapoints (ABD).
this part was a bit hard to follow at first, so please bear with me:

Attention between Datapoints (ABD)

The input to ABD is a reshaped layer output $\\mathbf{H}^{(l)} \\in \\mathbb{R}^{n \\mathrm{x}\\ h}, h = d * e$
MHSA is applied between datapoint representations, yielding:

$\\mathrm{ABD}(\\mathbf{H}^{(l)}) = \\mathrm{MHSA}(\\mathbf{H}^{(l)}) = \\mathbf{H}^{(l+1)} \\in \\mathbb{R}^{n\\ \\mathrm{x}\\ h}$

For example, in the first layer, we input $\\mathbf{H}^{(0)} \\in \\mathbf{R}^{n\\ \\mathrm{x}\\ d\\ \\mathrm{x}\\ e}$ , the linearly input embedding data. Then after ABD, the output is shaped back into $\\mathbb{R}^{n\\ \\mathrm{x}\\ d\\ \\mathrm{x}\\ e}$ from $\\mathbb{R}^{n\\ \\mathrm{x}\\ h}$ .
- The rFF operation is an MLP that is applied separately to all datapoints (“row-wise”)
The sort of prototypical case that the authors pose would be ideal would be if through these operations, you could learn an optimal way to embed a given $\\mathbf{Q}_i$ and $\\mathbf{K}_i$ close together (or far away). There might be additional benefits to stacking these layers and using them to learn more complex interactions between pairs of datapoints.

Attention between Attributes (ABA)

ABD is designed to facilitate comparisons between datapoints, given some features. ABA is designed to allow the model to refine a given datapoint’s representation by attending to its own state.
Each row (datapoint) is indexed by $\\mathbf{H}^{(l)}_i, i\\ \\in\\{1, ..., n\\}$ , giving:

$\\mathrm{ABA}(\\mathbf{H}_{i}^{(l)}) = \\mathrm{MHSA}(\\mathbf{H}_{i}^{(l)}) = \\mathbf{H}^{(l+1)} \\in \\mathbb{R}^{1\\ \\mathrm{x}\\ d\\ \\mathrm{x}\\ e}$ .

"}},{"slug":"check","layout":"blog","title":"df across datapoints","author":"alex","description":"A paper notes sum al. 2022. elephant","subtitle":"something","date":"2022-01-01","published":false,"tags":["ml","paper"],"youtube":"https://www.youtube.com/watch?v=zejXBg-2Vpk&list=PLoROMvodv4rNiJRchCzutFw5ItR_Z27CM&index=7","link":"https://arxiv.org/abs/2106.02584","github":"https://github.com/OATML/Non-Parametric-Transformers","category":"Note","md":{"html":"

What are Docker and Singularity/Apptainer?

Note: this is a guide I made for a student I was TA’ing for, and I thought it might be useful to others. I tried to make it relatively comprehensive, but it is not meant to be a complete guide and the language is meant to not be really, really technical.

The main idea of these softwares, which are referred to as containerization softwares, is to encapsulate a programming environment in a single file.

This is obviously highly useful for reproducible computing, but\ncan also help make your life easier by allowing you to perform the same computation on the same environment across different places.

What is the difference between Docker and Singularity/Apptainer?

In particular, you cannot use Docker images on Wynton—but you can use Singularity. One issue is that\nyou cannot create Singularity images on Wynton (with some notable exceptions).

How do I use a Docker or Singularity image?

How to use Docker

Pulling a Docker image from DockerHub

Say, for example, we want to use a Docker image equipped with the CUDA libraries and PyTorch.

This is sort of a simple case that might be easy, maybe we want to use a specific CUDA version or something similar.

What we can do, then, is type in at the command line [see here to see the specific versions of the images provided]:

docker pull pytorch/pytorch:latest # the text after the colon indicates the version; \n# alternatively: \n# docker pull pytorch/pytorch:1.31.1-cuda11.6-cudnn8-runtime

When we do this, we get an output that looks like:

"}}]

What does representation learning in biology even mean?

Understanding Diffusion Models: A Unified Perspective

Background: VAEs, Hierarchical VAEs and the ELBO

Variational autoencoders

What are Docker and Singularity/Apptainer?

What is the difference between Docker and Singularity/Apptainer?

How do I use a Docker or Singularity image?

How to use Docker

Pulling a Docker image from DockerHub

Using the image and using GPU’s

Binding directories using docker -v

How to use Singularity / Apptainer

Mounting volumes with -B

How do I build my own containers?

Converting to Singularity / Apptainer for use on Wynton

Building Singularity / Apptainer images directly

Theories on transformer function

Transformers are SVMs

Transformers are coding rate reducers

Introduction

NPTs take entire datasets as input

Alternating self-attention over datapoint embeddings and datapoint attributes are the central operations in NPT

A small review of self-attention

Modifications of attention to allow for attention between datapoints, and attention between attributes (features)

Attention between Datapoints (ABD)

Attention between Attributes (ABA)

What are Docker and Singularity/Apptainer?

What is the difference between Docker and Singularity/Apptainer?

How do I use a Docker or Singularity image?

How to use Docker

Pulling a Docker image from DockerHub

Binding directories using `docker -v`

Mounting volumes with `-B`