Representational Similarity Analysis (RSA)

The first time RSA (Representational Similarity Analysis) came up in one of my classes, I was lost. People kept saying “dissimilarity matrix”, and I couldn’t work out what that had to do with the brain, or what was actually being compared to what. It took a lot of questions to my TA before it made sense, so this is the explanation I wish I’d had at the start.

What Even Is a Representation?

Before getting to RSA, it helps to pin down what “representation” means. When you look at a cat, your brain isn’t storing a tiny JPEG of it somewhere in your neurons. Neurons in your visual cortex fire in a particular pattern, and that pattern represents the cat. Thinking about the word “cat” or hearing someone say it produces other patterns, in other brain regions.

Deep neural networks have representations too. Your brain might activate 100,000 neurons in some pattern when it sees a cat, and a network processing the same image might activate 512 artificial neurons in a pattern of its own. Both are representations of the cat, in completely different formats.

The Problem RSA Solves

So how do you compare representations that live in completely different spaces? For example:

1. Your visual cortex might represent images as patterns of activity across thousands of neurons (measured through fMRI voxels)
2. A deep neural network might represent the same images as patterns across hundreds of artificial neurons
3. Your auditory cortex might represent sounds as yet another pattern, across a different set of neurons

You can’t line these patterns up directly. They have different numbers of dimensions, they’re measured in different ways, and they might use completely different coding schemes.

Imagine comparing two descriptions of the same movie, one in English and one in Japanese. You can’t compare the words, but you can compare how the concepts relate to each other in each description.

How RSA Works: A Concrete Example

Say we want to compare how objects are represented in the human brain (measured with fMRI) and in a deep neural network (DNN). We show both of them the same five images: a cat, a dog, a car, a house and a tree.

For each image, the fMRI gives us a pattern of 10,000 voxel activations and the DNN gives us a pattern of 512 neuron activations. Those can’t be compared directly, so RSA compares how each system treats the images relative to one another:

1. For the fMRI data, we calculate how similar each image’s pattern is to every other image’s pattern. How similar is the cat pattern to the dog pattern? The cat pattern to the car pattern? And so on, for all pairs.

2. We do the same for the DNN patterns.

This gives us two similarity matrices. Here’s what they might look like, with made-up numbers:

Now we can ask questions like:

• Does the DNN find cats and dogs similar in the same way the brain does?
• Do both systems treat cats and cars as very different?
• Is the overall pattern of similarities in the DNN similar to the pattern in the brain?

Papers usually work with dissimilarity instead, often 1 minus the correlation, which is where the “dissimilarity matrices” that confused me come from. The logic is the same.

The Code

Here’s how you might calculate these matrices and compare them. The ... stands in for the rest of each pattern, so you can’t run this as is.

import numpy as np
from scipy.stats import spearmanr

# Example patterns (simplified)
fmri_patterns = {
    'cat':   [0.2, 0.5, 0.1, ...],  # 10,000 voxels
    'dog':   [0.3, 0.4, 0.2, ...],
    'car':   [0.1, 0.2, 0.5, ...],
    'house': [0.4, 0.1, 0.3, ...],
    'tree':  [0.2, 0.3, 0.4, ...]
}

dnn_patterns = {
    'cat':   [0.7, 0.2, ...],  # 512 neurons
    'dog':   [0.6, 0.3, ...],
    'car':   [0.1, 0.8, ...],
    'house': [0.2, 0.7, ...],
    'tree':  [0.4, 0.5, ...]
}

# Calculate similarity between two patterns
def similarity(pattern1, pattern2):
    # Often using correlation coefficient
    return np.corrcoef(pattern1, pattern2)[0,1]

# Create similarity matrices
def create_similarity_matrix(patterns):
    n = len(patterns)
    matrix = np.zeros((n, n))
    for i, stim1 in enumerate(patterns):
        for j, stim2 in enumerate(patterns):
            matrix[i,j] = similarity(patterns[stim1], patterns[stim2])
    return matrix

fmri_similarities = create_similarity_matrix(fmri_patterns)
dnn_similarities = create_similarity_matrix(dnn_patterns)

# Compare the two matrices, using only the values above the diagonal
upper = np.triu_indices_from(fmri_similarities, k=1)
rsa_score, _ = spearmanr(fmri_similarities[upper], dnn_similarities[upper])

The original patterns have 10,000 and 512 dimensions, but both matrices come out 5×5, with one row and one column per image. So you now have two matrices of the same shape, indexed by the same images, and you can compare them.

The last two lines of the code do that. Each matrix is symmetric, and its diagonal is all 1s because every pattern is identical to itself, so only the values above the diagonal carry information. With five images, that’s 10 numbers per matrix. Correlating those two lists of 10 numbers gives you a single RSA score. People often use Spearman’s rank correlation for this, since it only asks whether the two systems rank the image pairs in a similar order. The made-up numbers in the chart above give a score of about 0.91.

What It’s Used For

RSA gives you a way to connect different ways of measuring and thinking about the brain. You can use it to compare:

• Brain activity across different brain regions
• Brain activity across different people
• Brain activity with computer models
• Brain activity with behavior

In my class, it was used to compare how well different deep learning models matched human brain activity, but the same technique shows up in work on everything from object recognition to social cognition to motor control.

Common Pitfalls

1. Stimulus selection. Your matrices can only show structure that your stimuli are able to reveal. If all your stimuli are too similar or too different, you might not see anything meaningful.

2. Noise. If your measurements are too noisy, your similarity matrices will be unreliable, so check the reliability of your data.

3. Distance metric. Correlation, Euclidean distance and other measures can give you different results, so think about which properties of your data matter for your question.

4. Statistical testing. The usual p-value for a correlation assumes independent values, and the values in a similarity matrix aren’t independent (cat-dog and cat-car both involve the cat). Mantel tests and permutation tests are the usual alternatives.

5. Preprocessing. Normalization, detrending and other preprocessing can change your similarity matrices a lot, so document everything.

Final Thoughts

RSA clicked for me when I stopped thinking of it as a complicated mathematical technique and started seeing it as a way to compare patterns of relationships. You can’t directly compare apples and oranges, but you can compare how apples relate to other apples with how oranges relate to other oranges.

Now when I read a paper using RSA, I ask:

1. What patterns are they comparing?
2. What stimuli did they use to probe these patterns?
3. Why do they expect the similarity structures to match?
4. What would it mean if they find (or don’t find) the similarities they expect?