Representational Similarity Analysis (RSA)

When I first encountered Representational Similarity Analysis (RSA) in one of my classes, I felt completely lost. Everyone was throwing around terms like "dissimilarity matrices", but I couldn't quite grasp how these connected to understanding the brain, and what comparisons were actually being made. After diving asking a bunch of questions to my TA, I finally got it. Here's how I wish it had been explained to me at the start.

What Even Is a Representation?

Before we dive into RSA, let's talk about what we mean by "representation." Imagine you're looking at a cat. Your brain isn't storing a tiny JPEG of that cat somewhere in your neurons. Instead, different neurons in your visual cortex fire in specific patterns. That pattern of neural activity represents the cat. Similarly, when you think about the word "cat" or hear someone say "cat," different neural patterns emerge in different brain regions.

Think of it like this: when your brain sees a cat, it might activate 100,000 neurons in a particular pattern. When a deep neural network processes the same cat image, it might activate 512 artificial neurons in its own pattern. These are both "representations" of the cat, but they're in completely different formats and spaces.

The Core Problem RSA Solves

Here's the fundamental problem that RSA helps us solve: How do we compare representations that exist in completely different spaces? For example:

1. Your brain's visual cortex might represent images using patterns of activity across thousands of neurons (measured through fMRI voxels)
2. A deep neural network might represent the same images using patterns across hundreds of artificial neurons
3. Your brain's auditory cortex might represent sounds using yet another pattern across a different set of neurons

We can't directly compare these patterns because:

• They have different numbers of dimensions (thousands vs hundreds of neurons)
• They're measured in different ways (fMRI vs neural network activations)
• They might use completely different coding schemes

It's like trying to compare two descriptions of the same movie, where one is in English and one is in Japanese - you can't directly compare the words, but you can compare the relationships between concepts in each description.

How RSA Works: A Concrete Example

Let's walk through a real example. Imagine we're studying how objects are represented in:

1. The human brain (via fMRI)
2. A deep neural network (DNN)

We show five images to both systems:

• A cat
• A dog
• A car
• A house
• A tree

For each image:

• The fMRI gives us a pattern of 10,000 voxel activations
• The DNN gives us a pattern of 512 neuron activations

We can't directly compare these patterns because they're in completely different spaces. But here's the clever part of RSA:

1. For the fMRI data, we calculate how similar each image's pattern is to every other image's pattern:

   • How similar is the cat pattern to the dog pattern?
   • How similar is the cat pattern to the car pattern?
   • And so on for all pairs

2. We do the same for the DNN patterns:

   • How similar is the cat pattern to the dog pattern?
   • How similar is the cat pattern to the car pattern?
   • And so on

This gives us two similarity matrices that look like this:

Now we can ask interesting questions like:

• Does the DNN find cats and dogs similar in the same way the brain does?
• Do both systems show that houses and cars are very different?
• Is the overall pattern of similarities in the DNN similar to the pattern in the brain?

The Math Part (Don't Run Away!)

Let's make this concrete with some code. Here's how we might calculate these similarity matrices:

# Example patterns (simplified)
fmri_patterns = {
    'cat':   [0.2, 0.5, 0.1, ...],  # 10,000 voxels
    'dog':   [0.3, 0.4, 0.2, ...],
    'car':   [0.1, 0.2, 0.5, ...],
    'house': [0.4, 0.1, 0.3, ...],
    'tree':  [0.2, 0.3, 0.4, ...]
}
 
dnn_patterns = {
    'cat':   [0.7, 0.2, ...],  # 512 neurons
    'dog':   [0.6, 0.3, ...],
    'car':   [0.1, 0.8, ...],
    'house': [0.2, 0.7, ...],
    'tree':  [0.4, 0.5, ...]
}
 
# Calculate similarity between two patterns
def similarity(pattern1, pattern2):
    # Often using correlation coefficient
    return np.corrcoef(pattern1, pattern2)[0,1]
 
# Create similarity matrices
def create_similarity_matrix(patterns):
    n = len(patterns)
    matrix = np.zeros((n, n))
    for i, stim1 in enumerate(patterns):
        for j, stim2 in enumerate(patterns):
            matrix[i,j] = similarity(patterns[stim1], patterns[stim2])
    return matrix
 
# Now we can compare the similarity matrices!
fmri_similarities = create_similarity_matrix(fmri_patterns)
dnn_similarities = create_similarity_matrix(dnn_patterns)

The beauty is that even though our original patterns were in completely different spaces (10,000 voxels vs 512 neurons), our similarity matrices are directly comparable because they're both just showing how similar each stimulus is to every other stimulus in that representation space.

Why This is Actually Cool

When I finally got RSA, I realized it's solving a fundamental problem in neuroscience: How do we connect different ways of measuring and thinking about the brain? It's like having a universal translator that lets us compare:

• Brain activity across different brain regions
• Brain activity across different people
• Brain activity with computer models
• Brain activity with behavior

In the class I took, it's used to compare how well different deep learning models matched human brain activity. But the same technique is used to study everything from object recognition to social cognition to motor control.

Common Pitfalls

1. Stimulus Selection Matters: Your similarity matrices are only meaningful if you've chosen stimuli that can reveal interesting patterns. If all your stimuli are too similar or too different, you might not see meaningful structure.

2. Noise Level Matters: If your measurements are too noisy, your similarity matrices will be unreliable. Always check the reliability of your data!

3. Distance Metric Choice: Different ways of calculating similarity (correlation, Euclidean distance, etc.) can give you different results. Think carefully about what properties of your data are important for your question.

4. Statistical Testing: When comparing similarity matrices, you need to use appropriate statistical tests (like Mantel tests or permutation tests) - a regular correlation isn't always appropriate.

5. Preprocessing Steps: How you preprocess your data (normalization, detrending, etc.) can hugely affect your similarity matrices. Document everything!

Final Thoughts

RSA clicked for me when I stopped thinking about it as a complex mathematical technique and started seeing it as a way to compare patterns of relationships. Instead of trying to directly compare apples and oranges, it lets us compare how apples relate to other apples and how oranges relate to other oranges.

Now when I read papers using RSA, I ask:

1. What patterns are they comparing?
2. What stimuli did they use to probe these patterns?
3. Why do they expect the similarity structures to match?
4. What would it mean if they find (or don't find) the similarities they expect?

And suddenly, it all makes a lot more sense!