You Don't Need Matrix Calculus
This article argues that matrix calculus is often unnecessary for machine learning tasks and suggests that using index notation is a simpler, more robust alternative. The author demonstrates how writing out formulas with indices avoids the complexity of memorizing specific matrix differentiation rules.
Why it matters
Provides a practical, pedagogical perspective for machine learning practitioners to simplify their mathematical workflows.
Matrix calculus pops up on social media from time to time, primarily due to the increased popularity of machine learning. It lets you differentiate functions of vector or matrix variables even if you are not a fan of automatic differentiation. It involves all kinds of complex rules, depending on whether your vectors are row or column vectors, and it gets even worse for matrices. Let’s not speak of tensors.
You do not need matrix calculus . When physists started working with tensors almost a century ago, they figured out the easy way to do this: instead of trying to come up with a system for tensor differentiation \(\partial A / \partial B\), just write out the formulas for \(A_{ijk}\) and \(B_{ijk}\) and use ordinary differentiation.
Get smarter about the news
Sign up free for a feed built around what you actually care about, Dive Deeper research on any story, and the full text of every article.
Create free accountAlready have an account? Sign in