Softmax Approximation for FPGAs Using Taylor Series and Pade Approximants

Quick Summary

We implemented Online Softmax and found CPU and GPU-specific optimal implementations. This paper presents a third approach for FPGAs.

We teach programmers how to turn Arxiv research papers into C and Python code. Subscribe if you found this useful.

1.0 Paper Introduction

A Quantitative Evaluation of Approximate Softmax Functions for Deep Neural Networks (Leiva-Valverde et al., 2025)1 compares efficient implementations of softmax for FPGAs using Taylor series approximations and Pade approximants.

The paper’s motivation is the hardware constraints limiting exponential operations on FPGAs.

The authors use the root mean square error (RMSE) for accuracy assesment and conclude:

1. For low error: use quadratic interpolation with LUTs.
2. For performance: use Taylor and Padé approximations.

We implemented Online Softmax and found CPU and GPU-specific optimal implementations. This paper presents different optimal approachs for FPGAs.

2.0 Coding Softmax for FPGA

Code is available on GitHub. We use this softmax definition all through:

We call this Regular Softmax in our code:

We use naive Python implementations for Taylor and Pade approximants.

2.1 Taylor Series Approximation

There exists this known Taylor expansion for exponential:

Read more