[Tensorflow 101] What does it mean to reduce axis?

i.e. reduce_sum, reduce_max, reduce_mean, etc.

In Tensorflow code, you may have seen “reduce_*” many times. When I first used tf.reduce_sum, I thought, if it’s a sum, just say sum! Why do you have to put the prefix “reduce_” in front of every command?

def _scale_l2(x, norm_length):
alpha = tf.reduce_max(tf.abs(x), (1, 2), keep_dims=True) + 1e-12
l2_norm = alpha * tf.sqrt(
tf.reduce_sum(tf.pow(x / alpha, 2), (1, 2), keep_dims=True) + 1e-6)
x_unit = x / l2_norm
return norm_length * x_unit
# Let's initialize the tensor.
In [3]: x = tf.constant([[[1,2,3,4,5], [4,5,6,7,8]],
[[2,4,6,8,10],[3,6,9,12,15]],
[[8,8,8,8,8], [9,9,9,9,9]]])
In [9]: sess = tf.InteractiveSession()
# Let's see how it looks.
In [10]: x.eval()
Out[10]:
array([[[ 1, 2, 3, 4, 5],
[ 4, 5, 6, 7, 8]],
[[ 2, 4, 6, 8, 10],
[ 3, 6, 9, 12, 15]],
[[ 8, 8, 8, 8, 8],
[ 9, 9, 9, 9, 9]]], dtype=int32)

[Warm up Q.] What would be the output of tf.reduce_max(x, 0)?

In [11]: tf.reduce_max(x,0)
Out[11]: <tf.Tensor 'Max:0' shape=(2, 5) dtype=int32>
In [12]: tf.reduce_max(x,0).eval()
Out[12]:
array([[ 8, 8, 8, 8, 10],
[ 9, 9, 9, 12, 15]], dtype=int32)
In [14]: tf.reduce_max(x,1)
Out[14]: <tf.Tensor 'Max_3:0' shape=(3, 5) dtype=int32>
In [15]: tf.reduce_max(x,1).eval()
Out[15]:
array([[ 4, 5, 6, 7, 8],
[ 3, 6, 9, 12, 15],
[ 9, 9, 9, 9, 9]], dtype=int32)
In [16]: tf.reduce_max(x,(1,2)).eval()
Out[16]: array([ 8, 15, 9], dtype=int32)
In [17]: tf.reduce_max(x,(1,2))
Out[17]: <tf.Tensor 'Max_11:0' shape=(3,) dtype=int32>
In [22]: tf.reduce_max(x,(1,2), keepdims=True).eval()
Out[22]:
array([[[ 8]],
[[15]], [[ 9]]], dtype=int32)In [23]: tf.reduce_max(x,(1,2), keepdims=True)
Out[23]: <tf.Tensor 'Max_10:0' shape=(3, 1, 1) dtype=int32>

I’m an Engineering Manager at Scale AI and this is my notepad for Applied Math / CS / Deep Learning topics. Follow me on Twitter for more!

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store