[Paper] Squeeze-and-Excitation Networks

1 분 소요

“Squeeze-and-Excitation Networks”이란 논문에 대한 리뷰입니다.

원문은 링크에서 확인할 수 있습니다.

Key

Explicitly modelling the interdependencies btw the channels of conv layer
Squeeze: aggregating feature map  produce embedding of global distribution of channel-wise feature response
Excitation: self-gating mechanism for collections of per-channel modulation weights
Acts like attention which bias the allocation of computational resources toward the most informative components and in SENet, use spatial and channel attention.

Squeeze
RQ: learned filters operate with a local receptive field being unable to contextual information outside of this region
Robust to the operator, Maxpool or AvgPool.
Excitation
RQ: capture channel-wise dependencies
capable of learning a nonlinear interaction btw channels
learn a non-mutually-exclusive relationship to ensure that multiple channel’s are allowed to be emphasized
At the earlier layers, distribution across different classes is similar because they share feature channels in early stage
As deeper, the value of each channel becomes much more class-specific where the diversity of representation within a single class arises resulting in instance-specific response

Role at different depths
In early layer, it excites informative features in a “class-agnostic” manner
In later layer, in a highly “class-specific” manner
Information flow
In original CNN-based model structure, channel dependencies are implicitly embedded in filters where the local spatial correlation is captured by kernel
In SE block, squeeze step explicitly modelling global information and excitation step recalibrate filter response
Deepening the layer, use two Linear layer with dimension reduction and extension using nonlinear activation