Skip to content

Transformer实现默认用的Pre-Norm? #51

Open
@KKIverson

Description

@KKIverson

2.1.5中EncoderLayer里涉及到add&norm的操作,貌似都是先对原始输入做norm,然后将norm后的数据输入MHA/FFN,然后再和原始输入相加,这不是pre-norm的操作吗?按理说原生的Transformer不都是post-norm嘛~

Activity

YingxuanW

YingxuanW commented on Mar 21, 2024

@YingxuanW

看到这里同样有疑问,个人会这样实现:
x = norm(x + attention(x,x,x,mask))
x =norm(x + ffn(x))

hedes1992

hedes1992 commented on Jun 13, 2024

@hedes1992

看到这里同样有疑问,个人会这样实现: x = norm(x + attention(x,x,x,mask)) x =norm(x + ffn(x))

yes, I find the following is the right

https://github.com/hyunwoongko/transformer/blob/master/models/blocks/encoder_layer.py

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions

      Transformer实现默认用的Pre-Norm? · Issue #51 · intro-llm/intro-llm.github.io