Implement gpt_params(vocab, d_model, n_layers, d_ff=None, max_pos=0, tied=True) for a GPT-2-style decoder, where every linear layer and layer norm has a bias:
vocab × d_model, plus learned position embeddings max_pos × d_modeld × d plus a bias of size d), an MLP (d × d_ff plus bias, then d_ff × d plus bias) and two layer norms (scale and bias, each of size d). d_ff defaults to 4 * d_model.tied, and otherwise adds vocab × d_model (no bias)Also implement:
non_embedding_params(...) with the same arguments: the total minus the token and position embeddings (and minus the untied head). This is the count scaling laws use.forward_flops_per_token(n_params_non_embedding): about 2 × N