Skip to content

TypeError: LlamaSdpaAttention.forward() got an unexpected keyword argument 'lookahead' #59

Description

@zev123456

run:
USE_LADE=1 LOAD_LADE=1 python minimal.py
result:

Traceback (most recent call last):
  File "/home/workspace/LookaheadDecoding/minimal.py", line 32, in <module>
    greedy_output = model.generate(**model_inputs, max_new_tokens=1)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/lade/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/lade/lib/python3.12/site-packages/transformers/generation/utils.py", line 1718, in generate
    return self.greedy_search(
           ^^^^^^^^^^^^^^^^^^^
  File "/home/workspace/LookaheadDecoding/lade/decoding.py", line 24, in greedy_search_proxy
    return jacobi_greedy_search_multilevel(self, chat=False, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/workspace/LookaheadDecoding/lade/decoding.py", line 988, in jacobi_greedy_search_multilevel
    outputs = self.jforward_multilevel(
              ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/workspace/LookaheadDecoding/lade/models/modeling_llama.py", line 1519, in jforward_multilevel
    outputs = self.model.LlamaModeljforward(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/workspace/LookaheadDecoding/lade/models/modeling_llama.py", line 1224, in LlamaModeljforward
    layer_outputs = decoder_layer(
                    ^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/lade/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/lade/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/workspace/LookaheadDecoding/lade/models/modeling_llama.py", line 864, in forward
    hidden_states, self_attn_weights, present_key_value = self.self_attn.forward(
                                                          ^^^^^^^^^^^^^^^^^^^^^^^
TypeError: LlamaSdpaAttention.forward() got an unexpected keyword argument 'lookahead'

how solve this?

Activity

  1. jivanph commented on May 16, 2024

    @jivanph

    I am having the same issue.

  2. david-wei-01001 commented on May 17, 2024

    @david-wei-01001

    I am having the same issue

  3. david-wei-01001 commented on May 17, 2024

    @david-wei-01001

    But mine is having an unexpected "padding_mask"
    TypeError: LlamaSdpaAttention.forward() got an unexpected keyword argument 'padding_mask'

  4. jivanph commented on May 17, 2024

    @jivanph

    I solved this by downgrading my version of transformers. I believe version 4.36.2 (https://pypi.org/project/transformers/4.36.2/) worked for me.

  5. david-wei-01001 commented on May 18, 2024

    @david-wei-01001

    Yeah downgrade transformer to version 4.36.2 work for me as well
    So I guess this is resolved and the solution is to downgrade transformer to version 4.36.2

    The team can close this haha
    Thank you very much jivanph

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions