Newbie question on local model loading
#8
by chaltik - opened
I have downloaded the file llama-2-7b-chat.ggmlv3.q4_K_S.bin and placed it in the folder ../models/llama-2-7b-chat.ggmlv3.q4_K_S.
Calling AutoModel.from_pretrained('../models/llama-2-7b-chat.ggmlv3.q4_K_S') gives the error about not finding the file named pytorch_model.bin`.
Upon renaming the .bin file to such name I get this error:
OSError: Unable to load weights from pytorch checkpoint file for '../models/llama-2-7b-chat.ggmlv3.q4_K_S/pytorch_model.bin' at '../models/llama-2-7b-chat.ggmlv3.q4_K_S/pytorch_model.bin'. If you tried to load a PyTorch model from a TF 2.0 checkpoint, please set from_tf=True.
what is the correct way to load the model from binaries?
thanks very much!
You can load up the model by just referencing the directory on GGML models using c transformers.
from ctransformers import AutoModelForCausalLM
llm = AutoModelForCausalLM.from_pretrained('models/', model_type='gpt2')
print(llm('AI is going to'))
Loading directly from huggingface doesn't seem to work either. The mysterious error keeps suggesting using from_tf=True even when I have already used it there:
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id="TheBloke/Llama-2-7B-Chat-GGML".lower()
model =AutoModelForCausalLM.from_pretrained(model_id, from_tf=True)
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ in <module>:5 │
│ │
│ 2 │
│ 3 model_id="TheBloke/Llama-2-7B-Chat-GGML".lower() │
│ 4 │
│ ❱ 5 model =AutoModelForCausalLM.from_pretrained(model_id, from_tf=True) │
│ 6 │
│ │
│ /home/ec2-user/SageMaker/envs/py310/lib/python3.10/site-packages/transformers/models/auto/auto_f │
│ actory.py:493 in from_pretrained │
│ │
│ 490 │ │ │ ) │
│ 491 │ │ elif type(config) in cls._model_mapping.keys(): │
│ 492 │ │ │ model_class = _get_model_class(config, cls._model_mapping) │
│ ❱ 493 │ │ │ return model_class.from_pretrained( │
│ 494 │ │ │ │ pretrained_model_name_or_path, *model_args, config=config, **hub_kwargs, │
│ 495 │ │ │ ) │
│ 496 │ │ raise ValueError( │
│ │
│ /home/ec2-user/SageMaker/envs/py310/lib/python3.10/site-packages/transformers/modeling_utils.py: │
│ 2560 in from_pretrained │
│ │
│ 2557 │ │ │ │ │ │ │ "use_auth_token": token, │
│ 2558 │ │ │ │ │ │ } │
│ 2559 │ │ │ │ │ │ if has_file(pretrained_model_name_or_path, TF2_WEIGHTS_NAME, **h │
│ ❱ 2560 │ │ │ │ │ │ │ raise EnvironmentError( │
│ 2561 │ │ │ │ │ │ │ │ f"{pretrained_model_name_or_path} does not appear to hav │
│ 2562 │ │ │ │ │ │ │ │ f" {_add_variant(WEIGHTS_NAME, variant)} but there is a │
│ 2563 │ │ │ │ │ │ │ │ " Use `from_tf=True` to load this model from those weigh │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
OSError: thebloke/llama-2-7b-chat-ggml does not appear to have a file named pytorch_model.bin but there is a file
for TensorFlow weights. Use `from_tf=True` to load this model from those weights.
Thanks!
chaltik changed discussion status to closed