transformers 5.x AttributeError: BertTokenizer has no attribute _convert_token_to_id. Did you mean: 'convert_tokens_to_ids'?
[ Code ]
import nlpaug.augmenter.word as naw
texts = [
"Test message",
]
aug = naw.ContextualWordEmbsAug(model_path="bert-base-uncased", action="insert")
augmented_texts = aug.augment(texts)
[ Error message ]
Traceback (most recent call last):
File "/home/gangserver/devel/python/nlpaug_test.py", line 7, in <module>
aug = naw.ContextualWordEmbsAug(model_path="bert-base-uncased", action="insert")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/gangserver/devel/python/.venv/lib/python3.12/site-packages/nlpaug/augmenter/word/context_word_embs.py", line 98, in __init__
self.model = self.get_model(
^^^^^^^^^^^^^^^
File "/home/gangserver/devel/python/.venv/lib/python3.12/site-packages/nlpaug/augmenter/word/context_word_embs.py", line 533, in get_model
return init_context_word_embs_model(model_path, model_type, device, force_reload, batch_size, top_k,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/gangserver/devel/python/.venv/lib/python3.12/site-packages/nlpaug/augmenter/word/context_word_embs.py", line 34, in init_context_word_embs_model
model = nml.Bert(model_path, device=device, top_k=top_k, silence=silence, batch_size=batch_size)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/gangserver/devel/python/.venv/lib/python3.12/site-packages/nlpaug/model/lang_models/bert.py", line 34, in __init__
self.mask_id = self.token2id(self.MASK_TOKEN)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/gangserver/devel/python/.venv/lib/python3.12/site-packages/nlpaug/model/lang_models/bert.py", line 61, in token2id
return self.tokenizer._convert_token_to_id(token)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/gangserver/devel/python/.venv/lib/python3.12/site-packages/transformers/tokenization_utils_base.py", line 1293, in __getattr__
raise AttributeError(f"{self.__class__.__name__} has no attribute {key}")
AttributeError: BertTokenizer has no attribute _convert_token_to_id. Did you mean: 'convert_tokens_to_ids'?
[ Solution ]
Uninstall transformers version 5.x and install transformers version 4.57.6.
1. Uninstall transformers 5.x version
$ pip uninstall transformers
2. Install transformers version 4.57.6
$ pip install --upgrade transformers==4.57.6
[ Execution log ]
$ pip uninstall transformers
Found existing installation: transformers 5.3.0
Uninstalling transformers-5.3.0:
Would remove:
/home/gangserver/devel/pysrc/torch_gpu/.venv/bin/transformers
/home/gangserver/devel/pysrc/torch_gpu/.venv/lib/python3.12/site-packages/transformers-5.3.0.dist-info/*
/home/gangserver/devel/pysrc/torch_gpu/.venv/lib/python3.12/site-packages/transformers/*
Proceed (Y/n)? y
Successfully uninstalled transformers-5.3.0
$ pip install --upgrade transformers==4.57.6
Collecting transformers==4.57.6
Downloading transformers-4.57.6-py3-none-any.whl.metadata (43 kB)
(... Log omitted ...)
Downloading transformers-4.57.6-py3-none-any.whl (12.0 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 12.0/12.0 MB 2.0 MB/s 0:00:05
Downloading huggingface_hub-0.36.2-py3-none-any.whl (566 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 566.4/566.4 kB 2.4 MB/s 0:00:00
Installing collected packages: huggingface-hub, transformers
Attempting uninstall: huggingface-hub
Found existing installation: huggingface_hub 1.8.0
Uninstalling huggingface_hub-1.8.0:
Successfully uninstalled huggingface_hub-1.8.0
Successfully installed huggingface-hub-0.36.2 transformers-4.57.6
* Once the installation is complete after deletion, the module version changes as follows.
transformers : 5.3.0 => 4.57.6
huggingface_hub : 1.8.0 => 0.36.2