Error loading model or tokenizer: CamembertTokenizer requires the SentencePiece library but it was not found in your environment

Using Streamlit
,
1

App deployed on Community Cloud: https://levelingo.streamlit.app/
Link to app’s GitHub repo: text_difficulty_prediction/app at main · vgentile98/text_difficulty_prediction · GitHub

Error message:
Error loading model or tokenizer: CamembertTokenizer requires the SentencePiece library but it was not found in your environment. Checkout the instructions on the installation page of its repo: GitHub - google/sentencepiece: Unsupervised text tokenizer for Neural Network-based text generation. and follow the ones that match your environment. Please note that you may need to restart your runtime after installation.

I’ve added the following packages to my packages.txt file:

build-essential
cmake
pkg-config
libprotobuf-dev
libgoogle-perftools-dev
protobuf-compiler

I’ve added the following Python dependencies to my requirements.txt file:

streamlit
requests
torch
tokenizers
transformers
transformers[sentencepiece]
sentencepiece
newsapi-python

And imported the following libraries in my streamlit_app.py file:

import streamlit as st
import requests
import os
import transformers
import sentencepiece 
try:
    import sentencepiece as spm
    st.success('SentencePiece is successfully imported!')
except ImportError as e:
    st.error(f'Failed to import SentencePiece: {e}')
import torch
from transformers import CamembertTokenizer, CamembertForSequenceClassification, pipeline
import tokenizers
import streamlit.components.v1 as components

When I check whether the ‘sentencepiece’ library has been successfully installed in my Streamlit app, the response is positive. I cannot figure out why I still get this error message stating that the SentencePiece library was not found in my environment.

Thanks a lot for your help!!

Victoria

2

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.