I try to run scrapy with this os.system command:
cmd =‘scrapy crawl gather_details -a domain=’+ search_text + ’ -o emails.json’
os.system(cmd)
If i running on Local computer it work.
Can i run scapy with this command in share.Streamlite
sh: 1: scrapy: not found
2021-04-19 15:20:38.233 Uncaught app exception
Hi @estkae, welcome to the Streamlit community!
In your code repo, do you have a requirements.txt file that includes scrapy? When things work locally but not on sharing, it’s usually because people forgot to add a dependency.
Best,
Randy
Thanks for answer: no i did:
streamlit==0.80.0
requests==2.24.0
Scrapy==2.5.0
pandas==1.2.3
tld==0.12.5
selenium==3.141.0
parsel==1.6.0
neattext==0.1.0
There are a Problem in streamlit.
After the instellation of scrapy we have this issue:
Collecting scrapy==2.5.0
Downloading Scrapy-2.5.0-py2.py3-none-any.whl (254 kB)
After them we can not find scrapy
Traceback (most recent call last):
File “gather_details.py”, line 2, in
import scrapy
ModuleNotFoundError: No module named ‘scrapy’
import scrapy
import re
from tld import get_tld
from scrapy.crawler import CrawlerProcess
streamlit==0.80.0
requests==2.24.0
scrapy
pandas==1.2.3
tld==0.12.5
selenium==3.141.0
parsel==1.6.0
neattext==0.1.0
Can you post your code repo?
see above
I just forked the code and installed it on sharing, and it appears to be working
Successfully installed Automat-20.2.0 MarkupSafe-1.1.1 PyDispatcher-2.0.5 Send2Trash-1.5.0 Twisted-21.2.0 altair-4.1.0 argon2-cffi-20.1.0 astor-0.8.1 async-generator-1.10 attrs-20.3.0 backcall-0.2.0 base58-2.1.0 bleach-3.3.0 blinker-1.4 cachetools-4.2.1 certifi-2020.12.5 cffi-1.14.5 chardet-3.0.4 click-7.1.2 constantly-15.1.0 cryptography-3.4.7 cssselect-1.1.0 decorator-5.0.7 defusedxml-0.7.1 entrypoints-0.3 gitdb-4.0.7 gitpython-3.1.14 h2-3.2.0 hpack-3.0.0 hyperframe-5.2.0 hyperlink-21.0.0 idna-2.10 importlib-metadata-4.0.0 incremental-21.3.0 ipykernel-5.5.3 ipython-7.22.0 ipython-genutils-0.2.0 ipywidgets-7.6.3 itemadapter-0.2.0 itemloaders-1.0.4 jedi-0.18.0 jinja2-2.11.3 jmespath-0.10.0 jsonschema-3.2.0 jupyter-client-6.1.12 jupyter-core-4.7.1 jupyterlab-pygments-0.1.2 jupyterlab-widgets-1.0.0 lxml-4.6.3 mistune-0.8.4 nbclient-0.5.3 nbconvert-6.0.7 nbformat-5.1.3 neattext-0.1.0 nest-asyncio-1.5.1 notebook-6.3.0 numpy-1.20.2 packaging-20.9 pandas-1.2.3 pandocfilters-1.4.3 parsel-1.6.0 parso-0.8.2 pexpect-4.8.0 pickleshare-0.7.5 pillow-8.2.0 priority-1.3.0 prometheus-client-0.10.1 prompt-toolkit-3.0.18 protego-0.1.16 protobuf-3.15.8 ptyprocess-0.7.0 pyOpenSSL-20.0.1 pyarrow-3.0.0 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 pydeck-0.6.2 pygments-2.8.1 pyparsing-2.4.7 pyrsistent-0.17.3 python-dateutil-2.8.1 pytz-2021.1 pyzmq-22.0.3 queuelib-1.5.0 requests-2.24.0 scrapy-2.5.0 selenium-3.141.0 service-identity-18.1.0 setuptools-56.0.0 six-1.15.0 smmap-4.0.0 streamlit-0.80.0 terminado-0.9.4 testpath-0.4.4 tld-0.12.5 toml-0.10.2 toolz-0.11.1 tornado-6.1 traitlets-5.0.5 typing-extensions-3.7.4.3 tzlocal-2.1 urllib3-1.25.11 validators-0.18.2 w3lib-1.22.0 watchdog-2.0.2 wcwidth-0.2.5 webencodings-0.5.1 widgetsnbextension-3.5.1 zipp-3.4.1 zope.interface-5.4.0
[manager] Python dependencies were installed from /app/streamlitemail_app/requirements.txt using pip.
[manager] Processed dependencies!
2021-04-20 18:35:44.707 An update to the [server] config option section was detected. To have these changes be reflected, please restart streamlit.
[client] Connecting...
requests-2.24.0 scrapy-2.5.0 selenium-3.141.0 service-identity-18.1.0
Thanks a lot but the Problem is Contact Detail!
Bildschirmfoto 2021-04-21 um 01.47.55|690x276 In this case streamlit doesn’t find scrapy
If I had to guess, I would say that this line here doesn’t run in the same environment as the actual Streamlit app:
By calling a subprocess with an unspecified python executable, I suspect you are running in a system-level Python, as opposed to the Python where your requirements.txt file packages are installed. If so, this isn’t a Streamlit issue per se, but a PATH one.
I’ve opened a PR against your repo, which I believe fixes this issue.
Best,
Randy
Great great you are the best Thank you. iClose all issue.
BR Karl