Upload Zip - Open Shapefile within in Memory

Richard_Dean · September 2, 2023, 5:13pm

Long-Version
I am refactoring an application from a desktop GUI over to streamlit to be deployed online and I’m coming across an issue that I can’t seem to get my head around. While working on the desktop version of the app, it’s easy to manipulate files on the local machine without issue Share a clear and concise description of the issue. Aim for 2-3 sentences; but for a webapp I need to be able to get the data uploaded and then work on the contained files. So I want to be able to handle a zip file that contains the output from another program (ESRI Shapefiles and Excel files are the ones I’m interested in) do some work with those files and then produce a download that contains the updated files. This is causing me some headaches when I try to do this without extracting the zip file first

Short-Version
I need to open shapefiles and excel files that match certain filenames from within a zip file uploaded to the streamlit app by a user without extracting it first. Expected input is a zip file containing flat structure with various files including shapefiles and excel files (I haven’t got round to using openpyxl yet)

Code

import io
import zipfile
import streamlit as st
from io import BytesIO
import pandas as pd
import geopandas as gpd
from fiona.io import ZipMemoryFile


def read_shapefile_from_zip(zip_file, shapefile_to_open):
    with zipfile.ZipFile(zip_file, 'r') as my_zip:
        for file_list in my_zip.namelist():
            if file_list == shapefile_to_open:
                st.text(f'match {file_list}')
                with my_zip.open(file_list, "r") as data:
                    data = io.BytesIO(data.read())
                    st.text(data)
                    df = gpd.read_file(data)
                    st.dataframe(df)
                    
                    
shape_file = 'data1.shp'

file_uploaded = st.file_uploader(label='Please choose zip file to upload:', type='zip', key='zip_file')
if st.button(label='Run', key='run'):
    read_shapefile_from_zip(file_uploaded, shape_file)

Iv’e also tried this and a few others trying to make this work the way I want

    df = gpd.read_file(f'zip:///{ZipMemoryFile.open(zip_file)}!{shapefile_to_open}')
    st.dataframe(df)

Expecting the zip file to be read and the data dropped into a dataframe (st.dataframe in the function is there to see if it worked)

Actual behavior:

Error on main code is as follows

DriverError: ‘/vsimem/7d7e6cff88ab42ab8334824e09374ff9’ not recognized as a supported file format.

Traceback:

File "/home/richard/PycharmProjects/graph_node_ordering/venv/lib/python3.11/site-packages/streamlit/runtime/scriptrunner/script_runner.py", line 552, in _run_script
    exec(code, module.__dict__)File "/home/richard/PycharmProjects/graph_node_ordering/testing.py", line 87, in <module>
    read_shapefile_from_zip(file_uploaded, shape_file)File "/home/richard/PycharmProjects/graph_node_ordering/testing.py", line 64, in read_shapefile_from_zip
    df = gpd.read_file(data)
         ^^^^^^^^^^^^^^^^^^^File "/home/richard/PycharmProjects/graph_node_ordering/venv/lib/python3.11/site-packages/geopandas/io/file.py", line 281, in _read_file
    return _read_file_fiona(
           ^^^^^^^^^^^^^^^^^File "/home/richard/PycharmProjects/graph_node_ordering/venv/lib/python3.11/site-packages/geopandas/io/file.py", line 322, in _read_file_fiona
    with reader(path_or_bytes, **kwargs) as features:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^File "/home/richard/PycharmProjects/graph_node_ordering/venv/lib/python3.11/site-packages/fiona/collection.py", line 783, in __init__
    super().__init__(self.virtual_file, vsi=filetype, **kwds)File "/home/richard/PycharmProjects/graph_node_ordering/venv/lib/python3.11/site-packages/fiona/collection.py", line 243, in __init__
    self.session.start(self, **kwargs)File "fiona/ogrext.pyx", line 588, in fiona.ogrext.Session.startFile "fiona/ogrext.pyx", line 143, in fiona.ogrext.gdal_open_vector

So anyone have experience with this sort of thing that can help me get to the bottom of this before I just extract the files and delete after processing?

dataprofessor · September 3, 2023, 2:12am

Hi @Richard_Dean

There seems to be a similar issue on Stack Overflow that may provide some directions to look into.

github.com/Toblerity/Fiona

`fiona.listlayers` doesn't work on `MemoryFile` or `ZipMemoryFile` (but does on `'/vsizip/' + zipmemfile.name`)

opened 07:31AM - 27 Apr 22 UTC

closed 10:45PM - 13 Dec 22 UTC

bayersglassey-zesty

## Expected behavior and actual behavior. I expect to be able to call `fiona.…listlayers` on a `MemoryFile` or `ZipMemoryFile` without error. However, I get a DriverError with message like "No such file or directory" and "'/vsimem/e3e7971d814b4e6d992b9c7b4a9a3906/e3e7971d814b4e6d992b9c7b4a9a3906' not recognized as a supported file format". (That said, I *can* successfully call `fiona.listlayers('/vsizip/' + zipmemfile.name)`!.. but it seems hacky, so I don't feel like it's a trustworthy workaround...) ## Steps to reproduce the problem. ```python from io import BytesIO import requests import fiona # Link to a "WFIGS - 2022 Wildland Fire Perimeters to Date" dataset in # shapefile (.zip) format. # An HTTP GET to this URL should return a .zip file, currently (as of Apr 22, # 2022) around 4MB. # # A description of this data set is at: # # https://hub.arcgis.com/datasets/ # nifc::wfigs-2022-wildland-fire-perimeters-to-date/ # # The URL we use here is the one you get on that page by clicking # Full Details -> Download -> Shapefile. DATA_URL = ( "http://opendata.arcgis.com" "/api/v3/datasets/6bd5dd4e93154f6fae0de1f2c82d95bf_0/downloads/data" "?format=shp&spatialRefId=4326") response = requests.get(DATA_URL) response.raise_for_status() data = response.content ################################################### # Testing MemoryFile: memfile = fiona.io.MemoryFile(BytesIO(data)) # Something like: /vsimem/c1b4dde348c14064880a949f17a0ead6/c1b4dde348c14064880a949f17a0ead6 print(memfile.name) # Something like: # File ~/repos/zestyai/data-ingestion/env/lib/python3.8/site-packages/fiona/__init__.py:342, in listlayers(fp, vfs) # 339 if hasattr(fp, 'read'): # 341 with MemoryFile(fp.read()) as memfile: # --> 342 return _listlayers(memfile.name) # 344 else: # 346 if isinstance(fp, Path): # File fiona/ogrext.pyx:1659, in fiona.ogrext._listlayers() # File fiona/_shim.pyx:90, in fiona._shim.gdal_open_vector() # DriverError: '/vsimem/e8c5da047a1f4af084c35d7acf15c786/e8c5da047a1f4af084c35d7acf15c786' not recognized as a supported file format. print(fiona.listlayers(memfile)) # Something like: # File ~/repos/zestyai/data-ingestion/env/lib/python3.8/site-packages/fiona/__init__.py:362, in listlayers(fp, vfs) # 359 else: # 360 pobj = parse_path(fp) # --> 362 return _listlayers(vsi_path(pobj)) # File fiona/ogrext.pyx:1659, in fiona.ogrext._listlayers() # File fiona/_shim.pyx:90, in fiona._shim.gdal_open_vector() # DriverError: '/vsimem/c1b4dde348c14064880a949f17a0ead6/c1b4dde348c14064880a949f17a0ead6' not recognized as a supported file format. print(fiona.listlayers(memfile.name)) # Something like: # File ~/repos/zestyai/data-ingestion/env/lib/python3.8/site-packages/fiona/__init__.py:362, in listlayers(fp, vfs) # 359 else: # 360 pobj = parse_path(fp) # --> 362 return _listlayers(vsi_path(pobj)) # File fiona/ogrext.pyx:1659, in fiona.ogrext._listlayers() # File fiona/_shim.pyx:90, in fiona._shim.gdal_open_vector() # DriverError: '/vsizip//vsimem/c1b4dde348c14064880a949f17a0ead6/c1b4dde348c14064880a949f17a0ead6' does not exist in the file system, and is not recognized as a supported dataset name. print(fiona.listlayers('/vsizip/' + memfile.name)) ################################################### # Testing ZipMemoryFile: zipmemfile = fiona.io.ZipMemoryFile(BytesIO(data)) # Something like: /vsimem/8b3cf79423e54c2fad6aeaac115bff23/8b3cf79423e54c2fad6aeaac115bff23.zip print(zipmemfile.name) # Same kind of errors as for memfile, see above print(fiona.listlayers(zipmemfile)) print(fiona.listlayers(zipmemfile.name)) # ...interestingly, the following works! # Output is: ['FH_Perimeter'] print(fiona.listlayers('/vsizip/' + zipmemfile.name)) ``` ## Operating system ``` $ uname -a Linux noo-lappy 5.16.11-76051611-generic #202202230823~1646248261~20.04~2b22243~dev-Ubuntu SMP PREEMPT Th x86_64 x86_64 x86_64 GNU/Linux ``` ## Fiona and GDAL version and provenance I don't have `libgdal` installed, just `fiona` via pip. ``` $ dpkg -l | grep -i gdal $ pip freeze | grep -i fiona Fiona==1.8.21 $ ls env/lib/python3.8/site-packages/Fiona.libs/libgdal* env/lib/python3.8/site-packages/Fiona.libs/libgdal-e127aa65.so.30.0.1 ```

Also, have you tried unzipping the files and then working on the target file instead?