Deployment on Azure (AppService) + AD Authentication enabled + st.file_uploader = 403 Forbidden

šŸš€ Deployment
1

Hello,

Iā€™m using a production workload of AppService so all issues with regards to sockets and getting stuck on ā€˜connectingā€¦ā€™ (described here: https://discuss.streamlit.io/t/azure-appservice-with-authentication-issues) are fine, app loads correctly. Deployment is on a linux docker image hosted on ACR (Azure Container Registry).

However when a file gets uploaded into the st.file_uploader widget, the POST http request returns a 403 Forbidden.

image
image887Ɨ89 1.84 KB

When AD Authentication is disabled, file uploader works fine.

Any one else experiencing this?

2

Hey @JoeNSalloum -

I donā€™t know anything about Azure AD specifically, but I wonder if itā€™s an issue that using file_uploader sends it through a different port, and the browser isnā€™t able to sign the requests (since we havenā€™t incorporated anything like that).

I donā€™t know, just thinking out loud. But it sounds like thereā€™s a step missing to authenticate with AD Authentication.

Best,
Randy

3

Hi @randyzwitch ,

Thanks for your reply. The browser is sending the file using the upload_file enpoint, through the standard https port 443:

image

both healthz and stream endpoint work properly, as you can see:

image
image792Ɨ79 2.3 KB

Itā€™s only the upload_file.

Is there a log file I can inspect on the container itself that can potentially show me whatā€™s going on under the hood? Verbose would be ideal.

4

Thanks for confirming the port.

I wonder if either of these links are saying what Iā€™m thinking:

When youā€™re using file_uploader from the browser, I wonder if the issue is that Streamlit doesnā€™t have the AD token that you need. Similarly, I wonder if it would work with anonymous requests (second link).

5

Hi @randyzwitch,

Yes that was my suspicion as well, will look into adjusting the service principal, will let you know if it makes a difference. Although if that were the case I would expect all endpoints to be problematic, including healthz and stream; otherwise thereā€™s something special about what upload_file is doing on the backend.

As for anonymous requests, yes it does work as I said in my original thread.

6

Unfortunately Iā€™m also unfamiliar with AD authentication but one thing the upload_file endpoint does differently is requires a XSRF token. You could try disabling server.enableXsrfProtection to False to see if thatā€™s the issue.

7

Just to add on regarding this, you can always change your log level config option with logger.level. The available options are error, warning, info, or debug.

8

Hi @JoeNSalloum, @randyzwitch

I deployed my streamlit app to azure and I am facing similar issue with st.file_uploader(). This is not consistent. Sometimes the file uploads properly and move to next action and most of the time it doesnā€™t work as expected. Could you be able to figure out the resolution for it.?

Do I need to make any specific configuration on Azure machine? or this is an issue with stremlit?

server.enableXsrfProtection is also set to False.

Expected:

image
image690Ɨ662 80.6 KB

But on Server:
image
image806Ɨ786 30.1 KB

Kindly share if there is any progress.

9

Hi Chakra,

Unfortunately I havenā€™t found a resolution to this yet, even the latest file_uploader component suffers from the same problem. There seems to be something specific about how this component works thatā€™s different from the rest of the streamlit because with AD authentication enabled itā€™s the only one causing an issue.

1 Like
10

Hi @JoeNSalloum Thanks for the reply. The issue got resolved for me.

Here is the observations on my case:
While testing the application in the local docker container. It all worked fine for me. The reason being in the local it allocates only one pod for running the entire application. All uploads being referenced within the same pod.
Whereas; On Azure machine - Based on my replica setting, it created 4 pods on the server and the load balancer doesnā€™t properly handle the calls across pods. So I changed the number of replicas to 1 and the issue is no more. It got fixed. I am using almost 10 file_uploader() in the app and all works fine for me.
I am not a devops person :slight_smile: but it worked for me with the help of my devops engineer.

1 Like
11

Hi Chakra,

Thanks for the info, so it looks like youā€™re using AKS (Azure Kubernetes Service) as opposed to AppService pulling an image from ACR (Azure Container Registry) and spinning up a single (or multiple - if you scale out) container. In my case I solved the problem of multiple nodes by setting what Azure calls the ARR Affinity cookie; essentially a way to guarantee stateful applications to always hit the same node on the backend in a multi-node setup.

I tried your suggestion and brought down the number of nodes/VMs to 1 - same issue, but I thank you for taking the time to share your experience with everyone else :slight_smile:

1 Like