lime.lime_image.LimeImageExplainer.explain_instance is causing memory leakage

Deployment
1

I’m deploying an image classifier and adding a classification explanation using the lime.lime_image package.

I’m able to build the model, make a prediction, and create an explanation a few times, but the 2nd or 3rd time (depending on my number of samples for the explanation) I try to predict and explain and image, the server crashes. I’m 95% sure it’s because I’m running out of resources.

No information should be stored from the LimeImageExplainer should be stored between runs, but according to the Manage App sidebar output, it always crashes during the explain_instance() method while it’s running the samples to create the explanation. I know this because it creates an output progress bar as it runs the samples, so I can see that’s when the server runs out of resources.

Again, this only happens after a certain number of samples are run in total, even if they are spread across multiple instances of the model and LimeImageExplainer. I delete the explainer object after each run, but something is being left behind.

I’ve tried deleting every object and variable created by the function after each run, but still it seems to be failing to return the memory to the OS and eventually just eats it all up.

Here is my code for that particular function.

def explain_prediction(model, image):
    """Create an image an a mask to explain model prediction"""
    image = model.convert_image(image)
    try:
        explainer = LimeImageExplainer()
        exp = explainer.explain_instance(image[0],
                                        model.model.predict,
                                        num_samples=50)

        image, mask = exp.get_image_and_mask(0,
                                            positive_only=False, 
                                            negative_only=False,
                                            hide_rest=False,
                                            min_weight=0,
                                            progress_bar=True
                                        )
        explanation = mark_boundaries(image, mask)
        del explainer
        del exp
        del image
        del mask
        return explanation
    except AttributeError:
        print('Model not fit yet')`

I’m tried searching on Google for anything about this method leaking data, but I can’t find anything. Any advice would be appreciated!

2

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.