Web Scraper App for Automated Testing

Show the Community!
,
1

:link: [Try the app here!]

Hello Streamlit Community! I’m excited to share my latest project with you—a web scraper and testing tool designed specifically for automated testing . This app leverages multiple technologies to deliver a comprehensive solution for testing and extracting data from various search functionalities on the site.

on GitHub!
:star2: Give the a project star on GitHub to help others discover it.

I hope this project inspires ideas and sparks discussions on enhancing testing capabilities with Streamlit and web automation tools. Looking forward to your feedback!

Technology Stack

  • :link: Selenium WebDriver: Automates interactions with the website to run tests and gather data.
  • :stew: BeautifulSoup: Parses HTML data for efficient extraction of targeted information.
  • :toolbox: Pandas: Facilitates data manipulation and organization.
  • :parrot: Streamlit: Provides an interactive frontend for smooth and user-friendly operation.
  • :whale2: Docker: Creates a consistent environment for both development and production.
  • :white_check_mark: Pytest: Runs unit tests to maintain quality and reliability.

Project Overview

The Web Scraper App performs automated tests across several key search categories on YellowPages.com, such as:

  • Find People
  • Restaurants
  • Dentists
  • Plumbers
  • Contractors
  • Electricians
  • Auto Repair
  • Roofing
  • Attorneys
  • Hotels

Each test ensures that the website’s search functionality is working correctly and reliably across these categories.

Key Features

  • Exportable Results: The app exports all test results as a CSV file, making it easy to analyze and keep track of data over time.
2

Cool app concept. Came up with the below error:

Traceback (most recent call last):
  File "/mount/src/web-scraper/scraper.py", line 1, in <module>
    from selenium import webdriver
ModuleNotFoundError: No module named 'selenium'
3

Very thankfully, it seeme issuse relater to chorome web driver in linux, you can clone and run it with docker in your local system

4

Looking great! Are you planning to integrate web scraping AI models from Hugging Face for a more robust solution?