Welcome to CorgiBrowser’s Documentation

Crawler

Crawler:

• WebCrawler Class
  • class WebCrawler(cloud_integration, settings_manager=None)[source]
  • Parameters
  • Methods
  • SettingsManager Class
  • Methods
  • Modifying Crawler Settings
  • Settings Details
• Introduction
  • Crawling Process
  • Ethical Considerations
  • Technical Implementation
  • Architectural Overview
  • Practical Application Examples
• Crawler Component Documentation
  • Getting Started with the CorgiBrowser Crawler
  • Prerequisites
  • Installation
  • Configuration
  • Sample Code for Crawler Initialization
  • Understanding the Crawler Components
  • Advanced Configuration and Usage

Scraper

Scraper:

• Scraper Class
  • class Scraper(cloud_integration, settings_manager=None, scraper_dict={})[source]
  • Parameters
  • Methods
  • SettingsManager Component
  • Modifying Scraper Settings
  • Settings Details
• Introduction
  • Scraping Process
  • 4 Data Extraction from Template
  • 5 Data Storage in Partitioned Database
  • 6 Data Storage in Selected Directory
  • 7 Metadata Updating for Tracking and Analysis
  • 8 URL Extraction
  • 9 New URL Cleaning
  • 10 Queue Updating
• Scraper Component Documentation
  • Getting Started with the CorgiBrowser Scraper
  • Prerequisites
  • Installation
  • Configuration
  • Sample Code for Scraper Initialization
  • Understanding the Scraper Components
  • Advanced Configuration and Usage

Data Management

Data Management:

• CloudIntegration Class
  • class CloudIntegration(settings_manager)[source]
  • Parameters
  • Methods
  • Modifying Data Management Settings
• Intro
  • Overview of the DataManagement Module
  • Initialization Process
  • Cloud Provider Interface
  • Method Overview
• Table Partitioning
  • CorgiWebEntity
  • Table Partitioning Overview
  • Example of Partitioned Tables
  • CorgiWebEntity Schema
  • Usage in Web Crawling and Scraping
• Queue Partitioning
  • Dedicated Queues Per Domain
  • CorgiWebMessageSchemaVersion1
• CorgiHashTable
  • Class Definition
  • Scraping Process Steps
  • Scraping with Local Hash Cache
• CorgiWebQueuePreference Table
  • Class Definition
  • Usage in Queue Management
  • High-Level Overview
  • Example Configuration
• CorgiWebRequestLog Table
  • Class Definition
  • Usage and Application
• CorgiWebScrapeLog Table
  • Class Definition
  • Purpose and Usage
• CorgiWebThrottling Table
  • Class Definition
  • Throttling Process Overview
  • Usage Scenarios

Data Analysis And AI

Analysis And AI:

• Direction (In development)
  • Data Analysis Process
  • Ideas