Spotlight on GPT Crawler

Overview

GPT-Crawler is a Node.js-based tool that crawls websites to generate knowledge files for creating custom GPTs and AI assistants. It features configurable crawling parameters and supports multiple output formats.

Key Features

• Configurable URL and selector patterns
• Resource exclusion options
• File size and token limit controls
• Docker container support
• API server functionality
• Multiple deployment options

Usage Methods

• Local installation with Node.js
• Containerized execution via Docker
• API server deployment

Implementation

1. Configure crawler settings in config.ts
2. Run the crawler to generate output.json
3. Upload generated file to OpenAI
4. Create custom GPT or assistant using the knowledge file