crawl4ai Dify Data Source Plugin

Disclaimer: All code in this repository is AI-generated and has not been human-reviewed. It is provided as-is, without warranty of any kind. Review the source and test it thoroughly before using in production.

A Dify data source plugin that ingests web content into the knowledge pipeline using a self-hosted crawl4ai server. Starting from a seed URL, it follows internal links to crawl many pages and feeds each page's markdown into Dify.

Repository layout

.
├── plugin/          # Dify plugin source (package this directory)
│   ├── manifest.yaml
│   ├── main.py
│   ├── provider/
│   ├── datasources/
│   ├── _assets/
│   ├── PRIVACY.md
│   └── README.md    # usage & configuration docs
├── keys/
│   └── crawl4ai_plugin_key.public.pem   # public key, committed for verification
├── .forgejo/
│   └── workflows/build-sign-verify.yml  # CI: package → sign → verify → release
└── LICENSE

The signing private key is never stored in the repo. It lives in the Forgejo secret PRIVATE_KEY and is injected only during CI.

Requirements

• Dify >= 1.9.0
• A self-hosted crawl4ai 0.9.x server (see Self-Hosting Guide)

See plugin/README.md for full usage and configuration.

Install

Download the latest signed package (crawl4ai.signed.difypkg) from the Releases page and install it in Dify. The signature can be verified against the public key in keys/:

dify signature verify crawl4ai.signed.difypkg -p keys/crawl4ai_plugin_key.public.pem

Build, sign & verify

Push a v* tag and Forgejo Actions packages, signs and verifies the plugin, then attaches the signed package to a Release:

git tag v0.1.0
git push --tags

Manual local steps (requires the Dify CLI, pip install dify-plugin):

dify plugin package ./plugin -o crawl4ai.difypkg
dify signature sign crawl4ai.difypkg -p <private-key.pem>   # produces crawl4ai.signed.difypkg
dify signature verify crawl4ai.signed.difypkg -p keys/crawl4ai_plugin_key.public.pem

License

MIT