scrapyard

自动网页抓取和 RSS 生成库

快速入门

通过创建事件循环开始。

#[tokio::main]
async fn main() {
    // initialise values
    scrapyard::init(None).await;

    // load feeds from a config file
    // or create a default config file
    let feeds_path = PathBuf::from("feeds.json");
    let feeds = Feeds::load_json(&feeds_path).await
    .unwrap_or_else(|| {
        let default = Feeds::default();
        default.save_json();
        default
    });

    // start the event loop, this will not block
    feeds.start_loop().await;

    // as long as the program is running
    // the feeds will be updated regularly
    HttpServer::new(|| {})
        .bind(("0.0.0.0", 8080)).unwrap()
        .run().await.unwrap();
}

配置

默认情况下，配置文件可以在 ~/.config/scrapyard (Linux), /Users/[Username]/Library/Application/Support/scrapyard (Mac) 或 C:\Users\[Username]\AppData\Roaming\scrapyard (Windows) 中找到。

要更改配置目录位置，指定路径

let config_path = PathBuf::from("/my/special/path");
scrapyard::init(Some(config_path)).await;

以下是主配置文件 scrapyard.json 中的所有选项。

{
    "store": String, // i.e. /home/user/.local/share/scrapyard/
    "max-retries": Number, // number of retries before giving up
    "request-timeout": Number, // number of seconds before giving up request
    "script-timeout": Number, // number of seconds before giving up on the extractor script
}

添加源

要添加源，编辑 feeds.json。

{
    "origin": String, // origin of the feed
    "label": String, // text id of the feed
    "max-length": Number, // maximum number of items allowed in the feed
    "fetch-length": Number, // maximum number of items allowed to be fetched each interval
    "interval": Number, // number of seconds between fetching,
    "idle-limit": Number, // number of seconds without requests to that feed before fetching stops
    "sort": Boolean, // to sort by publish date or not
    "extractor": [String], // all command line args to run the extractor, i.e. ["node", "extractor.js"]

    "title": String, // displayed feed title
    "link": String, // displayed feed source url
    "description": String, // displayed feed description
    "fetch": Boolean // should the crate fetch the content, or let the script do it
}

您还可以在 PseudoChannel 中包含其他字段以覆盖默认空值。

获取源

参考 FeedOption 下的函数，有 2 种获取函数。

强制获取 总是请求新的源副本，忽略获取间隔。 懒获取 只有在现有副本过时时才会获取新的副本。这在没有使用自动获取循环时尤其相关。

提取脚本

提取脚本必须接受 1 个命令行参数，并向 stdout 打印 1 个 JSON 响应，普通的 console.log() 在 JS 中将这样做。您应该明白了。

第一个参数将指定文件路径，在该文件中包含爬虫的参数。

命令行输入

{
    "url": String, // origin of the info fetched
    "webstr": String?, // response from the url, only if feed.fetch = true
    "preexists": [ PseudoItem ], // don't output these again to avoid duplication
    "lengthLeft": Number // maximum length before the fetch-length quota is met

    // plus everything from feed.json
}

预期输出

{
    "items": [PseudoItem], // list of items extracted
    "continuation": String? // optionally continue fetching in the next url
}

许可协议：AGPL-3.0