stork_http

这是针对HTTP协议和基于HTML的网络抓取的stork实现。给定一个初始抓取页面，stork_http将找到页面上的所有可索引链接并将它们返回给您——随时可以再次抓取或存储以稍后返回，所有这些操作都使用futures以实现并行处理。

目前严格实施rel="nofollow"，无法更改，尽管随着时间的推移，随着更多过滤器的添加，这一点将得到改善。

示例用法

#
// start scanning https://example.com/ for links with the given filters
let stream = HttpStorkable::new("https://example.com/".parse()?)
    .with_filters(
        FilterSet::default()
            .add_filter(DomainFilter::new("www.iana.org"))
            .add_filter(SchemeFilter::new("https"))
    )
    .exec();
// get the first link from example.com and ensure its the one we expected
// it to be
let first_link_on_example: HttpStorkable = match stream.next().await {
    Some(Ok(link)) => {
        assert_eq!(link.val().text(), Some("More information...".to_string()));
        assert_eq!(link.val().url().as_str(), "https://www.iana.org/domains/example");
        assert_eq!(link.parent().unwrap().val().url().as_str(), "https://example.com/");
        link
    },
    _ => panic!("failed to get links from page")
};

// add another filter looking for the root path and start scanning for links
let filters = first_link_on_example.filters().clone()
    .add_filter(PathFilter::new(FilterType::Equals, "/"));
let stream = first_link_on_example
    .with_filters(filters)
    .exec();
// get the first link from the stream and ensure its a link to the homepage
match stream.next().await {
    Some(Ok(link)) => {
        assert_eq!(link.val().url().as_str(), "https://www.iana.org/");
        assert_eq!(link.parent().unwrap().val().url().as_str(), "https://www.iana.org/domains/example");
        assert_eq!(link.parent().unwrap().parent().unwrap().val().url().as_str(), "https://example.com/")
    },
    _ => panic!("failed to get links from page")
}
// ensure theres no other unexpected links on the homepage
assert!(stream.next().await.is_none(), "should've been only one homepage link on the page!");