Prometheus Web Exporter

Prometheus Web Exporter 是一个Prometheus导出器，用于收集关于网页的信息。它可以收集两种不同类型的信息。网络健康信息（响应时间和响应体大小）和基于内容的信息（与给定CSS查询匹配的页面元素数量）。

Web导出器使用与Firefox浏览器引擎（Servo）相同的技术来解析和运行CSS查询。因此，它相当快速，支持广泛的CSS查询。不幸的是，它不支持JavaScript，因此使用JavaScript进行的任何DOM操作都不会被Web导出器捕获。（不支持SPA。）

配置

配置使用名为web_exporter.yaml的文件进行，该文件位于可执行文件相同的目录中。以下是一个web_exporter.yaml文件的示例

# ip address server will listen. default: 0.0.0.0
# ip_address: "0.0.0.0"
#
# port that servier will listen. default: 3030
# port: 3030
#
# metrics path. default: /metrics
# metrics_path: "metrics"
#
# targets to crawl for each request.
targets:
  # 200 response with queries
  - url: "https://www.rust-lang.net.cn/"
    queries:
      - "#language-values div.flex-none section"
      - "header h1"
      - "footer div.attribution"
    extra_labels:
      name: homepage
  # 404 response with queries
  - url: "https://www.rust-lang.net.cn/invalid-page-with-404-response"
    headers:
      Referer: "https://www.rust-lang.net.cn/"
    queries:
      - "div.flex"
      - "div"
    extra_labels:
      name: 404 page
  # Network error. (Queries will not return any value since they will not be running.)
  - url: "https://www.page-does-not-exist.io/"
    queries:
      - "div"
    extra_labels:
      name: nonexistent_page
  # Invalid query (return value will be 0 and css query parse error will be logged.)
  - url: "https://www.rust-lang.net.cn/invalid-css-query"
    queries:
      - "**XX**"
    extra_labels:
      name: query_with_invalid_css
  # 200 page without any query (only response time and size will be returned.)
  - url: "https://www.rust-lang.net.cn/no-css-query"
  # google search test example with queryparameters and extra headers.
  - url: "https://www.google.com/search"
    queryparameters:
      q: rust
    headers:
      referer: "https://www.google.com/"
      user-agent: "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36"
    queries:
      - "div.g"
    extra_labels:
      name: google_search

上述配置将生成以下指标

web_exporter_response_duration_milliseconds{url="https://www.rust-lang.net.cn/", method="GET", error="false", name="homepage", status="200" } 787
web_exporter_response_response_size_bytes{url="https://www.rust-lang.net.cn/", method="GET", error="false", name="homepage", status="200" } 19220
web_exporter_query_count{url="https://www.rust-lang.net.cn/", method="GET", error="false", name="homepage", status="200", query="#language-values div.flex-none section" } 3
web_exporter_query_count{url="https://www.rust-lang.net.cn/", method="GET", error="false", name="homepage", status="200", query="header h1" } 1
web_exporter_query_count{url="https://www.rust-lang.net.cn/", method="GET", error="false", name="homepage", status="200", query="footer div.attribution" } 1
web_exporter_response_duration_milliseconds{error="false", url="https://www.rust-lang.net.cn/invalid-page-with-404-response", status="404", name="404 page", method="GET" } 142
web_exporter_response_response_size_bytes{error="false", url="https://www.rust-lang.net.cn/invalid-page-with-404-response", status="404", name="404 page", method="GET" } 8244
web_exporter_query_count{error="false", url="https://www.rust-lang.net.cn/invalid-page-with-404-response", status="404", name="404 page", method="GET", query="div.flex" } 6
web_exporter_query_count{error="false", url="https://www.rust-lang.net.cn/invalid-page-with-404-response", status="404", name="404 page", method="GET", query="div" } 14
web_exporter_response_duration_milliseconds{name="nonexistent_page", status="0", url="https://www.page-does-not-exist.io/", method="GET", error="true" } 83
web_exporter_response_response_size_bytes{name="nonexistent_page", status="0", url="https://www.page-does-not-exist.io/", method="GET", error="true" } 0
web_exporter_response_duration_milliseconds{error="false", name="query_with_invalid_css", method="GET", status="404", url="https://www.rust-lang.net.cn/invalid-css-query" } 110
web_exporter_response_response_size_bytes{error="false", name="query_with_invalid_css", method="GET", status="404", url="https://www.rust-lang.net.cn/invalid-css-query" } 8244
web_exporter_query_count{error="false", name="query_with_invalid_css", method="GET", status="404", url="https://www.rust-lang.net.cn/invalid-css-query", query="**XX**" } 0
web_exporter_response_duration_milliseconds{error="false", status="404", url="https://www.rust-lang.net.cn/no-css-query", method="GET" } 127
web_exporter_response_response_size_bytes{error="false", status="404", url="https://www.rust-lang.net.cn/no-css-query", method="GET" } 8244
web_exporter_response_duration_milliseconds{name="google_search", url="https://www.google.com/search", status="200", method="GET", error="false" } 964
web_exporter_response_response_size_bytes{name="google_search", url="https://www.google.com/search", status="200", method="GET", error="false" } 406579
web_exporter_query_count{name="google_search", url="https://www.google.com/search", status="200", method="GET", error="false", query="div.g" } 11
web_exporter_scrape_duration_milliseconds 972

如何获取它

使用Docker。

# Download sample configuration from github
wget https://raw.githubusercontent.com/huseyinyilmaz/web_exporter/master/sample_web_exporter.yaml
# Rename it to web_exporter.yaml
mv sample_web_exporter.yaml web_exporter.yaml
# Run Docker and mount configuration file to container
$ docker run \
       --rm -ti -d \
       -p 3030:3030 \
       --name web_exporter \
       -v $(pwd)/web_exporter.yaml:/usr/local/prometheus_web_exporter/web_exporter.yaml \
       huseyinyilmaz/web_exporter:v1.0.5
# endpoint should be on https://127.0.0.1:3030/metrics

使用docker-compose

首先，从仓库下载示例配置并将其放在与docker-compose.yaml文件相同的目录中。然后，您可以将以下配置添加到您的docker-compose文件中。

  webexporter:
    image: huseyinyilmaz/web_exporter:v1.0.5
    volumes:
      - ./web_exporter.yaml:/usr/local/prometheus_web_exporter/web_exporter.yaml
    environment:
      WEB_EXPORTER_LOG_LEVEL: warn
    ports:
      - 3030:3030
    expose:
      - 3030

使用Cargo

# First install web_exporter with cargo.
cargo install prometheus_web_exporter
# After installation cargo will show where it is installed it is likely on
# ~/.cargo/bin. Go to installation folder and download sample config next to executable.
cd ~/.cargo/bin
# Download sample configuration from github
wget https://raw.githubusercontent.com/huseyinyilmaz/web_exporter/master/sample_web_exporter.yaml
# Rename it to web_exporter.yaml
mv sample_web_exporter.yaml web_exporter.yaml
# run web_exporter
./prometheus_web_exporter
# endpoint should be on https://127.0.0.1:3030/metrics

使用预构建的二进制文件。

您可以从GitHub上的发行版部分下载适用于您的操作系统的二进制文件。获取二进制文件后，只需将配置放在二进制文件旁边即可。

常见问题解答

如何将其集成到Prometheus中？

  - job_name: 'webexporter'
    scrape_interval: 60s
    scrape_timeout: 50s
    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.
    static_configs:
    - targets: ['webexporter:3030']

如何更改抓取频率？

Web爬虫没有内部缓存，并且每次调用/metrics端点时都会运行抓取。可以通过更改premetheus配置中的scrape_interval值来更改抓取频率。

  - job_name: 'webexporter'
    scrape_interval: 30s # <- Change scrape frequency from here.
    scrape_timeout: 29s
    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.
    static_configs:
    - targets: ['webexporter:3030']

如何更改日志级别？

可以通过环境变量设置日志级别。要使用info日志级别运行项目，您可以像这样运行它

$ WEB_EXPORTER_LOG_LEVEL=info ./prometheus_web_exporter

如果您使用的是Docker镜像，日志级别已设置为WARN。

有一个指标返回带有标签"error": "true"，这是什么意思？

这意味着您正在抓取的网站无法完成HTTP请求，并返回网络错误。这些错误也将由导出器记录到标准输出。如果您使用Docker容器，您应该在Docker日志中看到错误。您可以在那里看到关于错误的详细信息。