Enable HTTP destination

Cloudflare Logpush now supports the ability to send logs to configurable HTTP endpoints.

Note that when using Logpush to HTTP endpoints, Cloudflare customers are expected to perform their own authentication of the pushed logs. For example, customers may specify a secret token in the URL or an HTTP header of the Logpush destination.

Manage via the Cloudflare dashboard

1. Log in to the Cloudflare dashboard ↗.

2. Select the Enterprise account or domain (also known as zone) you want to use with Logpush. Depending on your choice, you have access to account-scoped datasets and zone-scoped datasets, respectively.

3. Go to Analytics & Logs > Logpush.

4. Select Create a Logpush job.

5. In Select a destination, choose HTTP destination.

6. Enter the HTTP endpoint where you want to send the logs to, and select Continue.

7. Select the dataset to push to the storage service.

8. In the next step, you need to configure your logpush job:

   • Enter the Job name.
   • Under If logs match, you can select the events to include and/or remove from your logs. Refer to Filters for more information. Not all datasets have this option available.
   • In Send the following fields, you can choose to either push all logs to your storage destination or selectively choose which logs you want to push.

9. In Advanced Options, you can:

   • Choose the format of timestamp fields in your logs (RFC3339(default),Unix, or UnixNano).
   • Select a sampling rate for your logs or push a randomly-sampled percentage of logs.
   • Enable redaction for CVE-2021-44228. This option will replace every occurrence of ${ with x{.

10. Select Submit once you are done configuring your logpush job.

Manage via API

To create a Logpush job, make a POST request to the Logpush job creation endpoint URL with the appropriate parameters.

The supported parameters are as follows:

• Fields that are unchanged from other sources:
  • dataset (required): For example, http_requests.
  • name (optional): We suggest using your domain name as the job name.
  • output_options (optional): Refer to Log Output Options to configure fields, sample rate, and timestamp format.
• Unique fields:
  • destination_conf: Where to send the logs. This consists of an endpoint URL and HTTP headers used.
    • Any "header_*" URL parameters will be used to set request headers.
      • The HTTPS endpoint cannot have custom URL parameters that conflicts with any "header_*" URL parameters you have set.
      • These parameters must be properly URL-encoded (that is, use "%20" for a whitespace), otherwise some special characters may be decoded incorrectly.
    • destination_conf may have more URL parameters in addition to special "header_*" parameters.
      • Non URL-encoded special characters will be encoded when uploading.
    • Example: https://logs.example.com?header_Authorization=Basic%20REDACTED&tags=host:theburritobot.com,dataset:http_requests
  • max_upload_bytes (optional): The maximum uncompressed file size of a batch of logs. This setting value must be between 5 MB and 1 GB. Note that you cannot set a minimum file size; this means that log files may be much smaller than this batch size.
  • max_upload_records (optional): The maximum number of log lines per batch. This setting must be between 1,000 and 1,000,000 lines. Note that you cannot to specify a minimum number of log lines per batch; this means that log files may contain many fewer lines than this.

Note

The ownership_challenge parameter is not required to create a Logpush job to an HTTP endpoint. You need to make sure that the file upload to validate the destination accepts a gzipped test.txt.gz with content as {"content":"tests"} compressed, otherwise it will return an error, like error validating destination: error writing object: error uploading.

Example curl request

curl https://api.cloudflare.com/client/v4/zones/{zone_id}/logpush/jobs \
--header "X-Auth-Email: <EMAIL>" \
--header "X-Auth-Key: <API_KEY>" \
--header "Content-Type: application/json" \
--data '{
  "name": "theburritobot.com-https",
  "output_options": {
    "field_names": ["EdgeStartTimestamp", "RayID"],
    "timestamp_format": "rfc3339"
  },
  "destination_conf": "https://logs.example.com?header_Authorization=Basic%20REDACTED&tags=host:theburritobot.com,dataset:http_requests",
  "max_upload_bytes": 5000000,
  "max_upload_records": 1000,
  "dataset": "http_requests",
  "enabled": true
}'