The practitioner’s guide to BigCommerce webhooks

Development

Keeping in the know

BigCommerce uses webhooks to communicate events that occur on a store such as new orders or updated products.

A previous era of software required any customisations to be bundled up with the software and deployed as a single artifact. One benefit of such an approach is that you could hook directly into almost anything that happened in the core product. However, the freedom to modify any portion results in assuming responsibility for the whole.

With SaaS services such as BigCommerce, the core product cannot be modified – instead adopting a composition over inheritance approach. The functionality can be tailored to business needs by creating an independent service that interacts with the platform’s APIs.

Now that you can’t get under the hood, SaaS services commonly deploy webhooks which are “push notifications” that communicate when a key operation has occurred so that an external service can act upon it.

What we’ll cover

Our use case

This guide to using webhooks within BigCommerce was created from our own experiences building the Automated Categories app.

Automated Categories Logo

For this app, we want to be able to automatically assign products to relevant categories based on predefined conditions, e.g. add to Sale category when a sale price is set. To avoid the need to regularly poll all products to check for updates, we can ask BigCommerce to send a webhook whenever a product is created or updated. This enables our app to re-evaluate whether a product’s sale price and add to or remove it from the Sale category.

Creating a BigCommerce Webhook

Continuing with our use case as an example, the first task would be to create a webhook requesting a notification of new products is sent to a public URL in our service or app.

We do this by sending a POST request to /v3/hooks API endpoint, e.g. https://api.bigcommerce.com/stores/$STORE_HASH/v3/hooks

All requests to BigCommerce API, including this one, will need to be authenticated by providing an API key in the X-Auth-Token header. If you are creating a single click app this API key (also known as access token) will be provided as part of the authentication flow, alternatively manually create an API token.

The request body defines the event to subscribe to and the URL to send notifications to. { “scope”: “store/product/created”, “destination”: “https://example.com/webhook-listener”, “is_active”: true }

The destination endpoint must be provided over HTTPS.

Note that a separate request is needed for each scope that you wish to register. So, in this case, we would send another one for “store/product/updated”.

Receiving a webhook

When a webhook is triggered, your endpoint will receive a payload of this structure as shown in the official documentation: { “scope”: “store/product/created”, “store_id”: “1234567”, “data”: { “type”: “product”, “id”: 98765 }, “hash”: “dd70c0976e06b67aaf671e73f49dcb79230ebf9d”, “created_at”: 1622708559, “producer”: “stores/{store_hash}” }

  • Scope
    • Matches the scope that was subscribed to when creating the webhook. If you have chosen to specify the same destination for multiple webhooks then this can be used to differentiate which event has occurred.
  • Store ID
    • This is a unique identifier but is no longer the primary way of identifying the store. In single-click apps, you will not receive the store ID during the authentication flow, only the store hash. Therefore we recommend parsing the store hash from the producer field in the webhook payload.
  • Data
    • All webhook payloads will provide a data type and id that confirms the entity type and the unique reference for that entity.
  • Hash
  • Created At
  • Producer
    • This field should be parsed to obtain the store hash to identify the store. In other requests, this field can sometimes be referred to as the “context”.

Inspect & Replay requests with Ngrok

Ngrok is a fantastic tool, known primarily as a way of making a private development environment available to the public internet – something that will be required to receive webhook requests from BigCommerce. It also provides this over HTTPS, meeting BigCommerce’s requirement for a webhook destination URL. In operation, it provides a detailed log of each request received.

What’s more, it allows you to replay a request. This is particularly helpful during development as it avoids the need for manually triggering the event, e.g. placing a new order, editing a product again etc.

Viewing webhooks

BigCommerce will respond to any webhook creation request with confirmation on the resulting webhook but you can also fetch all webhooks that have been created by sending a GET request to /v3/hooks endpoint. { “data”: [ { “id”: 12345678, “client_id”: “fgra65j4m2w2gumr7v93abs37qlc”, “store_hash”: “iex594q5h”, “scope”: “store/product/created”, “destination”: “https://example-bigcommerce-app.com/webhook-listener”, “is_active”: true, “created_at”: 1622708559, “updated_at”: 1622708559 }, ], … }

Note: This endpoint will only return webhooks created by the current API key. As of writing this article, there is no mechanism for viewing all webhooks created on the store by any API key. However this has been recognised as an area of improvement that BigCommerce is working towards.

Webhook handling tips

Verifying a webhook

As destination URLs for webhooks are publicly available on the internet, we should not assume that every request is valid. Anyone can mimic a webhook request payload if they discover the endpoint. So caution should be applied on trusting the data.

In an ideal world, we would be able to verify that:

  • The sender of the request is BigCommerce
  • The payload has not been modified in transit

Sender Verification

One technique for sender verification is to have an IP whitelist to check against. However, it’s impractical for BigCommerce to publicise a list of IP addresses that the webhook system uses as it is hosted on Google Cloud Platform and therefore subject to change.

Another technique would be to use a signed payload using a shared secret, similar to what’s used during app callback URLs. The benefit of this approach is that it serves as both sender verification and payload verification. However, BigCommerce does not support this approach at this time.

What BigCommerce does provide is the ability to declare custom headers to be sent with the webhook payload. We recommend using this to specify a secret key that can be used to verify the authenticity of the sender. Ideally, this would be rotated over time to mitigate the risk of it being leaked.

This header can be specified when creating or updating a webhook: { “scope”: “store/product/created”, “destination”: “https://example-bigcommerce-app.com/webhook-listener”, “is_active”: true “headers”: [ { “secret”: “super-secret-password” } ] }

Payload Verification

As BigCommerce requires that all webhook destinations are served over HTTPS, which protects from man in the middle attacks, we can be comfortable knowing that it has not been modified in transit. With the above custom header approach for sender verification we can therefore have a high degree of confidence in the validity of the data but not as high as we would have done with a signed payload.

It’s worth mentioning that the webhook payload does include a hash, but BigCommerce does not provide a way of verifying it. There are some support requests with some info about the hashing strategy but to date we haven’t had any success in using them for verification.

Regardless of the verification method used, the safest approach is to not take anything for granted – always check directly with the BigCommerce API before taking any action.

At any rate, BigCommerce always sends the bare minimum of data, i.e. store, data type and ID which prevents assumptions from being made without checking with the BigCommerce API.

Responding quickly

Slow handling of incoming webhook requests risks BigCommerce considering it to have failed and may then deactivate the webhook.

On top of this any error generated by performing any processing tasks synchronously will cause BigCommerce to repeatedly retry sending the webhook at increasing intervals. If the failure count drops, it will pause sending webhooks to the destination for a short time. Further failures will result in the webhook being disabled.

It’s wise, then, to decouple webhook acknowledgement from the processing of the payload. This means you can quickly provide a successful response to BigCommerce. A simple strategy is to log the pertinent data e.g. by dispatching a job, or pub/sub event or simply storing the information in a database for processing at a later time.

Then, even if there is a problem with processing the event, e.g. you have reached your API rate limit, you are in control of when to attempt re-processing and aren’t risking further webhooks being disabled or delayed.

Duplicate webhooks

In our experience, webhook notifications with the same payloads are received more than once, i.e. hashes exactly match. { “created_at”: 1622634884, “store_id”: “100000000123”, “producer”: “stores/{store_hash}”, “scope”: “store/product/updated”, “hash”: “eba51f842a281841a6b512ec2e14ab8424b15eb4”, “data”: { “type”: “product”, “id”: 414 } }

Anecdotally this seems to be a race condition where both notifications are received almost simultaneously but from different IP addresses.

While it would be possible to keep a log of each unique webhook received and throw away duplicates, a more practical response (that is in itself susceptible to race conditions) is to ensure that any processing performed is idempotent, i.e. repeated processing of the same data doesn’t have any further impact.

Finally on this topic, sometimes multiple similar webhooks will be received in quick succession for a single admin action. e.g. changing product purchasability settings has been seen to trigger multiple webhooks with a 2-second difference in created_at (and therefore a different hash). An idempotent solution that can cope with a higher workload than would be estimated with the assumption of 1 webhook per admin or API action will serve you well.

Time-travelling webhooks

BigCommerce does not provide any guarantees over the order in which webhooks are received, which is common for this type of system. So it’s possible to receive a product updated event before you receive a product created event. Another reason that this could happen is if the destination URL was unavailable when the product created notification was first sent but available for the product updated event.

In practice, this means if there are specific tasks that need to be performed on a creation, e.g. adding the product to another system, a check for existence will need to be performed on any update (or delete) event as well.

That product doesn’t exist…yet

In a small number of circumstances, it’s possible to receive a webhook notification for an entity such as a product and when its details are requested via the API a 404 error is returned. e.g. A product with the id of 22409 was not found`. This is likely due to delays in the proliferation of the new or updated data in internal BigCommerce systems.

Special considerations should be made to catch this error and schedule a retry attempt.

Batch processing

When subscribing to product update events, particular store/product/*, store/product/updated be wary that you will likely receive a high number of requests. This is particularly true as any inventory change will trigger the store/product/updated scope as well.

Stores that have back-office integrations which are regularly updating products and inventory levels benefit from BigCommerce’s high API throughputs. The flip side, however, is that a large number of resulting webhooks will then be triggered and require processing.

While it can be easy to scale to cope with the incoming traffic, it can be difficult to process each event while staying within API limits. Incoming webhooks can come in faster than they can be processed when each one is handled in isolation.

There are at least two techniques to tackle this:

  • Delay processing and process as a batch
  • Recognise a bulk change and trigger a full catalog sync.

Before I describe these approaches in more detail, I’d just like to thank Troy Wolf and Scott Williams again for their advice on this area when we first faced this issue.

Batch Product Processing

Break the one-to-one relationship between incoming product update webhook and processing. One approach we’ve taken is to log the modified product and trigger a new job if one doesn’t already exist but set a delay, e.g. 10 seconds. When this runs, it processes all modified products in the database rather than a single product specified on the job.

A collection of product details can be requested using the /catalog/products endpoint and using the id:in query parameter to filter the request to only changed product IDs.

One thing to note on this however is that while the maximum number of products per page that can be requested is 250, you will quickly run into an error when the URL is longer than 2048 characters and the request is rejected. So a smaller page size will be needed.

Full Catalog Sync

Similar to the previous approach, a short delay in the processing of product updates can give an opportunity to detect if there are a large number of changes. Depending on the size of the catalogue, it may be more efficient to do a full sync, i.e. iterate through all pages of the /catalog/products endpoint at 250 products per page.

Handling deactivated webhooks

If BigCommerce is repeatedly unable to deliver webhook notifications because your service was unavailable or was unable to handle the requests (e.g. responded with 4XX or 5XX response), then it will disable the webhook.

When this happens, BigCommerce will send an email to the address associated with the API credentials. For an App, that will be the email used to login to devtools.bigcommerce.com. For API accounts created in admin, it is assumed that these will go to the store owner.

Upcoming improvements to webhooks are likely to include the ability to customise the list of email addresses that are notified of deactivated webhooks.

Once the service issue has been resolved, webhooks can be re-enabled very simply by sending an update request to change the is_active flag back to true.

This can be quite a manual process, particularly if you have multiple scopes and if you are subscribing from an app that supports multiple stores. In this situation, it’s wise to have a command that validates that all expected webhooks exist and are enabled. This can then be scheduled to run regularly to automatically re-enable any webhooks that happen to get disabled because of an outage.

Further information on Scopes

There is a full list of available scopes with an official description of each one. Here is a summarised list.

Cart

  • store/cart/*
  • store/cart/created
  • store/cart/updated
  • store/cart/deleted
  • store/cart/couponApplied
  • store/cart/abandoned
  • store/cart/converted
  • store/cart/lineItem/*
  • store/cart/lineItem/created
  • store/cart/lineItem/updated
  • store/cart/lineItem/deleted

Category

  • store/category/*
  • store/category/created
  • store/category/updated
  • store/category/deleted

Channel

  • store/channel/*

Customer

  • store/customer/*
  • store/customer/created
  • store/customer/updated
  • store/customer/deleted
  • store/customer/address/created
  • store/customer/address/updated
  • store/customer/address/deleted
  • store/customer/payment/instrument/default/updated

Order

  • store/order/*
  • store/order/created
  • store/order/updated
  • store/order/archived
  • store/order/statusUpdated
  • store/order/message/created
  • store/order/refund/created

Product

  • store/product/*
  • store/product/deleted
  • store/product/created
  • store/product/updated
  • store/product/inventory/updated
  • store/product/inventory/order/updated

Shipment

  • store/shipment/*
  • store/shipment/created
  • store/shipment/updated
  • store/shipment/deleted

SKU

  • store/sku/created
  • store/sku/updated
  • store/sku/deleted
  • store/sku/inventory/updated
  • store/sku/inventory/order/updated

Store

  • store/app/uninstalled
    • This support answer appears to clarify that this is fired when a store owner cancels their subscription and the store is removed. This can be used to perform similar tasks when the app uninstall endpoint is called.
  • store/information/updated
    • When creating our apps, the information provided in this endpoint is particularly useful for knowing the timezone the current store operates in, as well as other general store information like the BigCommerce plan level.

Subscriber

i.e. newsletter subscriber.

  • store/subscriber/*
  • store/subscriber/created
  • store/subscriber/updated
  • store/subscriber/deleted

Frequently Asked Questions

How do I see all webhooks that are configured on my store?

One of the current platform limitations is that a store owner cannot see all webhooks that have been set up on their store. There is a get all webhooks request however this will only return webhooks created by the same API key (client_id specifically).

This is an area that BigCommerce is looking into to provide a method of viewing all of a store’s webhooks.

My webhook hasn’t come through?

Webhook delivery is an asynchronous process so they’re not always delivered instantaneously but usually are very soon after the triggering event. During peak shopping periods or platform incidents there can be longer delays. Check the BigCommerce status page for more details.

Tom Robertshaw
Stay up to date with Hypa news and features