When businesses are just starting out with eCommerce, it is not uncommon to quickly bootstrap an eCommerce site using an all-in-one platform. This could be offered by an eCommerce provider such as Shopify or open-source CMS platforms such as Drupal or WordPress integrated into a WooCommerce-hosted instance for orders.

With success, organizations often outgrow the initial offerings and start to integrate more complex back-end systems such as Order Management (OMS), Enterprise Resource Planning (ERP), or Warehouse Management Systems (WMS). These projects are often associated with a re-platform to a new eCommerce system (or a major upgrade to the latest offering). During a re-platform, a common approach is to focus on new UX designs/customer experiences and features of the individual platforms - leaving the actual integrations between these systems to the end of a project.

In my experience, when projects do go off the rails, this is the tipping point, leading to a common set of problems for customers:

A lack of understanding of the entire lifecycle of business processes, such as:

Product/SKU creation and merchandising

Order processing and fulfillment

Sale audit processes

Inventory management

Customer Service Rep. (CSR) functions such as appeasements, returns processing, and other critical business processes

Delays and cost overruns in the overall project

Daily operational issues and failed customer experiences

Frustration of internal business users in using too many disparate systems with too many button clicks to get something done

Lack of confidence in the selected platforms, often leading to another re-platform in a short amount of time

Intelligent data integration is a critical component of successful, modern software projects.  This is especially true in complicated projects involving multiple vendors that must exchange data frequently and stay in sync with one another.   Efficiently exchanging data in near realtime (such as webhooks, REST APIs, and GraphQL) and having proper monitoring in place to validate the successful sending and receipt of data become essential parts of delivering robust integrations.  Today, all enterprise-grade e-commerce systems, such as commercetools, Big Commerce, Salesforce, Shopify Plus, Adobe Commerce (Magento), etc., come OOTB with easily accessible API interfaces and webhooks.

At Stone Rooster Inc. (SR), our success in managing inventory and orders for our clients is driven by having reliable data sources that both our clients and our internal teams can count on.  Whenever we work with new clients, we are always looking for opportunities to improve upon the fidelity and timeliness of our data integrations, and to easily expose this information as applicable on the front-end systems.

Planning for integrations between systems should be tackled immediately in a discovery phase, and work on the integrations should start on day one of the development phase, even before new wireframes for a site design may be finalized. To approach this, it can be useful to create the following artifacts:

Process Swimlanes

These types of diagrams are useful for understanding business use cases or lifecycles of things like product creation, order flow, return flow, etc. Below is an example of a swimlane of a simple order creation and fulfillment:

During the process of creating swimlanes, it is important to include subject matter experts (SMEs) from each area to participate in the creation and validation of the various flows, and then provide these to the development team to drive more exact technical requirements and diagrams.

Creating a Common Data Dictionary

Another important step is understanding in each system how fields are labeled, and how that data translates against the schema of another system. A very common mistake that developers make when rushed is to look at the name of a field and make an assumption about how that field should be used. For example, some systems can leave out decimals in their numbers, leading to dramatically different totals between systems (i.e. a $10 shirt may be stored in one system as 1000), or in different systems of measurement (e.g. UTC vs EST time). There is a reason the Mars Climate Orbiter failed so spectacularly in 1999 when different teams were using different systems of measurement!

Again, in this step, it is critical to get SMEs that represent each system, and carefully map fields, attributes, and formatting expectations out so that the development teams have clear guidance on how to proceed.

Defining Monitoring and Alerting Requirements

Another key step is making sure development teams know up front what monitoring and alerting expectations are in place, and how integrations should “gracefully fail”. For example, what should happen if a tax API is unavailable? Should the customer just get a generic error on the website and not be able to proceed with checkout? Should no tax be charged, or is there a backup default tax rate by the state that should be used?

Understanding and clearly setting these expectations during the initial write-up of tech requirements is critical to smooth operations and better customer and internal business user experiences after launch!

A Real-World Example

At Stone Rooster Inc, we’ve worked with several clients on several different platforms, sometimes with clients over multiple different re-platforms.

In this case, one of our clients was running on a platform that was not well integrated, and many of the original integrations were done as an afterthought at the end of the project.  As you might imagine from the first part of this post, this led to many daily operational challenges.  In this case, the client realized that they needed to re-platform to a modern eCommerce solution and selected Adobe Commerce (Magento).  They invited Stone Rooster and their ERP vendor to participate early in the technical workshop and help drive the final design of the system. 

The goal was to improve on the following aspects of data: 

Create a reliable way for all systems to communicate data based on an agreed-upon Product/SKU convention.

Make the exchange of data around inventory and orders timelier and easier to process (or reprocess if needed).

Create failsafe procedures and monitors to make sure data is being exchanged accurately with no loss of data.

Workshopping Solutions

In the previous implementation, integrations were primarily done against a replicated SQL DB because of performance concerns on that platform.  This created several issues, such as: 

No single source of truth for data.

Delays in replicated data sources for getting information out of and into each system.

Large opportunities for updates to get missed, resulting in frequent manual intervention by the business.

Since Adobe Commerce provides a modern, easy-to-use REST API and GraphQL interface, the technical team realized there were immediate opportunities to improve their integrations by leveraging the APIs available. 

The first step was to create a set of robust process diagrams around the following: 

Product/SKU creation in the ERP

Sending data from ERP to e-commerce and WMS solutions so that all systems had a common way to communicate with one another using a standard set of identifiers.

Creation of Purchase Orders, updates of receipt at the warehouse, or ability to report discrepancies in receiving.

Notifications about receipt of inventory in a pickable location and ready for sale (from a fulfillment perspective)

Ability to poll for new orders at regular intervals (i.e. 15 minutes)

Ability to update e-commerce systems throughout the day on changes in inventory.

Once all parties fully understood the key integrations and the lifecycle of each flow, the teams were able to agree on how each system labeled key attributes and create a data diagram mapping fields between each system. From here, it was relatively straightforward to map each function to the RES API and rapidly test each assumption made as part of the workshop. 

This level of understanding and rapid integration was already a major improvement for the client over the previous solution. Now, each system could rapidly poll the other systems for information or post data changes via API in real-time. 

Stone Rooster also helped work with the front-end team to build screens that would expose timely information to the customer about the current status of their order, including tracking numbers and links to the shipping provider’s tracking webpages.

However, in e-commerce systems and fulfillment, when dealing with high volumes of transactions, information can still be easily missed. 

Creating Checks And Failsafe Solutions

One problem that the team wanted to address was orders that sometimes got lost or inventory updates that never made it to the website. This resulted in bad customer experiences and lost sales. The technical team came up with some simple solutions to ensure that no data updates were missed. 

For any product/SKU updates, after an insert or update call was made, the systems would call the API back to validate that the information being posted matched the desired update. If not, alerts would be thrown to the product team to ensure that any issue was resolved. 

For orders, the Magento team created an “unacknowledged orders” API that could be called from the warehouse to quickly pull down new orders.  This was followed up by an “acknowledge order” API call back into Magento to validate the order had been received.  Any new orders that had not been acknowledged within 30 minutes would then be triggered in an alert, making sure teams knew very quickly if a processing job was down, or an order somehow got missed. 

For inventory, in addition to real-time updates throughout the day, an inventory overlay process was designed to ensure that all inventory in the WMS system listed in a pickable location was sent to the e-commerce site on a nightly basis. This ensured that each morning systems were guaranteed to be in sync in case any real-time inventory updates were missed during the day.

Finally, updates to shipments could be easily posted back to the commerce systems API, but in the case, a shipping update was ever missed, an aging order report was created so the team could quickly look for orders that had not been processed within a typical timeframe to validate if any updates to the order had been missed. 

The Final Product

The client was able to launch on time and with full support from all vendors as part of the launch.  The site was a major improvement over their previous site in terms of design, speed, and ability to allow customers to quickly find what they were looking for and checkout. While there were some inevitable minor issues, all teams involved were very quickly alerted to the issues and able to act upon and resolve any issues long before a customer may be impacted and call in to alert to the issue.  Overall SLAs and customer satisfaction were improved, and the business teams were able to spend more time focusing on new business initiatives and less time chasing down issues. 

Final Thoughts

While it can be tempting to focus on user-facing design and customer features/functionalities, solutioning for critical system integrations at the start of a project and including key staff to participate in the outputs of those workshops will help deliver successful projects both for the parties involved in delivering those solutions to the end customers interacting and benefiting from those robust solutions. 

Of course, this particular use case was based on a project completed successfully seven years ago that is still running very smoothly today. Since that time, a number of newer tools have come into the market to help add extra ease of use in terms of an adaptable rules engine for integrations as well as observability and actionability that I will explore in a future post!

It’s All About the Integrations….