A guide to Bespoke Data Loader Development & Services
This is a brief guide to integrating systems at your own offices (“on-premise systems”), with your Salesforce.com project. It also explains the pros and cons of the main general approaches to such a system integration.
Components of a typical Bespoke Data Loader
The diagram above shows some of the steps required in a typical system integration. More often than not on-premise systems provide a passive interface i.e. one where an external program is expected to request data. Sometimes it is possible to make such a request in such a way that only data which has changed since the previous request, is returned (Change Fetcher and Change Receiver in the diagram). This makes it possible to efficiently make regular checks and therefore provide more timely updates than in the case where a whole set of data must be downloaded on each request.
People often simply envisage the forwarding of data changes from one system to another. However, in practice, on-premise systems are sometimes shutdown temporarily and network connections are sometimes temporarily lost. It is often still possible for data changes to have been made by users of the on-premise system and Salesforce.com, whilst the 'loader' (implementing the system integration) was disconnected and unable to respond. When the loader restarts (and also the first time it is run), it will typically have to request the full set of data from the on-premise system and from Salesforce.com, in order to look for any data which changed whilst it was disconnected.
There are three proactive loader processes which must repeatedly run:-
-
Change processing
-
Restart process
-
Monitoring, control & retry
The main purpose of monitoring, is to notice when the on-premise system stops or the network route between the systems has been lost. When it is lost, a Controller is notified and the system will automatically attempt to restart. The monitoring will also notify a support team, if the system fails to restart; and again when the system recovers successfully.
All of these functions could be hosted on your own on-premise servers. There are however several good reasons to take the different approach, of hosting as much of the loader as possible on a cloud based integration server, as illustrated below (provided Designing Tomorrow Ltd).
Using a Cloud Based Integration Server
It makes sense to run monitoring on a Cloud Based Integration Server because:-
-
When your on-premise systems are interrupted, the monitoring process could be simultaneously lost, if it is hosted on-premise. It would then be unable to report the outage.
-
It is more practical for Designing Tomorrow Ltd to provide support for many loaders, where monitoring and notification is managed on a cloud based integration server's single console.
-
It is more costly for Designing Tomorrow Ltd (or any consultancy) to provide support for many loaders, when they are hosted on-premise in many locations. This cost must be passed on and so we attempt to minimise that cost by hosting only a small, stateless part of the implementation on-premise (“Encode File”, “Change Fetcher” and “Restart Fetcher” in the diagram).
By stateless, we mean that the on-premise part of the loader always simply runs in the same mode and doesn't have to remember what was happening before it was restarted. The Monitoring, Control and Retry facilities are stateful and are more complex, hence they are better located on the cloud based integration server.
Helping to get it right first time. Let us develop the loader for you.
Some of our customers have competent in-house developers. Nevertheless, system integration projects throw up the following challenges, with associated learning curves, all of which must be managed at once:-
-
There is a need to learn the many details of the Salesforce.com APIs whilst achieving both security and system availability. Salesforce.com provide several interface/API technologies for different use cases. Often, APEX web services or APEX triggers will be used, rather than the plain Salesforce API and Bulk APIs.
-
The web-services programming required for integration is different to database programming or web programming. It requires a good understanding of the underying internet protocols, to avoid security flaws.
-
The implementation work for the Restart process is often substantial and is different for each different on-premise system. It is essential to design this correctly and this can benefit from experience with similar projects. Getting this wrong, can leave problems which may not be obvious for some time.
-
The implementation of Control and Retry facilities is similar for each project but is an intricate process, which has to cover all eventualities. The Monitoring process has to be designed for the specific plausible failure modes for the specific on-premise systems and networks. It is essential to design this correctly and this can benefit from experience with similar projects.
-
Where multiple systems are involved, an integration solution should make it easy (by design) to determine quickly, which sub-system to investigate. This helps to minimize duplicated support effort.
Getting it working, in the real world.
The data loaders we support today, work with a variety of on-premise systems. Even though on paper, our customers' networks and virtualised servers should all run with very high availability, in practice there are numerous ways for system and network interruptions to occur (including resource contentions). When occasionally, customers suffer a major IT interruption on site (e.g. during a major infrastructure replacement / upgrade) the last thing they need is extra problems. Fortunately, our carefully designed system integrations wait patiently and reliably for the outage to pass and then auto-recover, with a notification of the recovery... One less problem for a busy in-house IT team and maximum system availability for end users.
Auto-recovery works so well that in practice, we only normally have to provide software maintenance support (i.e. ensuring that the integration continues to work when the Salesforce.com API is upgraded). There are also occasionally, queries which turn out not to be loader problems but which nevertheless have to be investigated. Again, the integrations / loadersare designed to minimise the time and cost for such investigations.
For more information, please contact us at:-
webcontact@designingtomorrow.co.uk
+44 (0)7806 508 705
(Revision 2.0 March 2013)
Comments
Post a Comment