Summary

ClinSpark production instances are deployed within the Amazon Web Services (AWS) cloud infrastructure in a fully redundant manner, to support the automatic failover and recovery of application assets in the event of a disaster scenario. A disaster recovery scenario is accounted for through the infrastructure design and an inherent characteristic of the deployment topology. It is not seen as a procedure, but instead, an inherent characteristic of the physical design.

The design requirements that determine this level of resilience are specified in the Application Infrastructure Architecture document and to a lesser degree in the Application Architecture document. These documents are updated and revised as required and are available to customers in release technical files (via service desk portal) or upon request.

The application is designed to:

• Maintain high availability – single points of failure are eliminated

• Have high fault tolerance – application and database servers fail. Service to the customer must not be interrupted and recovery time must be quick

• Survive a full datacenter disaster without data loss or significant inconvenience to a customer

Annual Testing

The ClinSpark product team performs a business continuity and disaster recovery test annually. We do not perform these tests against any customer production instances, but instead, instances that are ‘production-like’ to demonstrate/simulate the procedure.

Recovery following a full data center disaster is designed to be automatic and transparent. As such, there is currently not a manual disaster recovery procedure or process. Robustness in the face of a disaster is an application and infrastructure architectural characteristic. Recovery is designed to complete automatically, potentially before users or support teams even realize that an outage has occurred. Testing simulates outages at multiple layers of the infrastructure and verifies that automated recovery has taken place as designed.

To simulate this disaster scenario, each layer of the infrastructure within a single AWS Availability Zone (AZ) (equivalent to a data center) is forcibly brought down. This is done in two separate exercises. The first targets the Elastic Beanstalk application server layer. The second targets the RDS database master. In both cases, the stimulus is a forced failure of all resources within an AZ. The verification is an observation of how the infrastructure responds automatically. The system is expected to recover within the expected timeframe, with no support intervention. See also https://foundryhealth.atlassian.net/wiki/spaces/DOCS/pages/3708420496/ClinSpark+Application+SLA?src=search.

Upon request, the evidence and reports associated with annual tests can be made available to customers via the service desk. Due to their sensitive and proprietary nature, these assets are not published to this documentation help site.