Production systems may encounter unexpected software bugs, performance and scaling issues. Application owners typically need access to application and database logs, system metrics, and, in some cases, the database itself in order to analyze its data/schema. Supporting these activities while protecting the privacy of user data is challenging in healthcare application environments.

The Healthcare Blocks Enhanced Security Architecture (ESA) is a software and configuration reference architecture that balances DevOps efficiency with security best practices.

1. Hash sensitive data. Healthcare Blocks managed environments support HIPAA/NIST CST encrypted data-at-rest requirements by using encrypted Amazon Web Services EBS storage volumes. Customers can go one step further by hashing data containing PHI (protected health information) and PII (personal identifiable information) in the application layer, so that sensitive data is not readable in application logs and direct database queries. Instead, the application only decrypts and displays sensitive data when authorized in user-facing interfaces and/or API's.

Modern Web application frameworks like Ruby on Rails support application-level encryption out of the box. In other frameworks and programming languages, open source libraries can be utilized for the same purpose.

2. Keep PHI out of file names and resource locators. Implement your application to avoid embedding protected health information in file names and URLs. Storage services like AWS S3 only encrypt file contents. And including PHI in URLs creates a scenario in which a user can unexpectedly share PHI with an unauthorized recipient.

3. Ship logs to a dedicated log management services. Assuming sensitive data is hashed, copies of application and database logs can be sent to other destinations, avoiding the need for a DevOps resource to have to connect to a server to retrieve logs. Healthcare Blocks can provision an Elasticsearch cluster, which excels at storing structured logs. In addition, customers can use third-party log management services such as LogDNA.

4. Send app exceptions to a dedicated error tracking service. DevOps engineers traditionally connect to servers to be able to debug unhandled app exceptions by examining logs, interacting with an app-specific console, and/or entering a Docker container. A dedicated error tracking service like Sentry can be self-hosted in your Healthcare Blocks environment and integrated with your application code, providing a centralized Web interface for reviewing app issues. A SaaS option is also available.

5. Track metrics using a dedicated service. While it is possible to connect to a server and use tools included by the OS distribution and other open source options to capture system metrics, a better approach is to collect metrics using a dedicated service like Amazon CloudWatch, which provides near real-time and historical metrics via a rich user interface. Healthcare Blocks provides a CloudWatch Dashboard for new environments. Alternately, customers can use third-party options such as DataDog and New Relic.

6. Use a CI/CD service for application deployments. Instead of DevOps teams having to deploy directly to a server from a local machine, a dedicated service can be used to accomplish the same thing. Popular third-party options include GitHub Actions, Bitbucket Pipelines, and self-hosted Jenkins servers. When evaluating these services, access controls, audit trails, and granular security controls are important in order to prevent unauthorized access to your production environment.

Even prior to committing to a CI/CD service, customers can strengthen their security posture by requesting that individual Linux user access is removed from their Healthcare Blocks servers while retaining the Dokku interface for deployments and app management commands.

7. Filter public traffic through a Web Application Firewall. Healthcare Blocks recommends adding an AWS Web Application Firewall to automatically block common Web attacks and nuisances associated with bots. Historically the "Active Response" feature in OSSEC has been used in Healthcare Blocks environments for a similar purpose, but the AWS WAF is more powerful and faster at detecting/blocking suspicious activity.

8. Use read-only database users when troubleshooting issues. When connecting directly to a database to debug a problem using a database shell or desktop tool, you should use a read-only identity that can execute SELECT queries and not be able to modify data. Your application should have a separate set of credentials associated with an identity that has elevated permissions. Items 1 and 9 are important in this case. 

9. Connect to your production environment with a VPN. If your team needs to access production resources from local machines, a private VPN connection is highly recommended. Healthcare Blocks can provision an OpenVPN Access Server in your environment, which results in a stronger security posture vs. using external VPN services. Strict access controls, logging, and multi-factor authentication are included features.

To discuss the ESA with a member of the Healthcare Blocks team, please create a support ticket.