9. Configuration Management Policy
CIRG standardizes and automates configuration management through the use of Puppet scripts as well as documentation of all changes to production systems and networks. Puppet automatically configure all CIRG systems according to established and tested policies, and are used as part of our Disaster Recovery plan and process.
9.1 Applicable Standards
9.1.1 Applicable Standards from the HITRUST Common Security Framework
- 06 - Configuration Management
9.1.2 Applicable Standards from the HIPAA Security Rule
- 164.310(a)(2)(iii) Access Control & Validation Procedures
9.2 Configuration Management Policies
- Puppet is used to standardize and automate configuration management.
- No systems are deployed into CIRG environments without approval of the CIRG CTO.
- All changes to production systems, network devices, and firewalls are approved by the CIRG CTO before they are implemented to assure they comply with business and security requirements.
- All changes to production systems are tested before they are implemented in production.
- Implementation of approved changes are only performed by authorized personnel.
- Tooling to generate an up-to-date inventory of systems, including corresponding architecture diagrams for related products and services, is hosted on GitLab.
- All systems are categorized as production and utility to differentiate based on criticality.
- The Security Officer maintains scripts to generate inventory lists on demand using APIs provided by each cloud provider.
- These scripts are used to generate the diagrams and asset lists required by the Risk Assessment phase of CIRG’s Risk Management procedures (§4.3.1).
- After every use of these scripts, the Security Officer will verify their accuracy by reconciling their output with recent changes to production systems. The Security Officer will address any discrepancies immediately with changes to the scripts.
- All frontend functionality (developer dashboards and portals) is separated from backend (database and app servers) systems by being deployed on separate servers or containers.
- All software and systems are tested using unit tests and end to end tests.
- All committed code is reviewed using pull requests to assure software code quality and proactively detect potential security issues in development.
- CIRG utilizes development and staging environments that mirror production to assure proper function.
- CIRG also deploys environments locally using VirtualBox, libvirt, or Docker to assure functionality before moving to staging or production.
- All formal change requests require unique ID and authentication.
- CIRG uses the Security Technical Implementation Guides (STIGs) published by the Defense Information Systems Agency as a baseline for hardening systems.
- Windows-based systems use a baseline Active Directory group policy configuration in conjunction with the Windows Server 2012 STIG.
- Linux-based systems use a Red Hat Enterprise Linux STIG which has been adapted for Debian and improved based on the results of subsequent vulnerability scans and risk assessments.
- Clocks are continuously synchronized to an authoritative source across all systems using NTP or a platform-specific equivalent. Modifying time data on systems is restricted.
9.3 Provisioning Production Systems
- Before provisioning any systems, ops team members must file a request in the GitLab Deployment Ticket (DT) project.
- The VP Engineering or CTO must approve the provisioning request before any new system can be provisioned.
- Once provisioning has been approved, the ops team member must configure the new system according to the standard baseline chosen for the system’s role.
- For Linux systems, this means adding the appropriate roles to the system’s Puppet configuration file and forcing a Puppet run.
- For Windows systems, this means adding the appropriate roles to the system’s Puppet configuration file and forcing a Puppet run.
- If the system will be used to house production data (ePHI), the ops team member must add an encrypted block data volume to the VM during provisioning, or add an encrypted SAN volume.
- For systems on AWS, the ops team member must add an encrypted Elastic Block Storage (EBS) volume.
- For systems on other cloud providers, the ops team member must add a block data volume and set up OS-level data encryption using Puppet.
- Once the system has been provisioned, the ops team member must contact the security team to inspect the new system. A member of the security team will verify that the secure baseline has been applied to the new system, including (but not limited to) verifying the following items:
- Removal of default users used during provisioning.
- Network configuration for system.
- Data volume encryption settings.
- Intrusion detection and virus scanning software installed.
- All items listed below in the operating system-specific subsections below.
- Once the security team member has verified the new system is correctly configured, the team member must add that system to the Nessus security scanner configuration.
- The new system may be rotated into production once the CTO verifies all the provisioning steps listed above have been correctly followed and has marked the Issue with the
Approved
state.
9.3.1 Provisioning Linux Systems
- Linux systems have their baseline security configuration applied via Puppet profiles. These baseline Puppet profiles cover:
- Ensuring that the machine is up-to-date with security patches and is configured to apply patches in accordance with our policies.
- Stopping and disabling any unnecessary OS services.
- Installing and configuring the OSSEC IDS agent.
- Configuring 15-minute session inactivity timeouts on staging/production systems.
- Installing and configuring the Sophos virus scanner.
- Installing and configuring the NTP daemon, including ensuring that modifying system time cannot be performed by unprivileged users.
- Configuring LUKS volumes for providers that do not have native support for encrypted data volumes, including ensuring that encryption keys are protected from unauthorized access.
- Configuring authentication to the centralized LDAP/Kerberos servers.
- Configuring audit logging as described in the Auditing Policy section.
- Any additional Puppet roles applied to the Linux system must be clearly documented by the ops team member in the DT request by specifying the purpose of the new system.
9.3.2 Provisioning Windows Systems
- Windows systems have their baseline security configuration applied via the combination of Group Policy settings and Puppet profiles. These baseline settings cover:
- Joining the Windows Domain Controller and applying the Active Directory Group Policy configuration.
- Ensuring that the machine is up-to-date with security patches and is configured to apply patches in accordance with our policies.
- Stopping and disabling any unnecessary OS services.
- Installing and configuring the OSSEC IDS agent.
- Configuring 15-minute session inactivity timeouts on staging/production systems.
- Installing and configuring the Sophos virus scanner.
- Configuring transport encryption according to the requirements described in §17.9.
- Configuring the system clock, including ensuring that modifying system time cannot be performed by unprivileged users.
- Configuring audit logging as described in the Auditing Policy section.
- Any additional Puppet profiles applied to the Windows system must be clearly documented by the ops team member in the DT request by specifying the purpose of the new system.
9.3.3 Provisioning Management Systems
- Provisioning management systems such as Puppet servers, LDAP servers, or VPN appliances follows the same procedure as provisioning a production system.
- Provisioning the first Puppet server for a production cluster requires bootstrapping Puppet. The VP Engineering will oversee provisioning a new Puppet server.
- Once the Puppet server has been bootstrapped, the ops team member will apply the baseline configuration to the Puppet server by performing a Puppet agent operation as usual.
- Critical infrastructure services such as logging, monitoring, LDAP servers, or Windows Domain Controllers must be configured with appropriate Puppet roles/profiles.
- These Puppet states have been approved by the VP Engineering and CTO to be in accordance with all CIRG policies, including setting appropriate:
- Audit logging requirements.
- Password size, strength, and expiration requirements.
- Transmission encryption requirements.
- Network connectivity timeouts.
- Critical infrastruture roles applied to new systems must be clearly documented by the ops team member in the DT request.
9.4 Changing Existing Systems
- Subsequent changes to already-provisioned systems are unconditionally handled by one of the following methods:
- Changes to Puppet role or profile scripts or hiera values.
- For configuration changes that cannot be handled by Puppet, a runbook describing exactly what changes will be made and by whom.
- Configuration changes to Puppet code must be initiated by creating a Merge Request in GitLab.
- The ops team member will create a feature branch and make their changes on that branch.
- The ops team member must test their configuration change locally when possible, or on a development and/or staging sandbox otherwise.
- At least one other ops team member must review the Puppet change before merging the change into the main branch.
- In all cases, before rolling out the change to production, the ops team member must file an Issue in the DT project describing the change. This Issue must link to the reviewed Merge Request and/or include a link to the runbook.
- Once the request has been approved by the CTO, the ops team member may roll out the change into production environments.
9.5 Patch Management Procedures
- CIRG uses automated tooling to ensure systems are up-to-date with the latest security patches.
- On Debian Linux systems, the unattended-upgrades tool is used to apply security patches in phases.
- The security team maintains a mirrored snapshot of security patches from the upstream OS vendor. This mirror is synchronized bi-weekly and applied to development systems nightly.
- If the development systems function properly after the two-week testing period, the security team will promote that snapshot into the mirror used by all staging systems. These patches will be applied to all staging systems during the next nightly patch run.
- If the staging systems function properly after the two-week testing period, the security team will promote that snapshot into the mirror used by all production systems. These patches will be applied to all production systems during the next nightly patch run.
- Patches for critical kernel security vulnerabilities may be applied to production systems using hot-patching tools at the discretion of the Security Officer. These patches must follow the same phased testing process used for non-kernel security patches; this process may be expedited for severe vulnerabilities.
- On Windows systems, the baseline Group Policy setting configures Windows Update to implement the patching policy.
9.6 Software Development Procedures
- All development uses feature branches based on the main branch used for the current release. Any changes required for a new feature or defect fix are committed to that feature branch.
- These changes must be covered under 1) a unit test where possible, or 2) integration tests.
- Integration tests are required if unit tests cannot reliably exercise all facets of the change.
- Developers are strongly encouraged to follow the commit message conventions suggested by GitHub.
- Commit messages should be wrapped to 72 characters.
- Commit messages should be written in the present tense. This convention matches up with commit messages generated by commands like git merge and git revert.
- Once the feature and corresponding tests are complete, a pull request will be created using the GitHub/GitLab web interface. The pull request should indicate which feature or defect is being addressed and should provide a high-level description of the changes made.
- Code reviews are performed as part of the pull request procedure. Once a change is ready for review, the author(s) will notify other engineers using an appropriate mechanism, typically via an
@channel
message in Slack.
- Other engineers will review the changes, using the guidelines above.
- Engineers should note all potential issues with the code; it is the responsibility of the author(s) to address those issues or explain why they are not applicable.
- If the feature or defect interacts with ePHI, or controls access to data potentially containing ePHI, the code changes must be reviewed by the Security Officer before the feature is marked as complete.
- This review must include a security analysis for potential vulnerabilities such as those listed in the OWASP Top 10.
- This review must also verify that any actions performed by authenticated users will generate appropriate audit log entries.
- Once the review process finishes, each reviewer should leave a comment on the pull request saying “looks good to me” (often abbreviated as “LGTM”), at which point the original author(s) may merge their change into the release branch.
9.7 Software Release Procedures
- Software releases are treated as changes to existing systems and thus follow the procedure described in §9.4.