As we enter into Level 2 of COVID-19 alert levels, one of the primary responsibilities placed on businesses is around the requirement to be capable of supporting contact-tracing for individuals who come into their facilities.
Our approach for Level 2 will be a combination of the following:
- Continued general restriction of movements into the office.
- All meetings will be remote, and online.
- No courier deliveries to our office while in Level 2.
- All access to our office will be logged using a simple on-line form, accessed by QRCode displayed on the front-door, capturing the minimum govt recommended data fields: Name, Cellphone, Email, Primary office contact for visit. Time in, Time Out for all visitors.
- Staff will be using this same mechanism of logging in during Level 2.
- Data relating to those who access Enigma’s office space will be kept strictly private and not disclosed without due and reasonable cause.
- Data is being captured for the purpose of Enigma’s compliance with NZ Govt’s COVID-19 restrictions and requirements for business operation.
- As such, within this, there is an expectation that data can and will be shared with public health units as required.
- Data will be stored for at least two months, but no more than 4 months.
- Any queries can be addressed with Enigma’s Privacy Officer. Call 09-9129100 for details.
For the most part things for Enigma have properly returned to BAU, albeit with a change to our working environment with all staff now working from home.
During the second half of last week, we successfully managed to swing our off-site backups away from any dependancy on local tape media, and are now using AWS Cloud-based storage. We have utilised AWS (Amazon) in Sydney for this. We deemed this to be a critical change in order for us to maintain a robust approach to disaster recovery, without any reduction in our RTO or RPO (see definitions below). Our RTO is estimated to be ~<=48 hours and our RPO from off-site media is always intended to be ~24-48 hours (48 hrs would apply to weekend data entry, which tends to be far lower than week-day activity).
Given the urgent nature of our response we did not have any chance to stop and ask / inform our customers of these changes.
We do note some common sensitivities around data storage and also around data sovereignty, and have these considerations to share:
- You continue to deal with an NZ company directly; we remain responsible for creating and managing data backup sets for our hosted SaaS products.
- No other entity (other than AWS) is involved in holding or handling your data.
- Amazon (AWS), through controls and policy, does not have access to any of this data.
- The territory under which we operate, and the laws by which we are obligated to abide, have not changed.
Amazon has well established, data privacy and compliance policies – for more information please view:
In our view:
- We are satisfied that Amazon does not have any means to access our backup content, not only because of their own controls and policies, but also because the content which we are moving to store in their environment is encrypted before it is moved. Their FAQ page contains published statements such as: “We do not access or use your content for any purpose without your consent.”
- We are confident that the data is secure while at rest in AWS based on their public information, as well as the encryption which we have chosen to apply to that data. Our encrypted data is further encrypted at rest in AWS.
Despite all of the above, if you are a customer of Enigma, this does represent a change to the established way in which we deliver our services to you. If you, and / or your IT department / Privacy Officer have any concerns about this change to our service, then please get in touch with Enigma through your normal channels and let us know.
At this point in time, this is intended to be a temporary change to our services which will last for the duration of our COVID-19 lockdown; we need this alternative approach while we remain at levels which prevent us from being able to routinely access the data-centre environment for all non-critical issues.
Once our on-demand access to the data-centre has been restored, we will look to swing back to our tape based, off-site backup approach again.
It should be noted that during this lockdown period, we have ‘urgent, on request access’ to the data-centre. This is operated by appointment only; we have a 24/7 phone number to request such urgent access. We would reserve this type of access-request for cases where our staff strictly need to be on-site, such as a critical failure where we need to be local to diagnose a service fault.
Any questions, please contact Chris Wiltshire.
RTO ([Technical] Recovery Time Objective) is the time required to restore the system technically to an operational state after a failure.
RPO ([Technical] Recovery Point Objective) is measured backwards from the time of failure (not from time of recovery of service) and is the acceptable amount of data (in time units) to be lost before the time of failure.
Over the last 24 hours we have been working with a number of our key customers who have replied, informing us of their ‘Essential Service’ provider status, and asserting that our services formally play an essential part of their service delivery. As such we are now deemed to be part of the supply chain for Essential Service providers, and are also therefore now considered an Essential Service provider ourselves.
All of our staff are working full-time from our home locations, and for the most part we do not believe we will need to move around, or operate from our office location in order to continue to deliver those services.
We have already been making all reasonable preparations to ensure continuous delivery of our services, but with this new determination and with formal responses from these key customers, we have been empowered to notify our up-stream service providers of our change in status.
We would likely only need to invoke special privileges (of movement and of staff placement into facilities), in the event of any service failure, where we would need to put staff on-site to assist with hands-on diagnosis of faults, to remedy and rectify such a fault.
That is where our focus has been placed – we are in the process of notifying, and assessing the ability of, our providers who:
- Provide access to our co-located equipment
- Provide connectivity and network services, and associated power and co-location
- Provide hardware service and support, including replacement units under service warranty
This is ongoing work, but so far we have received positive responses from our providers as we are commonly not the only Essential Service provider which they have had to deal with.
It should be noted that we have also been working with MBIE around confusing and conflicting statements on their COVID19.govt.nz website – relating to the provision of services by a company who delivers both essential and non-essential services. Currently their site expresses a misleading expectation that those with mixed collections of essential and non-essential services must not deliver non-essential services. We have received confirmation directly from the MBIE helpline that in cases like ours, the intention was not to restrict the delivery of non-essential services if they do not create a situation where staff would be required to continue to be on-site. The wording of their current advice was geared more towards a ‘productive industry’ such as a factory which creates both essential and non-essential goods. They do not want staff continuing to be required to travel to a factory for the production of those non-essential goods. Their wording is NOT YET clear enough to be able to distinguish, for service sector industries like ours, where there is no expectation or requirement for staff to be on-premises for either Essential or Non-essential services.
If we were to need to respond in the manner outlined above, such a critical service failure would affect both our Essential and Non-essential services equally. To be clear, we would be mobilising limited, and key staff members *for the benefit of the Essential Services*, but that the benefit delivered to those services would also have a positive by-product effect of delivering greater uptime, also for those Non-essential services, since they are all serviced using the same equipment and services.
The delivery of Non-essential services alongside those Essential Services does not, in any way, impact or create any increased level of risk to our delivery of Essential Services. From our perspective, we will be equally aware of any issues which might affect either group of services, and they continue to be monitored and managed ‘as-one’.
If this situation changes, then we will be capable of segregating our services into groups, but it is not our intention to do so at this point in time.
Any questions or issues, then please contact Chris Wiltshire, General Manager – Enigma Solutions Ltd.
Thank you. Stay safe, stay well.
A further update from Enigma as we’re all responding to the 48 hour notice period of a Level-4 COVID-19 alert:
Enigma continues to be committed to delivering services as-usual in so far as: servicing our existing work, keeping our services up and running and available to customers and fully maintaining our services, network and systems. All of our staff are fully capable of working from our home offices. Our support lines will remain open and available.
Our system and network monitoring has always provided alerts and notifications of both proactive maintenance requirements as well as any critical failures. Our staff will remain diligent in our monitoring of these systems, we are able to respond to almost all alerts remotely without needing a hands-on response. Since we are not operating any face-to-face customer or public facing services the impact of having to work remotely on our core business has been minimal.
While we do deliver services into the Health and Private Sectors, we have not been able to find any specific instances which we believe would mark our work as meeting the Government’s definition of ‘Essential-Services’. If you feel that we have misinterpreted that, if you deliver Essential Services, and if you in turn rely on our services to deliver your Essential Services, then please get in touch with me to let me know.
As a final preparatory step, we are implementing temporary changes to our backup-services to enable us to take our backups, off-site without any need for a personal visit to our data-centre. We normally perform full daily backups, duplicate to tape, then shift those tapes off-site to our fireproof tape-safe (within our office).
As a temporary alternative, we are currently commissioning a cloud-based extension of our backup services using Amazon AWS based storage services (located in Sydney – Australia), we will digitally ship the backup content over there.
Our backups are encrypted before they are written to disk, they remain encrypted throughout their life. When they are at rest within Amazon’s storage they sit on a further encrypted volume. Our retention of those backups within Amazon’s services is currently designed to be 16 days (just over a two week period), after which they will be removed from Amazon’s services.
Our aim is that, by performing these changes, we will hold location independent backups for DR purposes. Our primary backup content will continue to reside within our own networks; this extra step has been taken to improve resilience in the case of any catastrophic loss of the production hosting environment.
In addition to this, we will be switching from full, daily backups, to one full-weekly backup, followed by incremental backups for the remainder of the week.
Our office number (09-9129100), and our DDIs continue to work as normal, as does our support line: 0800 PREDICT – If you have any questions, concerns or queries, please get in touch.
We wish you all the best; for you, your teams, your families and customers.
Chris Wiltshire – General Manager.
Email to staff (21st March 2020):
Today (Saturday 21st March) the PM raised our Covid-19 alert level to ‘2’.
You might want to visit that link, read and listen to what this means for everyone.
Importantly, to us all individually and to Enigma; this heightened level means that we have been asked to implement greater self-isolation practices, and for all businesses to put into place any available plans to ‘work-from-home’. Not all companies can do this as fully and as effectively as we believe we can. In our case, this means that from Monday, until further notice, we will all be working from our home environments for the foreseeable future.
I would expect each of you to have what you need in order for you to effectively work from home on Monday morning, even if this means arranging to collecting items from the office *over this weekend, before Monday morning*.
… Vish will continue to travel to the office daily, in order to manage the required backup tape swaps, and to continue to store the tapes in the office each day. In order for Vish to be as socially-distanced as every other one of us, this means that the rest of us cannot be in the office at the same time as Vish. – Please, from Monday morning, avoid dropping into the office without making arrangements to be in there (through me).
Our office cleaning will continue as normal, with a full clean taking place each weekend. Vish, in the meantime, you will need to perform your own additional cleaning on a daily basis (while you’re the only one in the office). – Please wipe down with disinfectant and a paper towel: commonly used door handles, kitchen surfaces, bathroom sink areas, alarm keypad, your desk surface etc. This is to keep the office as clean as possible from any potential contaminants which you might bring into it while you’re there. If you end up having to take time off because you become ill, then we want anyone else who has to go into the office to have confidence that you’ve been keeping it as clean as possible please.
So, please do whatever prep you need to before Monday’s 09:30 meeting, and we’ll see you online, from your homes.
Thanks, any questions, please give me a ring: 021624717.