For the most part things for Enigma have properly returned to BAU, albeit with a change to our working environment with all staff now working from home.
During the second half of last week, we successfully managed to swing our off-site backups away from any dependancy on local tape media, and are now using AWS Cloud-based storage. We have utilised AWS (Amazon) in Sydney for this. We deemed this to be a critical change in order for us to maintain a robust approach to disaster recovery, without any reduction in our RTO or RPO (see definitions below). Our RTO is estimated to be ~<=48 hours and our RPO from off-site media is always intended to be ~24-48 hours (48 hrs would apply to weekend data entry, which tends to be far lower than week-day activity).
Given the urgent nature of our response we did not have any chance to stop and ask / inform our customers of these changes.
We do note some common sensitivities around data storage and also around data sovereignty, and have these considerations to share:
- You continue to deal with an NZ company directly; we remain responsible for creating and managing data backup sets for our hosted SaaS products.
- No other entity (other than AWS) is involved in holding or handling your data.
- Amazon (AWS), through controls and policy, does not have access to any of this data.
- The territory under which we operate, and the laws by which we are obligated to abide, have not changed.
Amazon has well established, data privacy and compliance policies – for more information please view:
In our view:
- We are satisfied that Amazon does not have any means to access our backup content, not only because of their own controls and policies, but also because the content which we are moving to store in their environment is encrypted before it is moved. Their FAQ page contains published statements such as: “We do not access or use your content for any purpose without your consent.”
- We are confident that the data is secure while at rest in AWS based on their public information, as well as the encryption which we have chosen to apply to that data. Our encrypted data is further encrypted at rest in AWS.
Despite all of the above, if you are a customer of Enigma, this does represent a change to the established way in which we deliver our services to you. If you, and / or your IT department / Privacy Officer have any concerns about this change to our service, then please get in touch with Enigma through your normal channels and let us know.
At this point in time, this is intended to be a temporary change to our services which will last for the duration of our COVID-19 lockdown; we need this alternative approach while we remain at levels which prevent us from being able to routinely access the data-centre environment for all non-critical issues.
Once our on-demand access to the data-centre has been restored, we will look to swing back to our tape based, off-site backup approach again.
It should be noted that during this lockdown period, we have ‘urgent, on request access’ to the data-centre. This is operated by appointment only; we have a 24/7 phone number to request such urgent access. We would reserve this type of access-request for cases where our staff strictly need to be on-site, such as a critical failure where we need to be local to diagnose a service fault.
Any questions, please contact Chris Wiltshire.
RTO ([Technical] Recovery Time Objective) is the time required to restore the system technically to an operational state after a failure.
RPO ([Technical] Recovery Point Objective) is measured backwards from the time of failure (not from time of recovery of service) and is the acceptable amount of data (in time units) to be lost before the time of failure.
Over the last 24 hours we have been working with a number of our key customers who have replied, informing us of their ‘Essential Service’ provider status, and asserting that our services formally play an essential part of their service delivery. As such we are now deemed to be part of the supply chain for Essential Service providers, and are also therefore now considered an Essential Service provider ourselves.
All of our staff are working full-time from our home locations, and for the most part we do not believe we will need to move around, or operate from our office location in order to continue to deliver those services.
We have already been making all reasonable preparations to ensure continuous delivery of our services, but with this new determination and with formal responses from these key customers, we have been empowered to notify our up-stream service providers of our change in status.
We would likely only need to invoke special privileges (of movement and of staff placement into facilities), in the event of any service failure, where we would need to put staff on-site to assist with hands-on diagnosis of faults, to remedy and rectify such a fault.
That is where our focus has been placed – we are in the process of notifying, and assessing the ability of, our providers who:
- Provide access to our co-located equipment
- Provide connectivity and network services, and associated power and co-location
- Provide hardware service and support, including replacement units under service warranty
This is ongoing work, but so far we have received positive responses from our providers as we are commonly not the only Essential Service provider which they have had to deal with.
It should be noted that we have also been working with MBIE around confusing and conflicting statements on their COVID19.govt.nz website – relating to the provision of services by a company who delivers both essential and non-essential services. Currently their site expresses a misleading expectation that those with mixed collections of essential and non-essential services must not deliver non-essential services. We have received confirmation directly from the MBIE helpline that in cases like ours, the intention was not to restrict the delivery of non-essential services if they do not create a situation where staff would be required to continue to be on-site. The wording of their current advice was geared more towards a ‘productive industry’ such as a factory which creates both essential and non-essential goods. They do not want staff continuing to be required to travel to a factory for the production of those non-essential goods. Their wording is NOT YET clear enough to be able to distinguish, for service sector industries like ours, where there is no expectation or requirement for staff to be on-premises for either Essential or Non-essential services.
If we were to need to respond in the manner outlined above, such a critical service failure would affect both our Essential and Non-essential services equally. To be clear, we would be mobilising limited, and key staff members *for the benefit of the Essential Services*, but that the benefit delivered to those services would also have a positive by-product effect of delivering greater uptime, also for those Non-essential services, since they are all serviced using the same equipment and services.
The delivery of Non-essential services alongside those Essential Services does not, in any way, impact or create any increased level of risk to our delivery of Essential Services. From our perspective, we will be equally aware of any issues which might affect either group of services, and they continue to be monitored and managed ‘as-one’.
If this situation changes, then we will be capable of segregating our services into groups, but it is not our intention to do so at this point in time.
Any questions or issues, then please contact Chris Wiltshire, General Manager – Enigma Solutions Ltd.
Thank you. Stay safe, stay well.
A further update from Enigma as we’re all responding to the 48 hour notice period of a Level-4 COVID-19 alert:
Enigma continues to be committed to delivering services as-usual in so far as: servicing our existing work, keeping our services up and running and available to customers and fully maintaining our services, network and systems. All of our staff are fully capable of working from our home offices. Our support lines will remain open and available.
Our system and network monitoring has always provided alerts and notifications of both proactive maintenance requirements as well as any critical failures. Our staff will remain diligent in our monitoring of these systems, we are able to respond to almost all alerts remotely without needing a hands-on response. Since we are not operating any face-to-face customer or public facing services the impact of having to work remotely on our core business has been minimal.
While we do deliver services into the Health and Private Sectors, we have not been able to find any specific instances which we believe would mark our work as meeting the Government’s definition of ‘Essential-Services’. If you feel that we have misinterpreted that, if you deliver Essential Services, and if you in turn rely on our services to deliver your Essential Services, then please get in touch with me to let me know.
As a final preparatory step, we are implementing temporary changes to our backup-services to enable us to take our backups, off-site without any need for a personal visit to our data-centre. We normally perform full daily backups, duplicate to tape, then shift those tapes off-site to our fireproof tape-safe (within our office).
As a temporary alternative, we are currently commissioning a cloud-based extension of our backup services using Amazon AWS based storage services (located in Sydney – Australia), we will digitally ship the backup content over there.
Our backups are encrypted before they are written to disk, they remain encrypted throughout their life. When they are at rest within Amazon’s storage they sit on a further encrypted volume. Our retention of those backups within Amazon’s services is currently designed to be 16 days (just over a two week period), after which they will be removed from Amazon’s services.
Our aim is that, by performing these changes, we will hold location independent backups for DR purposes. Our primary backup content will continue to reside within our own networks; this extra step has been taken to improve resilience in the case of any catastrophic loss of the production hosting environment.
In addition to this, we will be switching from full, daily backups, to one full-weekly backup, followed by incremental backups for the remainder of the week.
Our office number (09-9129100), and our DDIs continue to work as normal, as does our support line: 0800 PREDICT – If you have any questions, concerns or queries, please get in touch.
We wish you all the best; for you, your teams, your families and customers.
Chris Wiltshire – General Manager.
Email to staff (21st March 2020):
Today (Saturday 21st March) the PM raised our Covid-19 alert level to ‘2’.
You might want to visit that link, read and listen to what this means for everyone.
Importantly, to us all individually and to Enigma; this heightened level means that we have been asked to implement greater self-isolation practices, and for all businesses to put into place any available plans to ‘work-from-home’. Not all companies can do this as fully and as effectively as we believe we can. In our case, this means that from Monday, until further notice, we will all be working from our home environments for the foreseeable future.
I would expect each of you to have what you need in order for you to effectively work from home on Monday morning, even if this means arranging to collecting items from the office *over this weekend, before Monday morning*.
… Vish will continue to travel to the office daily, in order to manage the required backup tape swaps, and to continue to store the tapes in the office each day. In order for Vish to be as socially-distanced as every other one of us, this means that the rest of us cannot be in the office at the same time as Vish. – Please, from Monday morning, avoid dropping into the office without making arrangements to be in there (through me).
Our office cleaning will continue as normal, with a full clean taking place each weekend. Vish, in the meantime, you will need to perform your own additional cleaning on a daily basis (while you’re the only one in the office). – Please wipe down with disinfectant and a paper towel: commonly used door handles, kitchen surfaces, bathroom sink areas, alarm keypad, your desk surface etc. This is to keep the office as clean as possible from any potential contaminants which you might bring into it while you’re there. If you end up having to take time off because you become ill, then we want anyone else who has to go into the office to have confidence that you’ve been keeping it as clean as possible please.
So, please do whatever prep you need to before Monday’s 09:30 meeting, and we’ll see you online, from your homes.
Thanks, any questions, please give me a ring: 021624717.
Last Updated: 18 Mar 2020.
Enigma has assessed likely impacts from COVID-19 and how they might affect our ability to deliver services.
We believe that we are well positioned and fully capable of delivering continued and uninterrupted services to you, at our normal levels, through the COVID-19 pandemic.
- Last week we implemented strengthened distancing measures for our staff at work, and will continue to do so for the immediate future.
- We are well prepared to be able to work remotely, and have conducted a full-office work from home day, with no discernible negative impact.
- We have a reliable infrastructure which requires a minimum amount of hands-on attention and intervention on a week to week basis.
- We have extensive system monitoring and reporting which is a well established and core aspect of our approach to meeting service level expectations.
- Our entire phone system, all normal extensions and calling groups work while we are working remotely, things like extension dialling and transferring calls are not impacted.
- Our staff are well used to using remote meeting tools (our preference is ZOOM), and use these routinely.
- We have a shared knowledge base (our wiki) which is accessible to our staff from home.
- Our support systems are web-based.
- We have a company wide ‘Slack’ environment which enables our developers to collaborate well, remotely.
Our staff will remain available to you, our customers, as normal, through all normal channels. We will be maintaining our normal working hours, and a normal full working week.
Should any of our individual staff members become ill during the pandemic, we will obviously prioritise their health and safety and their recovery. We will assess the impact on any in-flight development work and make required adjustments. Our team has sufficient breadth in their product knowledge and cover to be able to cover all aspects of production products.
If anything were to change, we will let you know at the earliest possible opportunity.
Currently all 10 of our full-time staff, and our two regular contract-based workers are fighting fit; we have had no direct, first hand concerns as of yet.
All further updates will be posted to this URL. https://www.enigma.co.nz/covid-19
In light of growing local concerns around COVID-19 and the potential for Auckland-based community-spread, Enigma has developed a policy aimed at keeping our staff, their families and our customers safe from impact (which Enigma might create, or be reasonably in control of).
Today, 10th March, Enigma has implemented a number of changes to policy and to our office setup:
- No casual visitors will be allowed to enter our office space. – Couriers etc will be greeted at our office door.
- All booked appointments will be expected to execute hand-hygiene measures upon arrival, visitors will be limited to our entry area, meeting room and bathroom areas. General office space and kitchen areas are out of bounds.
- Enigma staff will be prohibited from attending meetings at Hospital, DHB, PHO or clinic settings (except where this is absolutely imperative).
- Enigma staff are strongly encouraged to make every effort to arrange remote meetings wherever possible. We have good teleconferencing and online meeting tools which should enable us to carry on with all business meetings remotely.
- Office and personal hygiene policies and procedures have been circulated to staff in an effort to keep our office space ‘a clean and safe zone’.
- While we will be happy to wave and say hello, we won’t be shaking hands for the immediate future.
- Staff have been requested to consider whether travel is necessary, and also to be considerate of any mass-public-gatherings they attend.
- Staff have been requested to execute good personal hygiene at home (and in transit) and to encourage good practices across their families and direct contacts.
This position and our stance will be reviewed weekly.
If you have any questions about this, please contact email@example.com
A copy of our internal policy document is here: Covid-19-Policy_v1.01 – for anyone with shared interests / common concerns.
(Please also note our anti-spam requirements, linked to from the bottom of each and every page on this website)