February 29th Update on Apricot Issues
A message from our Chief Operating Officer:
Hello Apricot customer,
Thank you for being a valued customer of Bonterra’s Apricot platform. We want to acknowledge the performance degradation you’ve experienced recently. We understand how challenging this has been for your business and operations and resolving these performance issues has been our highest priority.
Between 12th and 23rd February, some customers experienced intermittent errors when attempting to log into and navigate around, the Apricot application. This was due to a bug within the Oracle MySQL database that the product uses. We have demanded an urgent fix to this bug from Oracle. It was uncovered as we upgraded to a newer version of the database and, while this upgrade was imperative for the long-term performance and stability of the system, the irregular nature of the bug’s effects made it difficult to identify.
Once identified, our technical teams performed system updates and improvements to avoid any additional disruption to your business operations. We have substantially increased the system’s capacity, monitoring, redundancy, backup frequency and overall resilience while we implement additional code level measures to nullify the effects of the bug. We have been actively consulting and working with AWS throughout this incident and they have also escalated the defect with Oracle to demand resolution on behalf of their customers, Bonterra included. We continue to consult with our dedicated AWS account team to further the scalability and resilience of Apricot as we roll out new features and functionality.
As your technology partner, we take any issue you experience with our application seriously and understand the impact they may have on your organization. You can feel confident that Bonterra has put additional processes in place to address these issues more quickly and effectively in the future.
Additional technical details can be accessed via the status page. Please continue to send any issues you experience to our Support team at apricot@bonterratech.com.
Thank you for your partnership.
Snehal Desai
Chief Operating Officer
Emergency Maintenance:
To mitigate the impact of these issues in the short term, we are implementing temporary relief measures. This includes scheduling emergency maintenance beginning, Saturday, February 24th, at 5 am ET for a duration of 30 minutes. During this time, all servers will undergo a reboot, and we will increase the server resources to ensure sufficient free memory throughout the week. Moving forward, weekly reboots will occur every Saturday at 5 am ET and last until 5:30 am ET, resulting in brief downtime (5-15 minutes).
These emergency maintenance procedures will remain in place until a long-term fix can be implemented or until MySQL resolves their defect.
Current State on Special Characters:
Special Characters Issues: On February 22nd we implemented a partial fix to address the special characters issues in Apricot. Here's a breakdown of the partial fix and its implications:
Form Designer & Field Labels: Regardless of whether the character is Latin1 or in non-Latin1 languages (such as Arabic or Korean) in Form Designer, if you come across incorrect characters in a field label or field option, we regret to inform you that a manual update is necessary. This involves editing the form, inputting the original, correct characters, applying the change, and re-publishing the form. We recognize that this requires additional effort on your part, and we sincerely apologize for any frustration it may cause. We understand that the ideal scenario would not involve manual efforts, but unfortunately, it's currently not possible for Bonterra to systematically apply these specific types of changes and publishes to forms.
Data Entry: Between the MySQL 8 upgrade and yesterday evening, characters entered were subject to a conversion issue, turning non-Latin characters into question marks. Unfortunately, there's no way to retrieve the original characters. For records untouched since the MySQL 8 upgrade, we anticipate that non-Latin characters should appear as they did before the upgrade.
However, if you continue to experience issues with characters from this point forward, and you've already made the necessary modifications to your Field Properties in Form Designer, please reach out to support with the details of your issue.
We are committed to addressing any lingering problems with how Apricot handles different types of characters. Your understanding and patience during this challenging period are greatly appreciated. If you have any further questions or concerns, please don't hesitate to reach out to us. We value your partnership and are dedicated to resolving any issues promptly.