Some stock/ETF charts are slow

Monitoring

We have managed to isolate the problem and apply a patch. We're back to normal now. We're currently trying to identify the root cause.

Jul 25, 2021 2:17 pm
Identified

We're still working on straightening the situation.

Jul 25, 2021 1:00 pm
Identified

We're still working on fixing the problem with the data vendor team. The problem is still rather minor; in case if it escalates, we're going to switch to the spare stock data vendor.

Jul 25, 2021 5:27 am
Monitoring

We're still working on fixing the problem with the data vendor team. The situation has improved and now only 2% to 5% of charts are impacted.

Jul 24, 2021 5:14 pm
Identified

Our major data provider is currently experiencing problems, which results in 2% to 10% of all stock/ETF chart requests being slower than usual.
We have reported the incident and now monitoring the situation closely.

Jul 24, 2021 2:07 pm

System Statuses

Operational Data Feed: Corporate Events (Earnings, Dividends and Splits)

Operational Data Feed: Crypto (Global Multi-Exchange)

Operational Data Feed: Forex (Global FX Aggregated)

Operational Data Feed: News Feed (Benzinga Pro)

Operational Data Feeds: Insider Trading and SEC Filings (Intrinio)

Operational Data Feed: US Equity Feed A (NYSE BQT, Tape A, Tape B; NASDAQ Tape C)

1 active incident

Operational Data Feed: US Equity Feed B (IEX; NYSE National, Chicago; NASDAQ BX, PHLX)

Operational Data Feed: US Futures (ICE, CME, Globex Futures)

Operational Data Feed: US Market Indices (CBOE)

Operational General: Phone System

Operational General: TrendSpider.com Website

Operational System: Alerts

Operational System: Chart Annotations / Drawing

Operational System: Entitlement and Data Control

Operational System: Market Scanner

Operational System: Strategy Tester

Operational System: Workspaces

Past Incidents

Jul 21, 2021

Real-time Updates for US Equities Experiencing Intermittent Service Interuption

Resolved

We have been monitoring for a while and have determined the system is stable. We are conducting root cause analysis to ensure we can prevent this type of outage in the future. For now, this issue is being closed.

Jul 21, 2021 4:45 pm
Investigating

We have reset the data feed and believe that the issue is now resolved. The Engineering team continues to monitor it to make sure that things operate smoothly.

Jul 21, 2021 1:57 pm
Investigating

We are currently experiencing an issue with real-time data feeds for US equities. The system is currently working but intermittently. The engineering team is aware of it and is working on it now.

Jul 21, 2021 1:45 pm
Jul 10, 2021

Scheduled maintenance 10th Jul 06:00am NY to 07:00 am NY

Resolved

Everything went as expected, thanks for your patience!

Jul 10, 2021 10:40 am
Planned

We'll be upgrading our infrastructure. Certain features (like in-app alert notifications) might have degraded performance during the time frame of the maintenance.

Jul 8, 2021 8:23 am
Jul 8, 2021

System Instability due to multiple servers failure

Resolved

The system is currently stable and the problem has been eliminated. Here's a short summary of what has happened and what we did to prevent situations like that in the future:

What happened
On Thursday 8th July, at 13:17 NY one of our servers became unstable and lost connectivity. The system was designed to be resilient in cases like that, so it has recovered using the rest of servers and behaved normally. However, at 13:31 NY a combination of events has happened: the lost server started coming back online (while still behaving flaky) and at the same time one of master servers became overloaded (because the system has distributed the payload of the faulty server in a dis-balanced manner).

These 2 events happening simultaneously have created an internal network routing problem, which caused up to 30% of all requests failing sporadically. The system started treating some services as "not available" because of that, and it had a cascading effect (more and more services were marked "unhealthy" and excluded from operations). The situation has escalated from "a couple services failure" to a system-wide problem within a short period of time.

What we did to prevent that in future
We have replaced the faulty server with a new one, since it proved to still have connectivity issues.
We have improved resilience of the system by increasing redundancy of our servers.
We have implemented procedures which ensure that the necessary set of master servers won't ever become overloaded like that.

If you were negatively impacted by the outage, please be sure to consider claiming an SLA credit under the TrendSpider SLA (www.trendspider.com/sla/) to compensate you for the trouble. We're sorry for the trouble, and we're learning from the incident to improve the system.

Jul 10, 2021 10:55 am
Monitoring

The system is currently stable.

We believe the matter is largely resolved and continue to monitor for signs of instability while researching the root cause. Details will be posted here once available.

Jul 8, 2021 8:01 pm
Monitoring

As of this post, service should be coming back online for most customers and users who were impacted by the outage. You may have to log back into the system if you were kicked out during the outage.

The engineering team is still investigating the root cause, which is believed to be networking related. The team is also continuing to monitor internal system health metrics to ensure continued stability, thus the case will not be closed just yet until we are sure that the matter is resolved.

If you were negatively impacted by the outage, please be sure to consider claiming an SLA credit under the TrendSpider SLA (www.trendspider.com/sla/) to compensate you for the trouble.

As of the time of this posting, the scanner has been artificially disabled just to minimize system load while the system recovers.

Jul 8, 2021 6:35 pm
Identified

We have identified that this matter impacts a large number of customers, just under 50% of all users. We are sorry for the trouble and are still working to find a resolution as fast as we can.

Jul 8, 2021 6:09 pm
Identified

The engineering team continues to work on the issue and hopes to be able to provide an ETA to resolution or quick resolution soon.

Jul 8, 2021 5:46 pm
Jul 6, 2021

No realtime updates for crypto markets

Resolved

This is now fixed, thanks for your patience.

Jul 6, 2021 1:43 pm
Identified

Our crypto data provider is experiencing a major problem, which leads to absence of real time updates for most crypto symbols. We have flagged the problem and now we're monitoring the situation.

Jul 6, 2021 1:28 pm
Jul 5, 2021

Scheduled maintenance of scanners: 5th Jul 1pm to 2pm NY

Resolved

Everything went as expected, thanks for your patience!

Jul 5, 2021 5:44 pm
Planned

We're going to be upgrading the infrastructure of our scanners. During the time frame specified, scanning might be slow or sporadically not available.
We do not expect any lengthy periods of instability.

Jul 5, 2021 4:23 pm