An update on login issues

On Wednesday and Thursday of this week, a small number of users experienced issues accessing and interacting with HipChat. Whether the impact is small or large doesn’t matter—we know you rely on HipChat. As we work to scale HipChat and the underlying architecture, we’ve committed to keeping you up to date on what is happening.

What happened

Beginning Wednesday morning, some users were not able to log into HipChat. This continued intermittently, for a total of approximately three hours through Thursday afternoon. In all, it affected around 5% of users.

A mixture of events led to the issues. In recent weeks, our development team released a few new client-side features and made subtle changes to the way we interact with user records. These two changes, along with many new integrations being used by our users, has led our user database to experience higher write contention than normal.

The high load slowed down many different backend components, which were meant to interact with user data. One acute example of this slowdown is authentication tokens.

Why logins were failing

For a myriad of reasons, the management of a user’s ‘logged in’ and ‘authenticated’ state across multiple devices is complicated. Each user device gets a unique token when an authentication is successful, which is valid for a window of time. Before a device token’s window expires, it will be refreshed by the client running on that device behind the scenes. The existing valid token is not replaced until after a new valid one has been attained.

Once a refresh process starts, it is only valid for a small window of time. The refresh process is heavily encrypted, and while not particularly computationally expensive, it isn’t exactly free either. That refresh process must balance sensitivity to latency, encryption complexity, and the security of maintaining a very short window within which a refresh process may happen once it starts.

Under certain unusual circumstances, sufficient latency builds up in different components of our backend causing problems for HipChat clients to get new refresh tokens and decrypt them before they expire, which then triggers another request to refresh a token, adding to the overall queue and generally getting logins into a bad state for short windows of time. This week we experienced these short windows of intermitted token refresh traffic jams, causing the degraded performance.

What we did

In response to this incident, we’ve taken several steps. First, we rolled some newly deployed code back, which removed one newly introduced attenuation. We optimized a few spots in our code to retain caches that we were flushing too early, substantially increasing the performance of some expensive edge cases. Combined, these have reduced the latency overall and specifically in this particular circumstance.

What we’re doing

We have several more improvements to make to stamp out even more of the unusual combination of factors we experienced this week. We’ve already scheduled a maintenance window this coming weekend to implement some of them. We’ll continue to refine this process for you (including new functionality) over the coming weeks.

We know how frustrating issues like this can be for your team, and we’re very sorry for any inconvenience this caused. We are committed to keeping you informed as we improve our architecture. Thank you for your understanding and patience.

The post An update on login issues appeared first on HipChat Blog.

What’s going on?

Ok, I’ve been working with my little business for a couple of years. I thought I’d gotten used to the way things went. But the last week has confused me.

I started with a new site a couple of months ago and everything was looking fine(within reason). My traffic is steadily growing from when I changed domain and site. Everything was looking good. But the last week my bounce rate has gone from 60% to over 70%. And no one is going to checkout. Admittedly I only had a handful a week at times, but it’s been a week and nothing, not one. Is it just me or has the Internet just lost interest in e-commerse? Literally I’ve been watching my site for two years and gotten used to the way it worked, now all of a sudden nothing. What’s happened? Is it just me? Can anyone give me some advise?


Hi Guys,

I am making some recommendations of Pagination so rel=next and prev is compulsory.

However, Google recommends a ‘view all’ page that is the canonical version linked to as canonical fro each paginated page.

But if we did this, the page itself as a view-all would be huge and load time would be incredibly slow as a result of it being very heavy.

Is there a way around this, or should I just stick to rel-next/prev, and forget the canonical to a view all page?


Moved to a new domain and redirecting to that domain… Now what?

My previously moved my website to a new domain. Now, I’m having my old domain redirect all my traffic to my new domain. I also submitted my site map to Google (the new one). Now, what should I do? I’m still seeing my old posts from my old domain up at the search engine.:roll:

Tons of visitors but no sale

I have a CMS page on my ecommerce site that is mainly informative. It is ranked very well on google and brings in a lot of visitors.
Out of 3000 unique visitors per month, this page alone gets close to 1800 of them. But I am having a serious problem converting these readers into buyers.
Are there any effective tricks and tips to achieve this?
One of the things I though of was to have a pop-up appear after few seconds offering a discount for a purchase. But this to me looks very commercial and can put off readers.

Can you guys please share your opinion?

Pandaaaa! Or is it something else

We have an ecommerce site we think go hit by panda some years ago.
We have been redeveloping the site for the last year but can’t seem to shake it.

Previously all the descriptions had been taken from the datafeed and there were multiple variations plus other site were using the feed.

We rewrote all the descriptions so they were unique and use the canonical tag to cut down on the product variations. So instead of having 15 pages of variations of the product we have about 5.
Despite all this there are still 2800 pages in the index but when we use there are only about 6 or 7 pages (60/70 listings)

We are confused why the products with unique descriptions aren’t all been shown and what are the other pages which are in the index that we can’t see.

Any ideas?

Thank you