Winter Rant

Saw this post on threads with this plot… At first it looks fine and all, but upon closer inspection, it seems to suggest that ChatGPT was released on/around Jan 2022. ChatGPT was however released on Nov 30, 2022.

I am not sure where this plot comes from. The closest I could find was this analysis: The Fall of Stack Overflow.

This obviously caused a stir with the advent of code generation tools. It even caused SO to put out a statement:

Over the last few weeks, we’ve seen inaccurate data and graphs circulating on social media channels regarding Stack Overflow’s traffic. We wanted to take the opportunity to provide additional context and information on the origin of that data, the traffic trends we are seeing, and the work we’re doing to ensure Stack Overflow remains a go-to destination for developers and technologists for years to come.

…

Although we have seen a small decline in traffic, in no way is it what the graph is showing (which some have incorrectly interpreted to be a 50% or 35% decrease). This year, overall, we’re seeing an average of ~5% less traffic compared to 2022. Stack Overflow remains a trusted resource for millions of developers and technologists.

Insights into Stack Overflow’s traffic

It also caused a deep dive by Gergely Orosz:

I pinged engineers at Stack Overflow to get their thoughts about what’s happening. What they said is that they are not seeing so dramatic a drop, internally, and that data shared with the most active contributors is inaccurate. I also reached out via official channels to Stack Overflow, and here’s what the company told me (the company later published a blog post with some of the below data included):

– 5%: the company wrote “overall, we’re seeing an average of ~5% less traffic compared to 2022.“
– 14%: the sharp decrease in traffic in April 2023. The company said: “we can likely attribute this to developers trying GPT-4 after it was released in March.”
– 14%: this is by how much search engine traffic is down, year-on-year.
– A predictable rise and fall, as with any sudden change. When global lockdowns started in 2020, Stack Overflow saw a spike and then a decrease in cloud migration questions and security-related ones. I sense the company is not surprised that AI had an impact on traffic and the types of questions.
– Q&A activity is definitely down: the company is aware of this metric taking a dive, and said they’re actively working to address it.

Are reports of StackOverflow’s fall greatly exaggerated?

All of this amounts to two things:

1. The plot above is wrong.
2. Generative AI is likely to have some impact on traditional tools like Stack Overflow.

But the scale of the impact is unclear. And sensationalized messaging in the form of poorly craft data plots is not the way to uncover and assess that impact.

– vijay, watching a show about monsters 🦖