In this video, Xiaofeng and Jialin from Roblox introduce their journey in building a robust runtime diagnostic system using Pyroscope. With over 70 million daily active users and 4.4 million creators contributing to the platform, ensuring reliability and efficiency is paramount. They discuss the challenges faced in debugging production issues and the manual, inefficient methods previously used. Through thorough investigation and collaboration with Grafana Labs, they developed an on-demand profiling workflow, enabling engineers to identify and address performance bottlenecks effectively. They share success stories where this system led to significant CPU reductions and throughput improvements. The ultimate goal is to integrate this system seamlessly into their toolchain, ensuring continuous improvement and uninterrupted experiences for Roblox users.

Major wins at Roblox:
🏆 Replacing Newtonsoft.Json with System.Text.Json reduced the CPU utilization by 36%
🏆 Optimizing the poor regular expression usage reduced CPU utilization by ~40-%
🏆 GUID creation optimization and the removal of IConfiguration injection increased RPS by 10x

Helpful links:
📃 Learn more about Pyroscope and Profiling in our docs: https://grafana.com/docs/pyroscope/latest/introduction/pyroscope-in-grafana/

☁️ Grafana Cloud is the easiest way to get started with Grafana dashboards, metrics, logs, and traces. Our forever-free tier includes access to 10k metrics, 50GB logs, 50GB traces and more. We also have plans for every use case. Sign up: https://grafana.com/get/

❓ Have a question that isn't related to this video? Check out the Official Grafana Community Forums and ask your question or find your answer: https://community.grafana.com/

👍 If you found this video useful, be sure to give it a thumbs up and subscribe to our channel for more helpful Grafana videos.

📱 Follow us for the latest and greatest on all things Grafana and our other OSS projects.

X: https://twitter.com/grafana
LinkedIn: https://www.linkedin.com/company/grafana-labs/mycompany
Facebook: https://www.facebook.com/grafana

#Grafana #Observability #ObservabilityCON #Roblox #Profiling #Pyroscope