In this Community Call, Senior Software Engineer Christian Haudum talks to us about bloom filter changes for Grafana Loki, including the deprecation of the bloom compactor and a pivot towards creating bloom filters for structured metadata. Bloom filters are a probabilistic data structure that we're using to improve query performance in Loki.

Community Calls are monthly meetings that are open to everyone interested in the development of Loki. They are an opportunity for software engineers working on Loki to discuss new features as well as for open-source users of Loki to ask questions. To participate in the next Community Call, subscribe to the calendar here: https://calendar.google.com/calendar/u/0/embed

HELPFUL LINKS:
Loki Community Call Agenda: https://docs.google.com/document/d/1MNjiHQxwFukm2J4NJRWyRgRIiK7VpokYyATzJ5ce-O8/edit#heading=h.xma6rt7n0xbi
Loki GitHub repo: https://github.com/grafana/loki
Loki docs: https://grafana.com/docs/loki/
Learn more about bloom filters for query acceleration: https://grafana.com/docs/enterprise-logs/latest/manage/query-acceleration-blooms/

TIMESTAMPS:
00:00:00 Intro
00:00:38 Bloom filter components changes
00:07:40 Pivoting towards bloom filters for structured metadata instead of log lines
00:12:46 Should we always use structured metadata (OTel) for logs going forward?
00:17:07 Are there still plans to use bloom filters on log lines?
00:19:58 Which structured metadata fields will be added to blooms?
00:21:50 What's the difference between having a large index and having a bloom filter?
00:26:27 Can you set a retention period for bloom filters?
00:28:20 How are bloom filters organized?
00:33:13 Do recording rules use blooms too?
00:36:15 How can users prepare to use bloom filters?\
00:39:25 What do we need to do to migrate to the new version of bloom filters?

☁️ Grafana Cloud is the easiest way to get started with Grafana dashboards, metrics, logs, and traces. Our forever-free tier includes access to 10k metrics, 50GB logs, 50GB traces and more. We also have plans for every use case. Sign up: https://grafana.com/get/

❓ Have a question that isn't related to this video? Check out the Official Grafana Community Forums and ask your question or find your answer: https://community.grafana.com/

👍 If you found this video useful, be sure to give it a thumbs up and subscribe to our channel for more helpful Grafana videos.

📱 Follow us for the latest and greatest on all things Grafana and our other OSS projects.

X: https://twitter.com/grafana
LinkedIn: https://www.linkedin.com/company/grafana-labs/mycompany
Facebook: https://www.facebook.com/grafana

#Grafana #Observability #loki