Every year, we take time out of our everyday activities to join thousands of techies just like us for the biggest event of the season–AWS re:Invent. For one week at the end of November, industry leaders gather in Las Vegas to learn about new technologies from AWS as well as other partners in the cloud, DevOps, and IT spaces. The new technology, innovative ideas, and smart, ambitious people we have met over the past few years have been priceless for all of us at Logz.io.

One of the major developments in software design and delivery over the last few years has been a movement away from monolith applications towards microservices. One of the sticking points I’ve seen on numerous microservice applications is logging. There are some unique challenges with microservice logging that need to be addressed. In this article, we’ll look at how we can make logging in a microservice as painless as in a monolith.

With so many choices available to us today, choosing log management software that’s just right for us has never been simpler. That is, if you know exactly what it is you are looking for. But for many users, the sheer amount of computer programs that perform the same tasks, and seem so similar(sometimes almost identical) to each other, can quickly become off-putting and confusing.

In the world of agile there’s a demand to solve grey areas throughout the design process at lightning speed. Prototypes help the scrum team test ideas and refine them. Without prototypes, we can’t test ideas until the feature or product has been built which can be a recipe for disaster. It’s like running a marathon without training.

Organizations that handle logging at scale eventually run into the same problem: too many events are being generated, and logging components can’t keep up. Even with persistent queues and other mitigating features enabled, there’s simply not enough of a buffer between log generators and log ingesters to handle the volume of log lines coming in.