Ensure data quality in your S3 data lake using Python, AWS Lambda, SNS, and Great Expectations. Data lakes used to have a bad reputation when it comes to data quality. In contrast to data warehouses, data doesn’t need to adhere to any predefined schema before we can load it in. Without proper testing and governance, your data lake can easily turn into a data swamp.

When building serverless applications, Lambda functions often form the backbone of the system. They might provide just a few lines of code, but these lines are usually what hold the whole architecture composed of many managed services together. Event-driven architecture is what this style is called, and it’s most prevalent in serverless applications. API gateways collect requests from your users, convert them to events, and send these along the way.

In this article, we’re covering all the latest updates from AWS in 2021 that serverless builders should be aware of. Before we start, let’s recall a few significant updates in serverless, announced at re:Invent 2020. One of the things that we see is that agility is really one of the primary drivers to one’s workload in the cloud and serverless is a good example of this. But the discussion often starts with cost.

Real-time processing provides a notable advantage over batch processing — data becomes available to consumers faster. In the traditional ETL, you would not be able to analyze events from today until tomorrow’s nightly jobs would finish. These days, many businesses rely on data being available within minutes, seconds, or even milliseconds. With streaming technologies, we no longer need to wait for scheduled batch jobs to see new data events.

The new Dashbird app is bringing your data together for a faster, more secure, and smoother observability experience with team collaboration in mind. The enhanced version of the Dashbird app is making your account more secure and your app navigation and data exploration faster, more intuitive, and all-around enjoyable. Additionally, you can now enable multi-factor authentication (MFA) for your Dashbird account. Check it out now!

NoSQL is a database management system that exists as an antithesis to SQL, in that it doesn’t store data in a relational model. As such, data can essentially be stored as anything, in any way a developer chooses, within reason of course. This flexibility comes from the fact that NoSQL doesn’t require a schema in the same way that SQL does.

Adding an API Gateway to your application is a good way to centralize some work you usually have to do for all of your API routes, like authentication or validation. But like every software system, it comes with its own problems. Solving errors in the cloud isn’t always straightforward, and API Gateway isn’t an exception. AWS API Gateway is an HTTP gateway, and as such, it uses the well-known HTTP status codes to convey its errors to you.

How to choose a decoupling service that suits your use case? In this article we’ll take you though some comparisons between AWS services – Kinesis vs SNS vs SQS – that allow you to decouple sending and receiving data. We’ll show you examples using Python to help you choose a decoupling service that suits your use case. Decoupling offers a myriad of advantages, but choosing the right tool for the job may be challenging.

A few weeks ago we posted the “How we built a serverless Stonks checker API for Wall Street Bets” article. And ever since, we’ve seen quite a lot of volume in the Stonks checker app. In this follow-up article, we will show you some interesting findings around the API. Over the past three weeks, we have seen a good amount of usage of the API we set up. You can see that there was a nice spike soon after the story broke.

This article was written for the Dashbird blog by Maciej Radzikowski, who builds serverless AWS solutions and shares his knowledge on BetterDev.blog. Kinesis Data Streams are the solution for real-time streaming and analytics at scale. As we learned last November, AWS themselves use it internally to keep, well, AWS working. Kinesis works very well with AWS Lambda.