A common challenge in data engineering is to combine traditional data warehousing and BI reporting with experiment-driven machine learning projects. Many data scientists tend to work more with Python and ML frameworks rather than SQL. Therefore, their data needs are often different from those of data analysts. In this article, we’ll explore why having a data lake often provides tremendous help for data science use cases.
We are pleased to announce that as of 13th April 2021, Dashbird has successfully completed its SOC 2 Type 2 audit. SOC 2 engagements are based on the AICPA’s Trust Service Criteria. SOC 2 audit reports focus on a Service Organization’s non-financial reporting controls as they relate to the Security of a system. The audit was conducted by Dansa D’Arata Soucia LLP.
Let’s start with what you should monitor in Lambda functions. In general, there are two areas – user experience and the cost of the system. User experience usually comes down to availability, latency, and feature set of a service, while the cost of operating a service is important to ensure the profitability of the business.
This simple FastAPI service will help you find data in a data lake Data lakes provide a myriad of benefits. They are data agnostic and don’t require you to define a schema upfront. However, without a proper structure, it may be challenging to find the data that you need. In this article, we’ll address this problem by creating a FastAPI abstraction allowing us to query the AWS Glue metadata catalog.
If you’re worried that switching to serverless infrastructure is too expensive for your business, you’re not alone. Total spending on cloud services will top $284 billion by 2024. The good news is there are many ways to track and lower your serverless operation costs without slowing down your business. Lambda and how can it help your business? Find out more by reading these Lambda frequently asked questions.
Today, we’ll cover some of the ways you might find quite useful in your everyday work. We’ll go through some of the logging best practices in AWS Lambda, and we will explain how and why these ways will simplify your AWS Lambda logging. For more information about similar topics, be sure to visit our blog. Let’s start with the basics (and if you have the basics covered, feel free to skip ahead): How does logging work with AWS Lambda?
2020 was a difficult year for all of us, and it was no different for engineering teams. Many software releases were postponed, and the industry slowed its development speed quite a bit. But at least at AWS, some teams released updates out of the door at the end of the year. AWS Lambda received two significant improvements: With these two new features and Lambda Layers, we now have three ways to add code to Lambda that isn’t directly part of our Lambda function.
One of the most common concerns when moving to the cloud is cost. Given that cloud allows you to turn IT costs from CAPEX (long-term investments ex. in hardware equipment and software licenses) into OPEX (day-to-day operating expenses), it’s crucial to choose the right service and estimate it properly. In this article, we’ll look at the common pitfalls and discuss how you can avoid them to truly benefit from the cloud’s elasticity.
It shouldn’t be a surprise that Lambda configuration error is one of the most common error messages, and we all know AWS error messages aren’t known for being especially detailed. Oftentimes you will come across other vague error messages like “encoding not enabled,” or “stream is failing,” and depending on the context, this could mean your services could be completely down.
In the 21st century, it’s quite easy to manipulate machines and computers. Our worries are no longer if something is doable, but if something can be perfected. Therefore, we mostly search for new ideas and ways to make our work impeccable. For example, if you’re using a particular software and you realize that the software is excellent, but it could be better in some ways that would allow you to work even faster, you’ll explore the alternatives.
