Posts

AWS Glue python ApplyMapping / apply_mapping example

-
Image
The ApplyMapping class is a type conversion and field renaming function for your data. To apply the map, you need two things: A dataframe The mapping list
Post a Comment
Read more

The Glue code that runs on AWS Glue and on Dev Endpoint

-
Image
When you develop code for Glue with the Dev Endpoint , you soon get annoyed with the fact that the code is different in Glue vs on Dev Endpoint glueContext is created in a different manner there's no concept of 'job' on dev endpoint, and therefore no arguments for the job, either So Mike from The MIS Theorist asked if there was a simpler way. And sure there is!
Post a Comment
Read more

AWS Glue, Dev Endpoint and Zeppelin Notebook

-
Image
AWS Glue is quite a powerful tool. What I like about it is that it's managed : you don't need to take care of infrastructure yourself, but instead AWS hosts it for you. You can schedule scripts to run in the morning and your data will be in its right place by the time you get to work. The downside is that developing scripts for AWS Glue is cumbersom , a real pain in the butt. I first tried to code the scripts through the console, but you end up waiting a lot only to realize you had a syntax error in your code.
5 comments
Read more

Using survival plot to analyze churn in Power BI

-
Image
I did not  guess I'd be working with Kaplan-Meier survival plots so soon. Analyzing churn and trying to figure out what kind of users churn more likely is not so easy. In order to calculate churn, you need to have a good volume of users that you can follow month over month, having a number of active users per month and what share of them are left behind every month. But what if you want to select a different set of users, another segment? Producing an analytics cube with the necessary dimensions takes time. And if you end up with a segment that doesn't have high volumes every month, interpreting the results can be quite tricky.
Post a Comment
Read more

Create a funnel analysis tool with Redshift and Power BI in 5 minutes

-
Image
If you're not collecting events from your product, get started right away! Events are a great way to collect behavioral data on how your users use your data: what paths they take, what errors they encounter, how long something takes etc. When you have events, there isn't a lot you cannot  analyze.
Post a Comment
Read more

How to ETL in Amazon AWS? AWS Glue for dummies

-
Image
You can do ETL in AWS in a few different ways: Glue DataPipeline A custom solution, e.g. a Docker
Post a Comment
Read more

An insights strategy for winning companies

-
Image
An executive summary Companies struggle to gain maximum benefit from analytics and insights since Analytics is seen as a support function, not a business partner and therefore is  not prioritized high enough Analytics is separated from business processes and insights are produced away from execution Under-resourced, inflexible analytics stack, which doesn’t enable speed to react to changing needs
Post a Comment
Read more

Popular posts from this blog

ThumbmarkJS: A free, open source device fingerprinting JavaScript library for the web

-
I needed a decent JavaScript fingerprinting library. I wanted something that was 'good enough': not crappy, but didn't need to be perfect. I noticed the great FingerprintJS , but sadly, they changed their license to a paid one. Boo! What is a good alternative to FingerprintJS? There are alternatives out there too, but to be honest, they all have faults. FingerprintJS is great, but they're monetizing their product in a way that I don't like. I might need hundreds of thousands of requests per month, but I can't pay thousands of dollars. It doesn't need to be perfect either, so I don't want to pay such a high premium. ImprintJS used to be a thing, but it's now archived for a few years already. ClientJS hasn't been updated for a few years either. It is promising, but I find it a little too complicated to extend and I see nowhere any statistics on how good is it. BroprintJS is the new kid on the block and hats off for trying, but it's very lim...
Read more

Snowflake UPSERT operation (aka MERGE)

-
You want to insert data to a table, but if a corresponding row already exists (by some rule, e.g. unique key) you want to update that instead of adding a new row, keeping the dataset's unique requirements intact. That's an "UPDATE AND INSERT" operation, or UPSERT. Some SQL languages have native support for it.  PostgreSQL has UPSERT as native. Also MySQL supports the operation with INSERT and ON DUPLICATE KEY UPDATE. How do you do UPSERT on Snowflake? Here's how: Snowflake UPSERT i.e. MERGE operation Snowflake's UPSERT is called MERGE and it works just as conveniently. It just has a different name. Here's the simple usage: MERGE INTO workspace.destination_table d USING workspace.source_table s ON d.id = s.id AND d.val1 = s.val1 WHEN MATCHED THEN update SET d.val2 = s.val2, d.val3 = s.val3 WHEN NOT MATCHED THEN INSERT (id, val1, val2, val3) VALUES (s.id, s.val1, s.val2, s.val3); Here the destination_table and source_table are of similar form,...
Read more