PyTorch and fastai have two main classes for representing and accessing a training set or validation set:
Dataset:: A collection that returns a tuple of your independent and dependent variable for a single item
DataLoader:: An iterator that provides a stream of mini-batches, where each mini-batch is a tuple of a batch of independent variables and a batch of dependent variables
On top of these, fastai provides two classes for bringing your training and validation sets together:
Datasets:: An object that contains a training
Datasetand a validation
DataLoaders:: An object that contains a training
DataLoaderand a validation…
In this blog post I will cover what is JSON data type, what options does PostgreSQL offers to store JSON data, how you can create AWS Glue connection to Aurora PostgreSQL database running in private subnet and how can you then use AWS Glue to write data into table with JSONB datatype into Aurora/RDS PostgreSQL database.
Metric is to drive human understanding and the loss is to drive automated learning.
As Arthur Samuel had mentioned the description of machine learning
Suppose we arrange for some automatic means of testing the effectiveness of any current weight assignment in terms of actual performance and provide a mechanism for altering the weight assignment so as to maximize the performance. We need not go into details of such a procedure to see that is could be name entirely automatic and to see that a machine so programmed would “learn” from its experience.
I want to start this blog with some of my personal notes before jumping on the book.
2. What is a norm?
A norm is a measure of distance and have three properties
For quite some time I have been thinking to start writing about data science and machine learning, but somehow it wasn’t happening. Recently, I joined 25 weeks long fastbook reading session organized by Weights & Biases and lead by Aman Arora ( awesome instructor 👏). This is my first time participating in a reading session and it has been amazing experience. There is so much participation during and after the lecture that it makes learning fun
In this blog post I discuss how to export 100GB non-partitioned table from Aurora PostgreSQL to Amazon S3. I will walk you through two approaches that you can use to export the data. Firstly I will demonstrate using
aws_s3, a PostgreSQL extension which Aurora PostgreSQL provides and then using
AWS Glue service. The post also covers the performance and scaling challenges when exporting the table using AWS Glue.
I was looking for a way to enable a user to upload bunch of small size (< 10 MB) csv files to a specific S3 bucket. Exploring the ways I came across AWS Amplify service and I felt AWS Amplify makes it easy to deploy a full-stack web app for someone like me who does not have much knowledge of web app development. …
Data transformation is an important aspect of Data Engineering and can be a challenging task depending on the dataset and the transformation requirements. A bug in data transformation can have a severe impact on the final data set generated leading to data issues. In this blog I am going to share my experience of having missing values in Pandas DataFrame, handling these missing values in Pandas and converting the Pandas DataFrame to Spark DataFrame.
To give a quick background, I was writing a data transformation (ETL) job in AWS Glue using PySpark which was to be executed every 15mins. The…
AWS Glue is a serverless ETL service to process large amount of datasets from various sources for analytics and data processing. Recently I came across “CSV data source does not support map data type” error for a newly created glue job. In a nutshell, the job was performing below steps:
And it was during this write step that the glue job was failing. Lets look into it in little more details -
datasource0 = glueContext.create_dynamic_frame_from_options(…
In this blog post I will discuss following scenarios to connect to databases from AWS Lambda function:
In this setup, Amazon Aurora PostgreSQL database is running in private subnet with public accessibility set to No. The connectivity and security detail are as follows: