BroadOakData.UK

Tag: python

  • Ticket sales and trend analysis using fake data, Amazon Q and QuickSight

    —

    by

    broadoakdata
    in

    Another article compiled with help of Amazon Q. This time focusing on football ticket sales and trends analysis. The intension is to get some help to understand what dataset is required to build quick dashboard using QuickSight. football ticket sales and trend analysis using fake data, amazon q, quicksight, sqlalchemy 2.0 and postgresql Response from…

  • Customer segmentation analysis using fake data and QuickSight with Amazon Q

    —

    by

    broadoakdata
    in

    I asked a question to Amazon Q “How to use QuickSight Amazon Q for customer segmentation?” and also asked several follow up questions. To get the scripts working it needed little of bit tweaking. In most cases results were quite accurate and are listed below: Using sample date Certainly! I can help you create a sample dataset in Python that includes attributes useful for customer segmentation. This dataset will include demographics, purchase history, and behavior patterns. Here’s a Python script to generate this sample data: Generated Python Script Generated code for…

  • Migrating data from SQL Server to PostgreSQL

    —

    by

    broadoakdata
    in

    Prerequisites Steps Python Code for testing connection Sample content of .env file Installed python packages Use requirements.txt file to install necessary packages. Content of the file listed below:

  • Looking at Company House Profile stream data

    —

    by

    broadoakdata
    in

    Companies House provides various stream end points to obtain data related to companies. This https://stream.companieshouse.gov.uk/companies provides latest changes recorded about a company. Not all data points will be present for a company. Data capture and storage options Processing data using AWS DynamoDB Local DynamoDB local stores its data in SQLite database file and therefor It…

  • Using Apache Flink to process data from Kafka and MySQL database

    —

    by

    broadoakdata
    in

    I need to use Apache Flink to process data which stored in Kafka and MySql. In my previous article I shared my notes on how to use a free MySQL server (db4free.net) instance for development work. Apache Flink is good processing engine and has nice features for manipulating data using Batch and/or Streaming processing. I…

  • Using Apache Flink to process apache web log files

    —

    by

    broadoakdata
    in

    In my previous article I shared my approach for processing apache web server log files using PySpark. Here I will try to accomplish the same task using Apache Flink. I am using Apache Flink python (PyFlink) package and Flink SQL. Flink SQL is an ANSI standard compliant SQL engine that can process data both using…

  • Quick look at Apache Flink

    —

    by

    broadoakdata
    in

    What is Apache Flink? According to Apache Flink’s website: Stateful Computations over Data Streams Apache Flink is a framework and distributed processing engine for stateful computations over unbounded and bounded data streams. Flink has been designed to run in all common cluster environments, perform computations at in-memory speed and at any scale. https://flink.apache.org We will…

  • Google Earth Engine and Geospatial analysis

    —

    by

    broadoakdata
    in

    Just started looking at Google Earth Engine (GEE). We are interested in using satellite images to explore potential use cases: There are total of 1052 satellites in the orbit and have generated exabytes of data. The satellite data is growing fast. Google has built one of the largest and most sophisticated data infrastructures in the…

  • Loading JSON data into Neo4j database

    —

    by

    broadoakdata
    in

    Following on from recent article which describes method for using Neo4j database on Google CoLab. We will show how to load JSON data into a Neo4j database. Prerequisites Implementation We will use Pandas and py2neo packages for the task in hand. Output

  • Read and write data using SQLAlchemy and Pandas dataframe

    —

    by

    broadoakdata
    in

    SQLAlchemy has made it easy to read and write data from databases. I am using Google Colab and free databases to store and manipulate data. In this article, I will read data from MySQL database and save the result to a PostgreSQL using Pandas dataframe to_sql method. Steps Prerequisites Needed following Python packages – Google…