Nearly every big data processing framework supports SDKs in multiple languages and often those SDKs will have differences in the supported features and transformations. It is a challenge for SDK developers to keep the quality and performance of software consistent across programming languages they would like to support. Similarly the data engineers will need to […]

In many analytical applications, the performance of queries in BigQuery are better than expected. Typically these queries scan a large number of rows, apply an analytical function and return the result to the calling program in a matter of seconds. However, even in an analytical application, you will often have a combination of analytical queries […]

Cloud Dataflow offers a unique feature called Dataflow Template to allow building customizable and reusable pipelines. Additionally, it is a much better way to segregate the development, test and production process of creating and running a data pipeline using Apache Beam. The documentation covers plenty of details about templates (classic and flex) as well as […]

While adopting cloud computing, one of the key asks especially within regulated industries (e.g. Finance, Healthcare, Government, Telco etc.) is the ability to have full control over the data. In my experience with Financial Services teams,  full control over data usually translates to  Controlling the location of data processing and data storage systems Controlling the […]

Cloud native data warehouse products have become quite popular in last few years primarily due to low administration overhead. Some of these products are closer to being an Infrastructure service i.e. developer needs to define CPU, Memory or Storage while others are platform service. In this space, Google’s BigQuery is a pure Serverless offering where […]

In this blogpost, I will share my learning about JSON processing capabilities of Oracle database. My dataset contains 1.5 million tweets collected over 2 days for a given set of tokens (more about the tokens later in this post). And IMO it represents a good size of real world dataset for experiments rather than having […]

Oracle Cloud Database Service offers an excellent option for databases requiring extremely high disk IO performance using locally attached NVMe storage in HighIO and DenseIO compute shapes. However, there has been some discussion / doubts around using NVMe storage for databases. The cause of concern was persistence of data on NVMe devices across reboot of […]

Oracle SQLcl (Command Line interface) offers many developer friendly features. The detail about this utility and a complete list of features can be found at it’s documentation site . Among these, I find ‘alias’ and ‘repeat’ particularly useful as it allows me to easily build adhoc monitoring utilities that we so often need. And I […]