E2e bigquerymultitable #1277

itsmekumari · 2023-08-18T10:50:01Z

E2e bigquerymultitable

poojantcs · 2023-08-22T12:02:34Z

Please incorporate the changes from this PR: data-integrations/database-plugins#418

itsankit-google · 2023-08-24T17:23:08Z

.github/workflows/e2e.yml

@@ -1,4 +1,4 @@
-# Copyright © 2021 Cask Data, Inc.
+# Copyright © 2023 Cask Data, Inc.


nit: 2021-2023

itsankit-google · 2023-08-24T17:25:12Z

src/e2e-test/java/io/cdap/plugin/bigquerymultitable/stepdesign/MultiTableClient.java

+import java.sql.SQLException;
+import java.sql.Statement;
+
+import static io.cdap.plugin.bigquerymultitable.stepdesign.BQMultiTableValidation.getMysqlConnection;


nit: avoid using static imports, it makes the code unreadable.

itsankit-google · 2023-08-24T17:26:48Z

pom.xml

+    <dependency>
+      <groupId>mysql</groupId>
+      <artifactId>mysql-connector-java</artifactId>
+      <version>8.0.19</version>
+      <scope>test</scope>
+    </dependency>


If the dependency is only required for e2e-tests, plz add it under e2e-tests profile.

itsankit-google · 2023-08-24T17:30:03Z

src/e2e-test/java/io/cdap/plugin/bigquerymultitable/runners/TestRunner.java

+  features = {"src/e2e-test/features"},
+  glue = {"io.cdap.plugin.bigquerymultitable.stepsdesign", "io.cdap.plugin.gcs.stepsdesign", "stepsdesign",
+    "io.cdap.plugin.common.stepsdesign"},
+  tags = {"@BQMT_SINK and not @PLUGIN-1669"},


I am not sure if we can go ahead with the PR before PLUGIN-1669 is implemented.
Almost all the runtime scenarios require multi table plugin and will be skipped.

cc: @Vipinofficial11 @poojantcs

Moving this PR to draft for now, To run any of the tests which uses BQMT plugin we will have to add a feature to the scripts so we can use plugins from CDAP Hub in the test executions.

itsankit-google · 2023-08-24T17:32:37Z

src/e2e-test/java/io/cdap/plugin/bigquerymultitable/runners/TestRunnerRequired.java

+  glue = {"io.cdap.plugin.bigquerymultitable.stepsdesign", "io.cdap.plugin.gcs.stepsdesign", "stepsdesign",
+    "io.cdap.plugin.common.stepsdesign"},
+  tags = {"@BQMT_Required"},
+  //TODO: Enable test once issue is fixed https://cdap.atlassian.net/browse/PLUGIN-1134


Can you plz help me understand how is PLUGIN-1134 related?

I cannot see any scenario involving GCSMove plugin in this PR.

itsankit-google · 2023-08-24T17:34:36Z

src/e2e-test/resources/pluginParameters.properties

+sourceTable=dummy
+tableNameField=tableName
+jdbcPluginName=mysql
+connectionString=jdbc:mysql://10.232.0.88/cdfmysqldb


Plz do not push any Ip Addresses/hostnames in github.

Deleted the IP

src/e2e-test/java/io/cdap/plugin/bigquerymultitable/stepdesign/BQMultiTableValidation.java

src/e2e-test/java/io/cdap/plugin/bigquerymultitable/stepdesign/BigQueryMultiTable.java

Aryan-Verma · 2023-08-27T15:01:11Z

src/e2e-test/java/io/cdap/plugin/bigquerymultitable/stepdesign/MultiTableClient.java

+
+  public static void createSourceDatatypesTable(String sourceTable) throws SQLException, ClassNotFoundException {
+    try (Connection connect = getMysqlConnection(); Statement statement = connect.createStatement()) {
+      String createTableQuery = "CREATE TABLE " + sourceTable +


How are we validating multiple tables when we are creating just one table here?

As Big Query Multi Table Sink Plugin splits into two Tables. So we have used two existing tables tabA and tabB. Further, extracting the data from both the tables for the validation. This is the source table that is having a column table name which has two values tabA and tabB, which gets split into two table tabA and tabB by BQMT sink plugin.

src/e2e-test/resources/pluginDataCyAttributes.properties

updated correct conn string, removed space in yml, removed verify goal in pom. changed sql connector java version to run in github actions and bqmt secret updated

Vipinofficial11 added the build Trigger unit test build label Aug 18, 2023

Vipinofficial11 assigned itsmekumari and Vipinofficial11 and unassigned itsmekumari and Vipinofficial11 Aug 18, 2023

sau42shri requested review from Aryan-Verma and DJSagarAhire and removed request for DJSagarAhire August 22, 2023 08:11

priyabhatnagar25 force-pushed the e2e_bigquerymultitable branch 2 times, most recently from 6b14d85 to 638e149 Compare August 22, 2023 10:04

priyabhatnagar25 force-pushed the e2e_bigquerymultitable branch 3 times, most recently from 79d82d2 to cd2286e Compare August 24, 2023 11:46

itsankit-google reviewed Aug 24, 2023

View reviewed changes

data-integrations deleted a comment from priyabhatnagar25 Aug 24, 2023

Vipinofficial11 force-pushed the e2e_bigquerymultitable branch from cd2286e to c5e3721 Compare August 24, 2023 17:43

Aryan-Verma reviewed Aug 27, 2023

View reviewed changes

priyabhatnagar25 force-pushed the e2e_bigquerymultitable branch from c5e3721 to e538b69 Compare August 28, 2023 07:25

bharatgulati force-pushed the e2e_bigquerymultitable branch 3 times, most recently from 89e0b0b to f0c2b16 Compare September 1, 2023 05:23

Vipinofficial11 marked this pull request as draft September 1, 2023 06:18

priyabhatnagar25 force-pushed the e2e_bigquerymultitable branch from f0c2b16 to bc7b626 Compare September 1, 2023 06:42

bharatgulati added build Trigger unit test build and removed build Trigger unit test build labels Oct 9, 2023

neerajsinghal05 force-pushed the e2e_bigquerymultitable branch from f9e4123 to 8b2ad66 Compare November 16, 2023 11:34

rahuldash171 added build Trigger unit test build and removed build Trigger unit test build labels Nov 16, 2023

priyabhatnagar25 force-pushed the e2e_bigquerymultitable branch 3 times, most recently from 4c9db2e to 74e2152 Compare November 22, 2023 15:38

priyabhatnagar25 force-pushed the e2e_bigquerymultitable branch 8 times, most recently from 170af4a to 568fff4 Compare December 5, 2023 10:43

rahuldash171 force-pushed the e2e_bigquerymultitable branch from 91af24c to ae0c3b0 Compare December 20, 2023 09:52

rahuldash171 added build Trigger unit test build and removed build Trigger unit test build labels Dec 20, 2023

rahuldash171 force-pushed the e2e_bigquerymultitable branch 4 times, most recently from d6ccc26 to d808e6f Compare December 20, 2023 14:27

rahuldash171 marked this pull request as ready for review December 21, 2023 08:49

rahuldash171 force-pushed the e2e_bigquerymultitable branch from d808e6f to 0d56779 Compare December 21, 2023 08:59

priyabhatnagar25 added 2 commits January 17, 2024 09:30

BQMT e2e Tests Scenarios

3579401

Removed Issue tag Plugin-1669 in Runtime scenarios and test runner file,

9740cc8

updated correct conn string, removed space in yml, removed verify goal in pom. changed sql connector java version to run in github actions and bqmt secret updated

Vipinofficial11 force-pushed the e2e_bigquerymultitable branch from 54d7577 to ae12e20 Compare January 17, 2024 04:02

Debug runner path failures for BQMT

f8611c0

Vipinofficial11 force-pushed the e2e_bigquerymultitable branch from ae12e20 to f8611c0 Compare January 17, 2024 04:11

		@@ -1,4 +1,4 @@
		# Copyright © 2021 Cask Data, Inc.
		# Copyright © 2023 Cask Data, Inc.

E2e bigquerymultitable #1277

Are you sure you want to change the base?

E2e bigquerymultitable #1277

Uh oh!

Conversation

itsmekumari commented Aug 18, 2023

Uh oh!

poojantcs commented Aug 22, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants