DataKit数据迁移3前置校验失败的处理
前置校验项目及不通过的处理
1、Kafka服务可用性检查：
使用jps在portal执行机上执行查看，保证以下三个服务进程存在
[test@dev-openeuler-arm ~]$ jps
3757401 SchemaRegistryMain
3757072 SupportedKafka
3756341 QuorumPeerMain
如果在portal以正常安装的情况下，三个服务进程异常终止，可使用如下命令启动三个服务进程。
-- 可以先执行停止kafka的命令，确保Kafka进程已停止，避免启动时出错，停止kafka的命令如下
java -Dpath=/{portal_path}/portal/ -Dorder=stop_kafka -Dskip=true -jar /{portal_path}/portal/portalControl-*-exec.jar
-- 启动kafka进程的命令
java -Dpath=/{portal_path}/portal/ -Dorder=start_kafka -Dskip=true -jar /{portal_path}/portal/portalControl-*-exec.jar
-- 出现类似如下日志信息，表示启动kafka成功
log4j: reset attribute= "false".
log4j: Threshold ="null".
log4j: Level value for root is [debug].
log4j: root level set to DEBUG
log4j: Class name: [org.apache.log4j.ConsoleAppender]
log4j: Parsing layout of class: "org.apache.log4j.PatternLayout"
log4j: Setting property [conversionPattern] to [%d{HH:mm:ss,SS} %-5p (%C{1}:%M) - %m%n].
log4j: Setting property [levelMin] to [INFO].
log4j: Setting property [levelMax] to [ERROR].
log4j: Setting property [acceptOnMatch] to [true].
log4j: Adding filter of type [class org.apache.log4j.varia.LevelRangeFilter] to appender named [log.console].
log4j: Adding appender named [log.console] to category [root].
log4j: Class name: [org.apache.log4j.DailyRollingFileAppender]
log4j: Setting property [file] to [/data/test/portal/portal//logs/portal_.log].
log4j: Setting property [append] to [true].
log4j: Setting property [datePattern] to [yyyy-MM-dd].
log4j: Parsing layout of class: "org.apache.log4j.PatternLayout"
log4j: Setting property [conversionPattern] to [%d{HH:mm:ss,SS} %-5p (%C{1}:%M) - %m%n].
log4j: Setting property [levelMin] to [INFO].
log4j: Setting property [levelMax] to [ERROR].
log4j: Setting property [acceptOnMatch] to [true].
log4j: Adding filter of type [class org.apache.log4j.varia.LevelRangeFilter] to appender named [log.file].
log4j: setFile called: /data/test/portal/portal//logs/portal_.log, true
log4j: setFile ended
log4j: Appender [log.file] to be rolled at midnight.
log4j: Adding appender named [log.file] to category [root].
19:35:00,492 INFO (ParamsUtils:initMigrationParamsFromProps) - properties = {awt.toolkit=sun.awt.X11.XToolkit, java.specification.version=11, sun.cpu.isalist=, sun.jnu.encoding=UTF-8, java.class.path=/data/test/portal/portal/portalControl-6.0.0rc1-exec.jar, java.vm.vendor=BiSheng, sun.arch.data.model=64, path=/data/test/portal/portal/, java.vendor.url=https://gitee.com/openeuler/bishengjdk-11/, user.timezone=Asia/Shanghai, os.name=Linux, java.vm.specification.version=11, sun.java.launcher=SUN_STANDARD, user.country=US, order=start_kafka, sun.boot.library.path=/data/test/env/java/bisheng-jdk-11.0.20/lib, sun.java.command=/data/test/portal/portal/portalControl-6.0.0rc1-exec.jar, jdk.debug=release, sun.cpu.endian=little, user.home=/home/test, user.language=en, java.specification.vendor=Oracle Corporation, java.version.date=2023-07-18, java.home=/data/test/env/java/bisheng-jdk-11.0.20, file.separator=/, java.vm.compressedOopsMode=Zero based, line.separator=
, java.specification.name=Java Platform API Specification, java.vm.specification.vendor=Oracle Corporation, java.awt.graphicsenv=sun.awt.X11GraphicsEnvironment, java.protocol.handler.pkgs=org.springframework.boot.loader, sun.management.compiler=HotSpot 64-Bit Tiered Compilers, java.runtime.version=11.0.20+11, user.name=test, skip=true, path.separator=:, os.version=4.19.90-2110.8.0.0119.oe1.aarch64, java.runtime.name=OpenJDK Runtime Environment, file.encoding=UTF-8, java.vm.name=OpenJDK 64-Bit Server VM, java.vendor.version=BiSheng, java.vendor.url.bug=https://gitee.com/openeuler/bishengjdk-11/issues/, java.io.tmpdir=/tmp, java.version=11.0.20, user.dir=/data/test/portal, os.arch=aarch64, java.vm.specification.name=Java Virtual Machine Specification, java.awt.printerjob=sun.print.PSPrinterJob, sun.os.patch.level=unknown, java.library.path=/data/test/portal/portal/tools/chameleon/chameleon-6.0.0rc1:/data/test/portal/portal/tools/chameleon/chameleon-5.1.1:/data/test/portal/portal/tools/chameleon/chameleon-6.0.0rc1:/data/test/portal/portal/tools/chameleon/chameleon-6.0.0rc1:/data/test/portal/portal/tools/chameleon/chameleon-6.0.0rc1:/data/test/portal/portal/tools/chameleon/chameleon-6.0.0:/data/xz_u2/base/opt/huawei/install/om/lib:/data/xz_u2/base/opt/huawei/install/om/script/gspylib/clib::/usr/java/packages/lib:/lib:/usr/lib:/usr/lib64:/lib64, java.vm.info=mixed mode, java.vendor=BiSheng, java.vm.version=11.0.20+11, java.specification.maintenance.version=2, sun.io.unicode.encoding=UnicodeLittle, java.class.version=55.0}
19:35:00,567 INFO (MigrationConfluentInstanceConfig:getSystemParamAndParseEntity) - get MigrationConfluentInstanceConfig from system param = MigrationConfluentInstanceConfig(id=null, zookeeperPort=null, kafkaPort=null, zkIp=null, kafkaIp=null, installDir=null, bindPortalId=null, zkIpPort=null, kafkaIpPort=null, schemaRegistryIpPort=null, schemaRegistryIp=null, schemaRegistryPort=null, bindPortalHost=null, thirdPartySoftwareConfigType=null)
19:35:00,569 INFO (KafkaUtils:changeConfluentDirFromSysParam) - no need change param
19:35:00,576 INFO (FileUtils:createFile) - File /data/test/portal/portal/portal.portId.lock already exists.
19:35:00,587 INFO (FileUtils:createFile) - File /data/test/portal/portal/workspace/1 already exists.
19:35:00,587 INFO (FileUtils:createFile) - File /data/test/portal/portal/workspace/1/tmp/ already exists.
19:35:00,592 INFO (FileUtils:createFile) - File /data/test/portal/portal/workspace/1/logs already exists.
19:35:00,631 INFO (FileUtils:createFile) - File /data/test/portal/portal/workspace/1/status/ already exists.
19:35:00,632 INFO (FileUtils:createFile) - File /data/test/portal/portal/workspace/1/status/incremental/ already exists.
19:35:00,632 INFO (FileUtils:createFile) - File /data/test/portal/portal/workspace/1/status/portal.txt already exists.
19:35:00,632 INFO (FileUtils:createFile) - File /data/test/portal/portal/workspace/1/status/full_migration.txt already exists.
19:35:00,632 INFO (FileUtils:createFile) - File /data/test/portal/portal/workspace/1/status/incremental_migration.txt already exists.
19:35:00,632 INFO (FileUtils:createFile) - File /data/test/portal/portal/workspace/1/status/reverse_migration.txt already exists.
19:35:00,632 INFO (FileUtils:createFile) - File /data/test/portal/portal/workspace/1/logs/debezium/ already exists.
19:35:00,633 INFO (FileUtils:createFile) - File /data/test/portal/portal/workspace/1/logs/datacheck/ already exists.
19:35:00,650 INFO (ParamsUtils:changeDatacheckLogLevel) - global log level param is empty
19:35:00,796 INFO (MigrationConfluentInstanceConfig:getSystemParamAndParseEntity) - get MigrationConfluentInstanceConfig from system param = MigrationConfluentInstanceConfig(id=null, zookeeperPort=null, kafkaPort=null, zkIp=null, kafkaIp=null, installDir=null, bindPortalId=null, zkIpPort=null, kafkaIpPort=null, schemaRegistryIpPort=null, schemaRegistryIp=null, schemaRegistryPort=null, bindPortalHost=null, thirdPartySoftwareConfigType=null)
19:35:00,851 INFO (RuntimeExecUtils:executeStartOrder) - start command = /data/test/portal//portal/tools/debezium/confluent-5.5.1/bin/zookeeper-server-start -daemon /data/test/portal//portal/tools/debezium/confluent-5.5.1/etc/kafka/zookeeper.properties
19:35:00,938 INFO (RuntimeExecUtils:executeStartOrder) - Start zookeeper.
19:35:02,964 INFO (MqTool:start) - kafkaOrder====/data/test/portal//portal/tools/debezium/confluent-5.5.1/bin/kafka-topics --list --bootstrap-server 192.168.0.118:9092
19:35:03,07 INFO (RuntimeExecUtils:executeStartOrder) - start command = /data/test/portal//portal/tools/debezium/confluent-5.5.1/bin/kafka-server-start -daemon /data/test/portal//portal/tools/debezium/confluent-5.5.1/etc/kafka/server.properties
19:35:03,93 INFO (RuntimeExecUtils:executeStartOrder) - Start kafka.
19:35:05,102 INFO (RuntimeExecUtils:removeFile) - Remove file /data/test/portal/portal/tmp/test_.txt finished.
19:35:07,108 INFO (RuntimeExecUtils:removeFile) - Remove file /data/test/portal/portal/tmp/test_.txt finished.
19:35:07,174 INFO (RuntimeExecUtils:executeStartOrder) - start command = /data/test/portal//portal/tools/debezium/confluent-5.5.1/bin/schema-registry-start -daemon /data/test/portal//portal/tools/debezium/confluent-5.5.1/etc/schema-registry/schema-registry.properties
19:35:07,181 INFO (RuntimeExecUtils:executeStartOrder) - Start kafka schema registry.
19:35:10,281 INFO (MqTool:start) - Start kafka success.
2、检查源端和目标端数据库是否可连接
MySQL: mysql -h ip -P port -u user -ppassword -S /~/mysql.sock\
OpenGauss: gsql -r -d database -p port -U user -W password
3、权限检查
使用jps在portal执行机上执行查看，保证以下三个服务进程存在
[test@dev-openeuler-arm ~]$ jps
3757401 SchemaRegistryMain
3757072 SupportedKafka
3756341 QuorumPeerMain
如果在portal以正常安装的情况下，三个服务进程异常终止，可使用如下命令启动三个服务进程。
-- 可以先执行停止kafka的命令，确保Kafka进程已停止，避免启动时出错，停止kafka的命令如下
java -Dpath=/{portal_path}/portal/ -Dorder=stop_kafka -Dskip=true -jar /{portal_path}/portal/portalControl-*-exec.jar
-- 启动kafka进程的命令
java -Dpath=/{portal_path}/portal/ -Dorder=start_kafka -Dskip=true -jar /{portal_path}/portal/portalControl-*-exec.jar
-- 出现类似如下日志信息，表示启动kafka成功
log4j: reset attribute= "false".
log4j: Threshold ="null".
log4j: Level value for root is [debug].
log4j: root level set to DEBUG
log4j: Class name: [org.apache.log4j.ConsoleAppender]
log4j: Parsing layout of class: "org.apache.log4j.PatternLayout"
log4j: Setting property [conversionPattern] to [%d{HH:mm:ss,SS} %-5p (%C{1}:%M) - %m%n].
log4j: Setting property [levelMin] to [INFO].
log4j: Setting property [levelMax] to [ERROR].
log4j: Setting property [acceptOnMatch] to [true].
log4j: Adding filter of type [class org.apache.log4j.varia.LevelRangeFilter] to appender named [log.console].
log4j: Adding appender named [log.console] to category [root].
log4j: Class name: [org.apache.log4j.DailyRollingFileAppender]
log4j: Setting property [file] to [/data/test/portal/portal//logs/portal_.log].
log4j: Setting property [append] to [true].
log4j: Setting property [datePattern] to [yyyy-MM-dd].
log4j: Parsing layout of class: "org.apache.log4j.PatternLayout"
log4j: Setting property [conversionPattern] to [%d{HH:mm:ss,SS} %-5p (%C{1}:%M) - %m%n].
log4j: Setting property [levelMin] to [INFO].
log4j: Setting property [levelMax] to [ERROR].
log4j: Setting property [acceptOnMatch] to [true].
log4j: Adding filter of type [class org.apache.log4j.varia.LevelRangeFilter] to appender named [log.file].
log4j: setFile called: /data/test/portal/portal//logs/portal_.log, true
log4j: setFile ended
log4j: Appender [log.file] to be rolled at midnight.
log4j: Adding appender named [log.file] to category [root].
19:35:00,492 INFO (ParamsUtils:initMigrationParamsFromProps) - properties = {awt.toolkit=sun.awt.X11.XToolkit, java.specification.version=11, sun.cpu.isalist=, sun.jnu.encoding=UTF-8, java.class.path=/data/test/portal/portal/portalControl-6.0.0rc1-exec.jar, java.vm.vendor=BiSheng, sun.arch.data.model=64, path=/data/test/portal/portal/, java.vendor.url=https://gitee.com/openeuler/bishengjdk-11/, user.timezone=Asia/Shanghai, os.name=Linux, java.vm.specification.version=11, sun.java.launcher=SUN_STANDARD, user.country=US, order=start_kafka, sun.boot.library.path=/data/test/env/java/bisheng-jdk-11.0.20/lib, sun.java.command=/data/test/portal/portal/portalControl-6.0.0rc1-exec.jar, jdk.debug=release, sun.cpu.endian=little, user.home=/home/test, user.language=en, java.specification.vendor=Oracle Corporation, java.version.date=2023-07-18, java.home=/data/test/env/java/bisheng-jdk-11.0.20, file.separator=/, java.vm.compressedOopsMode=Zero based, line.separator=
, java.specification.name=Java Platform API Specification, java.vm.specification.vendor=Oracle Corporation, java.awt.graphicsenv=sun.awt.X11GraphicsEnvironment, java.protocol.handler.pkgs=org.springframework.boot.loader, sun.management.compiler=HotSpot 64-Bit Tiered Compilers, java.runtime.version=11.0.20+11, user.name=test, skip=true, path.separator=:, os.version=4.19.90-2110.8.0.0119.oe1.aarch64, java.runtime.name=OpenJDK Runtime Environment, file.encoding=UTF-8, java.vm.name=OpenJDK 64-Bit Server VM, java.vendor.version=BiSheng, java.vendor.url.bug=https://gitee.com/openeuler/bishengjdk-11/issues/, java.io.tmpdir=/tmp, java.version=11.0.20, user.dir=/data/test/portal, os.arch=aarch64, java.vm.specification.name=Java Virtual Machine Specification, java.awt.printerjob=sun.print.PSPrinterJob, sun.os.patch.level=unknown, java.library.path=/data/test/portal/portal/tools/chameleon/chameleon-6.0.0rc1:/data/test/portal/portal/tools/chameleon/chameleon-5.1.1:/data/test/portal/portal/tools/chameleon/chameleon-6.0.0rc1:/data/test/portal/portal/tools/chameleon/chameleon-6.0.0rc1:/data/test/portal/portal/tools/chameleon/chameleon-6.0.0rc1:/data/test/portal/portal/tools/chameleon/chameleon-6.0.0:/data/xz_u2/base/opt/huawei/install/om/lib:/data/xz_u2/base/opt/huawei/install/om/script/gspylib/clib::/usr/java/packages/lib:/lib:/usr/lib:/usr/lib64:/lib64, java.vm.info=mixed mode, java.vendor=BiSheng, java.vm.version=11.0.20+11, java.specification.maintenance.version=2, sun.io.unicode.encoding=UnicodeLittle, java.class.version=55.0}
19:35:00,567 INFO (MigrationConfluentInstanceConfig:getSystemParamAndParseEntity) - get MigrationConfluentInstanceConfig from system param = MigrationConfluentInstanceConfig(id=null, zookeeperPort=null, kafkaPort=null, zkIp=null, kafkaIp=null, installDir=null, bindPortalId=null, zkIpPort=null, kafkaIpPort=null, schemaRegistryIpPort=null, schemaRegistryIp=null, schemaRegistryPort=null, bindPortalHost=null, thirdPartySoftwareConfigType=null)
19:35:00,569 INFO (KafkaUtils:changeConfluentDirFromSysParam) - no need change param
19:35:00,576 INFO (FileUtils:createFile) - File /data/test/portal/portal/portal.portId.lock already exists.
19:35:00,587 INFO (FileUtils:createFile) - File /data/test/portal/portal/workspace/1 already exists.
19:35:00,587 INFO (FileUtils:createFile) - File /data/test/portal/portal/workspace/1/tmp/ already exists.
19:35:00,592 INFO (FileUtils:createFile) - File /data/test/portal/portal/workspace/1/logs already exists.
19:35:00,631 INFO (FileUtils:createFile) - File /data/test/portal/portal/workspace/1/status/ already exists.
19:35:00,632 INFO (FileUtils:createFile) - File /data/test/portal/portal/workspace/1/status/incremental/ already exists.
19:35:00,632 INFO (FileUtils:createFile) - File /data/test/portal/portal/workspace/1/status/portal.txt already exists.
19:35:00,632 INFO (FileUtils:createFile) - File /data/test/portal/portal/workspace/1/status/full_migration.txt already exists.
19:35:00,632 INFO (FileUtils:createFile) - File /data/test/portal/portal/workspace/1/status/incremental_migration.txt already exists.
19:35:00,632 INFO (FileUtils:createFile) - File /data/test/portal/portal/workspace/1/status/reverse_migration.txt already exists.
19:35:00,632 INFO (FileUtils:createFile) - File /data/test/portal/portal/workspace/1/logs/debezium/ already exists.
19:35:00,633 INFO (FileUtils:createFile) - File /data/test/portal/portal/workspace/1/logs/datacheck/ already exists.
19:35:00,650 INFO (ParamsUtils:changeDatacheckLogLevel) - global log level param is empty
19:35:00,796 INFO (MigrationConfluentInstanceConfig:getSystemParamAndParseEntity) - get MigrationConfluentInstanceConfig from system param = MigrationConfluentInstanceConfig(id=null, zookeeperPort=null, kafkaPort=null, zkIp=null, kafkaIp=null, installDir=null, bindPortalId=null, zkIpPort=null, kafkaIpPort=null, schemaRegistryIpPort=null, schemaRegistryIp=null, schemaRegistryPort=null, bindPortalHost=null, thirdPartySoftwareConfigType=null)
19:35:00,851 INFO (RuntimeExecUtils:executeStartOrder) - start command = /data/test/portal//portal/tools/debezium/confluent-5.5.1/bin/zookeeper-server-start -daemon /data/test/portal//portal/tools/debezium/confluent-5.5.1/etc/kafka/zookeeper.properties
19:35:00,938 INFO (RuntimeExecUtils:executeStartOrder) - Start zookeeper.
19:35:02,964 INFO (MqTool:start) - kafkaOrder====/data/test/portal//portal/tools/debezium/confluent-5.5.1/bin/kafka-topics --list --bootstrap-server 192.168.0.118:9092
19:35:03,07 INFO (RuntimeExecUtils:executeStartOrder) - start command = /data/test/portal//portal/tools/debezium/confluent-5.5.1/bin/kafka-server-start -daemon /data/test/portal//portal/tools/debezium/confluent-5.5.1/etc/kafka/server.properties
19:35:03,93 INFO (RuntimeExecUtils:executeStartOrder) - Start kafka.
19:35:05,102 INFO (RuntimeExecUtils:removeFile) - Remove file /data/test/portal/portal/tmp/test_.txt finished.
19:35:07,108 INFO (RuntimeExecUtils:removeFile) - Remove file /data/test/portal/portal/tmp/test_.txt finished.
19:35:07,174 INFO (RuntimeExecUtils:executeStartOrder) - start command = /data/test/portal//portal/tools/debezium/confluent-5.5.1/bin/schema-registry-start -daemon /data/test/portal//portal/tools/debezium/confluent-5.5.1/etc/schema-registry/schema-registry.properties
19:35:07,181 INFO (RuntimeExecUtils:executeStartOrder) - Start kafka schema registry.
19:35:10,281 INFO (MqTool:start) - Start kafka success.
MySQL: mysql -h ip -P port -u user -ppassword -S /~/mysql.sock\ OpenGauss: gsql -r -d database -p port -U user -W password
