Datax助力輕鬆遷移SQLServer數據至GreatSQL

来源:https://www.cnblogs.com/greatsql/p/18150097
-Advertisement-
Play Games

1.環境說明 1.1源端SQLSserver 版本 IP 埠 Microsoft SQL Server 2017 192.168.140.160 1433 1.2目標端GreatSQL 版本 IP 埠 GreatSQL-8.0.32 192.168.139.86 3308 2.安裝環境 2.1安 ...


1.環境說明

1.1源端SQLSserver

版本 IP 埠
Microsoft SQL Server 2017 192.168.140.160 1433

1.2目標端GreatSQL

版本 IP 埠
GreatSQL-8.0.32 192.168.139.86 3308

2.安裝環境

2.1安裝SQLServer環境

環境說明：藉助Docker使用鏡像啟動資料庫

2.1.1安裝docker

1.安裝基礎軟體包

$ yum install -y wget net-tools nfs-utils lrzsz gcc gcc-c++ make cmake libxml2-devel openssl-devel curl curl-devel unzip sudo ntp libaio-devel wget vim ncurses-devel autoconf automake zlib-devel python-devel epel-release openssh-server socat ipvsadm conntrack yum-utils

2.配置 docker-ce 國內 yum 源（阿裡雲）

$ yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

3.安裝 docker 依賴包

$ yum install -y  device-mapper-persistent-data lvm2

4.安裝 docker-ce

$ yum install docker-ce -y

5.啟動容器

$ systemctl start docker && systemctl enable docker

2.1.2 拉取鏡像

$ docker pull mcr.microsoft.com/mssql/server:2017-latest

2.1.3運行容器

$ docker run -e "ACCEPT_EULA=Y" -e "SA_PASSWORD=********" \
-p 1433:1433 --name sqlserver2017 \
-d mcr.microsoft.com/mssql/server:2017-latest

此處記得密碼設置為複雜密碼

參數解釋：

  • -e "ACCEPT_EULA=Y"：預設選擇同意協議許可證

  • -e "SA_PASSWORD=********":設置連接密碼，密碼不能太短太簡單，否則不滿足sqlserver密碼規範，容器會停止運行

  • -p 1433:1433: 宿主機埠映射到容器埠（前者為宿主機）

  • --name sqlserver2017：容器別名

  • -d: 後臺運行

  • mcr.microsoft.com/mssql/server:2017-latest：鏡像名稱:標簽

2.1.4使用資料庫

1.進入容器

$ docker exec -it sqlserver2017 bash

2.連接資料庫

$ /opt/mssql-tools/bin/sqlcmd -S localhost -U SA -P "********"

3.查詢資料庫

1> select name from sys.Databases;
2> go

4.創建資料庫

1> create database testdb;
2> go

5.創建表並插入數據

use testdb
create table t1(id int)
go
Insert into t1 values(1),(2)
go

2.2安裝GreatSQL環境

使用Docker鏡像進行安裝，直接拉取GreatSQL鏡像即可

$ docker pull greatsql/greatsql

並創建GreatSQL容器

$ docker run -d --name greatsql --hostname=greatsql -e MYSQL_ALLOW_EMPTY_PASSWORD=1 greatsql/greatsql

2.3安裝datax

DataX安裝需要依賴的環境

  • JDK(1.8以上，推薦1.8)

  • Python(推薦Python2.6.X及以上)

安裝步驟，解壓即用，但是解壓完成不做其他操作運行job會報錯

$ cd /soft

$ ll
total 3764708
-rw-r--r--  1 root root  853734462 Dec  9 04:06 datax.tar.gz

$ tar xf datax.tar.gz

$ python /soft/datax/bin/datax.py /soft/datax/job/job.json 

DataX (DATAX-OPENSOURCE-3.0), From Alibaba !
Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved.
2023-07-19 11:19:17.483 [main] WARN  ConfigParser - 插件[streamreader,streamwriter]載入失敗，1s後重試... Exception:Code:[Common-00], Describe:[您提供的配置文件存在錯誤信息，請檢查您的作業配置 .] - 配置信息錯誤，您提供的配置文件[/soft/datax/plugin/reader/._mysqlreader/plugin.json]不存在. 請檢查您的配置文件. 
2023-07-19 11:19:18.488 [main] ERROR Engine - 
經DataX智能分析,該任務最可能的錯誤原因是:
com.alibaba.datax.common.exception.DataXException: Code:[Common-00], Describe:[您提供的配置文件存在錯誤信息，請檢查您的作業配置 .] - 配置信息錯誤，您提供的配置文件[/soft/datax/plugin/reader/._mysqlreader/plugin.json]不存在. 請檢查您的配置文件.
at com.alibaba.datax.common.exception.DataXException.asDataXException(DataXException.java:26)
at com.alibaba.datax.common.util.Configuration.from(Configuration.java:95)
at com.alibaba.datax.core.util.ConfigParser.parseOnePluginConfig(ConfigParser.java:153)
at com.alibaba.datax.core.util.ConfigParser.parsePluginConfig(ConfigParser.java:125)
at com.alibaba.datax.core.util.ConfigParser.parse(ConfigParser.java:63)
at com.alibaba.datax.core.Engine.entry(Engine.java:137)
at com.alibaba.datax.core.Engine.main(Engine.java:204)

解決報錯:刪除plugin目錄下以及plugin/reader和plugin/writer目錄下的所有._開頭的文件

需要刪除三個目錄下的隱藏文件

  • plugin/
  • plugin/reader/
  • plugin/writer/
$ rm -rf /opt/app/datax/plugin/._*
$ rm -rf /opt/app/datax/plugin/reader/._*
$ rm -rf /opt/app/datax/plugin/writer/._*

運行一個測例，檢測datax是否安裝成功

$ python /soft/datax/bin/datax.py  /soft/datax/job/job.json 

DataX (DATAX-OPENSOURCE-3.0), From Alibaba !
Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved.
 2023-07-19 11:22:12.298 [main] INFO  VMInfo - VMInfo# operatingSystem class => sun.management.OperatingSystemImpl
2023-07-19 11:22:12.305 [main] INFO  Engine - the machine info  => 
osInfo: Oracle Corporation 1.8 25.251-b08
jvmInfo: Linux amd64 4.19.25-200.1.el7.bclinux.x86_64
cpu num: 48
totalPhysicalMemory: -0.00G
freePhysicalMemory: -0.00G
maxFileDescriptorCount: -1
currentOpenFileDescriptorCount: -1
GC Names [PS MarkSweep, PS Scavenge]

MEMORY_NAME                    | allocation_size                | init_size                      
PS Eden Space                  | 256.00MB                       | 256.00MB                       
Code Cache                     | 240.00MB                       | 2.44MB                         
Compressed Class Space         | 1,024.00MB                     | 0.00MB                         
PS Survivor Space              | 42.50MB                        | 42.50MB                        
PS Old Gen                     | 683.00MB                       | 683.00MB                       
Metaspace                      | -0.00MB                        | 0.00MB  

2023-07-19 11:22:12.320 [main] INFO  Engine - 
{"content":[{"reader":{"name":"streamreader",
"parameter":{"column":[
{"type":"string","value":"DataX"},
{"type":"long","value":19890604},
{"type":"date","value":"1989-06-04 00:00:00"},
{"type":"bool","value":true},
{"type":"bytes","value":"test"}
],"sliceRecordCount":100000}
},"writer":{"name":"streamwriter","parameter":{"encoding":"UTF-8","print":false}}}],
"setting":{"errorLimit":{"percentage":0.02,"record":0},
"speed":{"byte":10485760}}}

2023-07-19 11:22:12.336 [main] WARN  Engine - prioriy set to 0, because NumberFormatException, the value is: null
2023-07-19 11:22:12.337 [main] INFO  PerfTrace - PerfTrace traceId=job_-1, isEnable=false, priority=0
2023-07-19 11:22:12.338 [main] INFO  JobContainer - DataX jobContainer starts job.
2023-07-19 11:22:12.339 [main] INFO  JobContainer - Set jobId = 0
2023-07-19 11:22:12.352 [job-0] INFO  JobContainer - jobContainer starts to do prepare ...
2023-07-19 11:22:12.352 [job-0] INFO  JobContainer - DataX Reader.Job [streamreader] do prepare work .
2023-07-19 11:22:12.352 [job-0] INFO  JobContainer - DataX Writer.Job [streamwriter] do prepare work .
2023-07-19 11:22:12.352 [job-0] INFO  JobContainer - jobContainer starts to do split ...
2023-07-19 11:22:12.353 [job-0] INFO  JobContainer - Job set Max-Byte-Speed to 10485760 bytes.
2023-07-19 11:22:12.354 [job-0] INFO  JobContainer - DataX Reader.Job [streamreader] splits to [1] tasks.
2023-07-19 11:22:12.354 [job-0] INFO  JobContainer - DataX Writer.Job [streamwriter] splits to [1] tasks.
2023-07-19 11:22:12.371 [job-0] INFO  JobContainer - jobContainer starts to do schedule ...
2023-07-19 11:22:12.375 [job-0] INFO  JobContainer - Scheduler starts [1] taskGroups.
2023-07-19 11:22:12.376 [job-0] INFO  JobContainer - Running by standalone Mode.
2023-07-19 11:22:12.384 [taskGroup-0] INFO  TaskGroupContainer - taskGroupId=[0] start [1] channels for [1] tasks.
2023-07-19 11:22:12.388 [taskGroup-0] INFO  Channel - Channel set byte_speed_limit to -1, No bps activated.
2023-07-19 11:22:12.388 [taskGroup-0] INFO  Channel - Channel set record_speed_limit to -1, No tps activated.
2023-07-19 11:22:12.396 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[0] attemptCount[1] is started
2023-07-19 11:22:12.697 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[0] is successed, used[302]ms
2023-07-19 11:22:12.698 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] completed it's tasks.
2023-07-19 11:22:22.402 [job-0] INFO  StandAloneJobContainerCommunicator - Total 100000 records, 2600000 bytes | Speed 253.91KB/s, 10000 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.020s |  All Task WaitReaderTime 0.033s | Percentage 100.00%
2023-07-19 11:22:22.402 [job-0] INFO  AbstractScheduler - Scheduler accomplished all tasks.
2023-07-19 11:22:22.402 [job-0] INFO  JobContainer - DataX Writer.Job [streamwriter] do post work.
2023-07-19 11:22:22.403 [job-0] INFO  JobContainer - DataX Reader.Job [streamreader] do post work.
2023-07-19 11:22:22.403 [job-0] INFO  JobContainer - DataX jobId [0] completed successfully.
2023-07-19 11:22:22.403 [job-0] INFO  HookInvoker - No hook invoked, because base dir not exists or is a file: /soft/datax/hook
2023-07-19 11:22:22.404 [job-0] INFO  JobContainer - 
 [total cpu info] => 
averageCpu                     | maxDeltaCpu                    | minDeltaCpu                    
-1.00%                         | -1.00%                         | -1.00%
 [total gc info] => 
 NAME                 | totalGCCount       | maxDeltaGCCount    | minDeltaGCCount    | totalGCTime        | maxDeltaGCTime     | minDeltaGCTime     
 PS MarkSweep         | 0                  | 0                  | 0                  | 0.000s             | 0.000s             | 0.000s             
 PS Scavenge          | 0                  | 0                  | 0                  | 0.000s             | 0.000s             | 0.000s             

2023-07-19 11:22:22.404 [job-0] INFO  JobContainer - PerfTrace not enable!
2023-07-19 11:22:22.404 [job-0] INFO  StandAloneJobContainerCommunicator - Total 100000 records, 2600000 bytes | Speed 253.91KB/s, 10000 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.020s |  All Task WaitReaderTime 0.033s | Percentage 100.00%
2023-07-19 11:22:22.406 [job-0] INFO  JobContainer - 
任務啟動時刻                    : 2023-07-19 11:22:12
任務結束時刻                    : 2023-07-19 11:22:22
任務總計耗時                    :                 10s
任務平均流量                    :          253.91KB/s
記錄寫入速度                    :          10000rec/s
讀出記錄總數                    :              100000
讀寫失敗總數                    :                   0

3.SQLServer2GreatSQL全量遷移

3.1 源端（SQLServer）造測試數據

$ docker exec -it 47bd0ed79c26 /bin/bash

$ /opt/mssql-tools/bin/sqlcmd -S localhost -U SA -P "********"

1> create database testdb
1> use testdb
1> insert into t1 values(1),(2),(3);
2> go
1> select * from t1;
2> go
id         
\-----------
​          1
​          2
​          3

3.2 目標端（GreatSQL）創建表結構

greatsql> create database testdb;
greatsql> use testdb;
greatsql> create table t1 (id int primary key);

3.3 編寫Datax的job文件

$ cat /soft/datax/job/sqlserver_to_greatsql.json
{
​    "job": {
​        "content": [
​            {
​                "reader": {
​                    "name": "sqlserverreader",
​                    "parameter": {
​                        "connection": [
​                            {
​                                "jdbcUrl": ["jdbc:sqlserver://127.0.0.1:1433;DatabaseName=testdb"],
​                                "table": ["t1"]
​                            }
​                        ],
​                        "password": "********",
​                        "username": "SA",
​                        "column": ["*"]
​                    }
​                },
​                "writer": {
​                    "name": "mysqlwriter",
​                    "parameter": {
​                        "column": ["*"],
​                        "connection": [
​                            {
​                                "jdbcUrl": "jdbc:mysql://10.17.139.86:3308/testdb",
​                                "table": ["t1"]
​                            }
​                       ],
​                        "password": "******",
​                        "session": [],
​                        "username": "admin",
​                        "writeMode": "insert"
​                    }
​                }
​            }
​        ],
​        "setting": {
​            "speed": {
​                "channel": "5"
​            }
​        }
​    }
}

3.4 運行Datax遷移任務

$ python /soft/datax/bin/datax.py /soft/datax/job/sqlserver_to_greatsql.json 

DataX (DATAX-OPENSOURCE-3.0), From Alibaba !
Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved.

2023-11-28 09:58:44.087 [main] INFO  VMInfo - VMInfo# operatingSystem class => sun.management.OperatingSystemImpl
2023-11-28 09:58:44.104 [main] INFO  Engine - the machine info  => 
osInfo: Oracle Corporation 1.8 25.181-b13
jvmInfo: Linux amd64 3.10.0-957.el7.x86_64

cpu num: 8
totalPhysicalMemory: -0.00G
freePhysicalMemory: -0.00G
maxFileDescriptorCount: -1
currentOpenFileDescriptorCount: -1
GC Names [PS MarkSweep, PS Scavenge]
MEMORY_NAME                    | allocation_size                | init_size                      
PS Eden Space                  | 256.00MB                       | 256.00MB                       
Code Cache                     | 240.00MB                       | 2.44MB                         
Compressed Class Space         | 1,024.00MB                     | 0.00MB                         
PS Survivor Space              | 42.50MB                        | 42.50MB                        
PS Old Gen                     | 683.00MB                       | 683.00MB                       
Metaspace                      | -0.00MB                        | 0.00MB                         

2023-11-28 09:58:44.137 [main] INFO  Engine - 
{
"content":[
{"reader":{
"name":"sqlserverreader",
"parameter":{
"column":["*"],
"connection":[
{"jdbcUrl":["jdbc:sqlserver://127.0.0.1:1433;DatabaseName=testdb"],
"table":["t1"]}],
"password":"*************",
"username":"SA"}},

"writer":{"name":"mysqlwriter","parameter":{"column":["*"],

"connection":[{"jdbcUrl":"jdbc:mysql://10.17.139.86:3308/testdb",
"table":["t1"]}],
"password":"********",
"session":[],
"username":"admin",
"writeMode":"insert"}}}],
"setting":{"speed":{"channel":"5"}}}

2023-11-28 09:58:44.176 [main] WARN  Engine - prioriy set to 0, because NumberFormatException, the value is: null
2023-11-28 09:58:44.179 [main] INFO  PerfTrace - PerfTrace traceId=job_-1, isEnable=false, priority=0
2023-11-28 09:58:44.180 [main] INFO  JobContainer - DataX jobContainer starts job.
2023-11-28 09:58:44.183 [main] INFO  JobContainer - Set jobId = 0
2023-11-28 09:58:44.542 [job-0] INFO  OriginalConfPretreatmentUtil - Available jdbcUrl:jdbc:sqlserver://127.0.0.1:1433;DatabaseName=testdb.
2023-11-28 09:58:44.544 [job-0] WARN  OriginalConfPretreatmentUtil - 您的配置文件中的列配置存在一定的風險. 因為您未配置讀取資料庫表的列，當您的表欄位個數、類型有變動時，可能影響任務正確性甚至會運行出錯。請檢查您的配置並作出修改.
Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.
2023-11-28 09:58:45.099 [job-0] INFO  OriginalConfPretreatmentUtil - table:[t1] all columns:[id].
2023-11-28 09:58:45.099 [job-0] WARN  OriginalConfPretreatmentUtil - 您的配置文件中的列配置信息存在風險. 因為您配置的寫入資料庫表的列為*，當您的表欄位個數、類型有變動時，可能影響任務正確性甚至會運行出錯。請檢查您的配置並作出修改.
2023-11-28 09:58:45.102 [job-0] INFO  OriginalConfPretreatmentUtil - Write data [
insert INTO %s (id) VALUES(?)
], which jdbcUrl like:[jdbc:mysql://10..17.139.86:16310/testdb?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]
2023-11-28 09:58:45.103 [job-0] INFO  JobContainer - jobContainer starts to do prepare ...
2023-11-28 09:58:45.103 [job-0] INFO  JobContainer - DataX Reader.Job [sqlserverreader] do prepare work 
2023-11-28 09:58:45.104 [job-0] INFO  JobContainer - DataX Writer.Job [mysqlwriter] do prepare work .
2023-11-28 09:58:45.104 [job-0] INFO  JobContainer - jobContainer starts to do split ...
2023-11-28 09:58:45.105 [job-0] INFO  JobContainer - Job set Channel-Number to 5 channels.
2023-11-28 09:58:45.112 [job-0] INFO  JobContainer - DataX Reader.Job [sqlserverreader] splits to [1] tasks.
2023-11-28 09:58:45.114 [job-0] INFO  JobContainer - DataX Writer.Job [mysqlwriter] splits to [1] tasks.
2023-11-28 09:58:45.135 [job-0] INFO  JobContainer - jobContainer starts to do schedule ...
2023-11-28 09:58:45.139 [job-0] INFO  JobContainer - Scheduler starts [1] taskGroups.
2023-11-28 09:58:45.142 [job-0] INFO  JobContainer - Running by standalone Mode.
2023-11-28 09:58:45.151 [taskGroup-0] INFO  TaskGroupContainer - taskGroupId=[0] start [1] channels for [1] tasks.
2023-11-28 09:58:45.157 [taskGroup-0] INFO  Channel - Channel set byte_speed_limit to -1, No bps activated.
2023-11-28 09:58:45.158 [taskGroup-0] INFO  Channel - Channel set record_speed_limit to -1, No tps activated.
2023-11-28 09:58:45.173 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[0] attemptCount[1] is started
2023-11-28 09:58:45.181 [0-0-0-reader] INFO  CommonRdbmsReader$Task - Begin to read record by Sql: [select * from t1 
] jdbcUrl:[jdbc:sqlserver://127.0.0.1:1433;DatabaseName=testdb].
2023-11-28 09:58:45.398 [0-0-0-reader] INFO  CommonRdbmsReader$Task - Finished read record by Sql: [select * from t1 
] jdbcUrl:[jdbc:sqlserver://127.0.0.1:1433;DatabaseName=testdb].
2023-11-28 09:58:45.454 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[0] is successed, used[284]ms
2023-11-28 09:58:45.455 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] completed it's tasks.
2023-11-28 09:58:55.175 [job-0] INFO  StandAloneJobContainerCommunicator - Total 3 records, 3 bytes | Speed 0B/s, 0 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.000s |  All Task WaitReaderTime 0.000s | Percentage 100.00%
2023-11-28 09:58:55.175 [job-0] INFO  AbstractScheduler - Scheduler accomplished all tasks.
2023-11-28 09:58:55.175 [job-0] INFO  JobContainer - DataX Writer.Job [mysqlwriter] do post work.
2023-11-28 09:58:55.176 [job-0] INFO  JobContainer - DataX Reader.Job [sqlserverreader] do post work.
2023-11-28 09:58:55.176 [job-0] INFO  JobContainer - DataX jobId [0] completed successfully.
2023-11-28 09:58:55.176 [job-0] INFO  HookInvoker - No hook invoked, because base dir not exists or is a file: /soft/datax/hook
2023-11-28 09:58:55.177 [job-0] INFO  JobContainer - 
 [total cpu info] => 
averageCpu                     | maxDeltaCpu                    | minDeltaCpu                    
-1.00%                         | -1.00%                         | -1.00%
 [total gc info] => 
 NAME                 | totalGCCount       | maxDeltaGCCount    | minDeltaGCCount    | totalGCTime        | maxDeltaGCTime     | minDeltaGCTime     
 PS MarkSweep         | 1                  | 1                  | 1                  | 0.061s             | 0.061s             | 0.061s             
 PS Scavenge          | 1                  | 1                  | 1                  | 0.039s             | 0.039s             | 0.039s             
2023-11-28 09:58:55.177 [job-0] INFO  JobContainer - PerfTrace not enable!
2023-11-28 09:58:55.177 [job-0] INFO  StandAloneJobContainerCommunicator - Total 3 records, 3 bytes | Speed 0B/s, 0 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.000s |  All Task WaitReaderTime 0.000s | Percentage 100.00%
2023-11-28 09:58:55.179 [job-0] INFO  JobContainer - 
任務啟動時刻                    : 2023-11-28 09:58:44
任務結束時刻                    : 2023-11-28 09:58:55
任務總計耗時                    :                 10s
任務平均流量                    :                0B/s
記錄寫入速度                    :              0rec/s
讀出記錄總數                    :                   3
讀寫失敗總數                    :                   0

3.5 到目標端驗證數據

greatsql> select * from t1;
+----+
| id |
+----+
|  1 |
|  2 |
|  3 |
+----+
3 rows in set (0.01 sec)

4. SQLServer to GreatSQL增量遷移

4.1 源端（SQLServer）創建測試數據

2> create table t2 (id int,createtime datetime);
3> go
1> insert into t2 values(1,GETDATE());
2> g
(1 rows affected)
1> insert into t2 values(2,GETDATE());
2> go
(1 rows affected)
1> insert into t2 values(3,GETDATE());
2> go
(1 rows affected)
1> insert into t2 values(4,GETDATE());
2> go
(1 rows affected)
1> insert into t2 values(5,GETDATE());
2> go
(1 rows affected)
1> insert into t2 values(6,GETDATE());
2> go
(1 rows affected)
1> select * from t2;
2> go
id          createtime             
---------- -----------------------
​          1 2023-11-28 02:18:20.790
​          2 2023-11-28 02:18:27.040
​          3 2023-11-28 02:18:32.103
​          4 2023-11-28 02:18:37.690
​          5 2023-11-28 02:18:41.450
​          6 2023-11-28 02:18:46.330

4.2 編寫Datax的全量遷移job文件

$ cat sqlserver_to_greatsql_inc.json 
{
​    "job": {
​        "content": [
​            {
​                "reader": {
​                    "name": "sqlserverreader",
​                    "parameter": {
​                        "connection": [
​                            {
​                                "jdbcUrl": ["jdbc:sqlserver://127.0.0.1:1433;DatabaseName=testdb"],
​                                "table": ["t2"]
​                            }
​                        ],
​                        "password": "********",
​                        "username": "SA",
​                        "column": ["*"]
​                    }
​                },
​                "writer": {
​                    "name": "mysqlwriter",
​                    "parameter": {
​                        "column": ["*"],
​                        "connection": [
​                            {
​                                "jdbcUrl": "jdbc:mysql://10.17.139.86:3308/testdb",
​                                "table": ["t2"]
​                            }
​                        ],
​                        "password": "!QAZ2wsx",
​                        "session": [],
​                        "username": "admin",
​                        "writeMode": "insert"
​                    }
​                }
​            }
​        ],
​        "setting": {
​            "speed": {
​                "channel": "5"
​            }
​        }
​    }
}

4.3 運行Datax全量遷移任務

$ python /soft/datax/bin/datax.py /soft/datax/job/sqlserver_to_greatsql_inc.json 

 DataX (DATAX-OPENSOURCE-3.0), From Alibaba !
Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved.
2023-11-28 10:19:59.279 [main] INFO  VMInfo - VMInfo# operatingSystem class => sun.management.OperatingSystemImpl
2023-11-28 10:19:59.286 [main] INFO  Engine - the machine info  => 
osInfo: Oracle Corporation 1.8 25.181-b13
jvmInfo: Linux amd64 3.10.0-957.el7.x86_64
cpu num: 8
totalPhysicalMemory: -0.00G
freePhysicalMemory: -0.00G
maxFileDescriptorCount: -1
currentOpenFileDescriptorCount: -1
GC Names [PS MarkSweep, PS Scavenge]

MEMORY_NAME                    | allocation_size                | init_size                      
PS Eden Space                  | 256.00MB                       | 256.00MB                       
Code Cache                     | 240.00MB                       | 2.44MB                         
Compressed Class Space         | 1,024.00MB                     | 0.00MB                         
PS Survivor Space              | 42.50MB                        | 42.50MB                        
PS Old Gen                     | 683.00MB                       | 683.00MB                       
Metaspace                      | -0.00MB                        | 0.00MB                         

2023-11-28 10:19:59.302 [main] INFO  Engine - 
{"content":[{"reader":{"name":"sqlserverreader","parameter":{"column":[
"*"],"connection":[{"jdbcUrl":["jdbc:sqlserver://127.0.0.1:1433;DatabaseName=testdb"],
"table":["t2"]}],"password":"*************","username":"SA"}},
"writer":{"name":"mysqlwriter","parameter":{"column":["*"],
"connection":[{"jdbcUrl":"jdbc:mysql://10..17.139.86:16310/testdb","table":["t2"]}],
"password":"********",
"session":[],
"username":"admin",
"writeMode":"insert"}}}],
"setting":{"speed":{"channel":"5"}}}

2023-11-28 10:19:59.319 [main] WARN  Engine - prioriy set to 0, because NumberFormatException, the value is: null
2023-11-28 10:19:59.321 [main] INFO  PerfTrace - PerfTrace traceId=job_-1, isEnable=false, priority=0
2023-11-28 10:19:59.321 [main] INFO  JobContainer - DataX jobContainer starts job.
2023-11-28 10:19:59.324 [main] INFO  JobContainer - Set jobId = 0
2023-11-28 10:19:59.629 [job-0] INFO  OriginalConfPretreatmentUtil - Available jdbcUrl:jdbc:sqlserver://127.0.0.1:1433;DatabaseName=testdb.
2023-11-28 10:19:59.630 [job-0] WARN  OriginalConfPretreatmentUtil - 您的配置文件中的列配置存在一定的風險. 因為您未配置讀取資料庫表的列，當您的表欄位個數、類型有變動時，可能影響任務正確性甚至會運行出錯。請檢查您的配置並作出修改.
Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.
2023-11-28 10:20:00.027 [job-0] INFO  OriginalConfPretreatmentUtil - table:[t2] all columns:[
id,createtime].
2023-11-28 10:20:00.027 [job-0] WARN  OriginalConfPretreatmentUtil - 您的配置文件中的列配置信息存在風險. 因為您配置的寫入資料庫表的列為*，當您的表欄位個數、類型有變動時，可能影響任務正確性甚至會運行出錯。請檢查您的配置並作出修改.
2023-11-28 10:20:00.029 [job-0] INFO  OriginalConfPretreatmentUtil - Write data [
insert INTO %s (id,createtime) VALUES(?,?)
], which jdbcUrl like:[jdbc:mysql://10..17.139.86:16310/testdb?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]
2023-11-28 10:20:00.030 [job-0] INFO  JobContainer - jobContainer starts to do prepare ...
2023-11-28 10:20:00.031 [job-0] INFO  JobContainer - DataX Reader.Job [sqlserverreader] do prepare work .
2023-11-28 10:20:00.031 [job-0] INFO  JobContainer - DataX Writer.Job [mysqlwriter] do prepare work .
2023-11-28 10:20:00.032 [job-0] INFO  JobContainer - jobContainer starts to do split ...
2023-11-28 10:20:00.032 [job-0] INFO  JobContainer - Job set Channel-Number to 5 channels.
2023-11-28 10:20:00.037 [job-0] INFO  JobContainer - DataX Reader.Job [sqlserverreader] splits to [1] tasks.
2023-11-28 10:20:00.038 [job-0] INFO  JobContainer - DataX Writer.Job [mysqlwriter] splits to [1] tasks.
2023-11-28 10:20:00.060 [job-0] INFO  JobContainer - jobContainer starts to do schedule ...
2023-11-28 10:20:00.063 [job-0] INFO  JobContainer - Scheduler starts [1] taskGroups.
2023-11-28 10:20:00.066 [job-0] INFO  JobContainer - Running by standalone Mode.
2023-11-28 10:20:00.073 [taskGroup-0] INFO  TaskGroupContainer - taskGroupId=[0] start [1] channels for [1] tasks.
2023-11-28 10:20:00.080 [taskGroup-0] INFO  Channel - Channel set byte_speed_limit to -1, No bps activated.
2023-11-28 10:20:00.080 [taskGroup-0] INFO  Channel - Channel set record_speed_limit to -1, No tps activated.
2023-11-28 10:20:00.093 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[0] attemptCount[1] is started
2023-11-28 10:20:00.101 [0-0-0-reader] INFO  CommonRdbmsReader$Task - Begin to read record by Sql: [select * from t2 
] jdbcUrl:[jdbc:sqlserver://127.0.0.1:1433;DatabaseName=testdb].
2023-11-28 10:20:00.262 [0-0-0-reader] INFO  CommonRdbmsReader$Task - Finished read record by Sql: [select * from t2 
] jdbcUrl:[jdbc:sqlserver://127.0.0.1:1433;DatabaseName=testdb].
2023-11-28 10:20:00.334 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[0] is successed, used[243]ms
2023-11-28 10:20:00.335 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] completed it's tasks.
2023-11-28 10:20:10.087 [job-0] INFO  StandAloneJobContainerCommunicator - Total 6 records, 54 bytes | Speed 5B/s, 0 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.000s |  All Task WaitReaderTime 0.000s | Percentage 100.00%
2023-11-28 10:20:10.088 [job-0] INFO  AbstractScheduler - Scheduler accomplished all tasks.
2023-11-28 10:20:10.088 [job-0] INFO  JobContainer - DataX Writer.Job [mysqlwriter] do post work.
2023-11-28 10:20:10.089 [job-0] INFO  JobContainer - DataX Reader.Job [sqlserverreader] do post work.
2023-11-28 10:20:10.090 [job-0] INFO  JobContainer - DataX jobId [0] completed successfully.
2023-11-28 10:20:10.091 [job-0] INFO  HookInvoker - No hook invoked, because base dir not exists or is a file: /soft/datax/hook
2023-11-28 10:20:10.094 [job-0] INFO  JobContainer - 
 [total cpu info] => 
averageCpu                     | maxDeltaCpu                    | minDeltaCpu                    
-1.00%                         | -1.00%                         | -1.00%                      
 [total gc info] => 
 NAME                 | totalGCCount       | maxDeltaGCCount    | minDeltaGCCount    | totalGCTime        | maxDeltaGCTime     | minDeltaGCTime     
 PS MarkSweep         | 1                  | 1                  | 1                  | 0.034s             | 0.034s             | 0.034s             
 PS Scavenge          | 1                  | 1                  | 1                  | 0.031s             | 0.031s             | 0.031s             

2023-11-28 10:20:10.094 [job-0] INFO  JobContainer - PerfTrace not enable!
2023-11-28 10:20:10.095 [job-0] INFO  StandAloneJobContainerCommunicator - Total 6 records, 54 bytes | Speed 5B/s, 0 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.000s |  All Task WaitReaderTime 0.000s | Percentage 100.00%
2023-11-28 10:20:10.097 [job-0] INFO  JobContainer - 
任務啟動時刻                    : 2023-11-28 10:19:59
任務結束時刻                    : 2023-11-28 10:20:10
任務總計耗時                    :                 10s
任務平均流量                    :                5B/s
記錄寫入速度                    :              0rec/s
讀出記錄總數                    :                   6
讀寫失敗總數                    :                   0

4.4 驗證全量遷移的數據

greatsql> select * from t2;
+----+---------------------+
| id | createtime          |
+----+---------------------+
|  1 | 2023-11-28 02:18:21 |
|  2 | 2023-11-28 02:18:27 |
|  3 | 2023-11-28 02:18:32 |
|  4 | 2023-11-28 02:18:38 |
|  5 | 2023-11-28 02:18:41 |
|  6 | 2023-11-28 02:18:46 |
+----+---------------------+

還可以用 checksum table x 來驗證，大表就不能全表select * 了

4.5 源端（SQLServer）插入增量數據

2> insert into t2 values(7,'202311-28 03:18:46.330');
3> go
Changed database context to 'jem_db'.
(1 rows affected)
1> insert into t2 values(8,'2023-11-28 03:20:46.330');
2> go
(1 rows affected)
1> insert into t2 values(9,'2023-11-28 03:25:46.330');
2> go
(1 rows affected)
1> insert into t2 values(10,'2023-11-28 03:30:46.330');
2> go
(1 rows affected)
1> select * from t2;
2> go
id          createtime             
----------- -----------------------
​          1 2023-11-28 02:18:20.790
​          2 2023-11-28 02:18:27.040
​          3 2023-11-28 02:18:32.103
​          4 2023-11-28 02:18:37.690
​          5 2023-11-28 02:18:41.450
​          6 2023-11-28 02:18:46.330
​          7 2023-11-28 03:18:46.330
​          8 2023-11-28 03:20:46.330
​          9 2023-11-28 03:25:46.330
​         10 2023-11-28 03:30:46.330

4.6 編寫Datax增量遷移job文件

$ cat sqlserver_to_greatsql_inc.json 
{
​    "job": {
​        "content": [
​            {
​                "reader": {
​                    "name": "sqlserverreader",
​                    "parameter": {
​                        "connection": [
​                            {
​                                "jdbcUrl": ["jdbc:sqlserver://127.0.0.1:1433;DatabaseName=testdb"],
​                                "table": ["t2"]
​                            }
​                        ],
​                        "password": "********",
​                        "username": "SA",
​                        "column": ["*"],
​                        "where":"createtime > '${start_time}' and createtime < '${end_time}'"
​                    }
​                },
​                "writer": {
​                    "name": "mysqlwriter",
​                    "parameter": {
​                        "column": ["*"],
​                        "connection": [
​                            {
​                                "jdbcUrl": "jdbc:mysql://10..17.139.86:16310/testdb",
​                                "table": ["t2"]
​                            }
​                        ],
​                        "password": "!QAZ2wsx",
​                        "session": [],
​                        "username": "admin",
​                        "writeMode": "insert"
​                    }
​                }
​            }
​        ],
​        "setting": {
​            "speed": {
​                "channel": "5"
​            }
​        }
​    }
}

4.7 運行Datax增量遷移任務

$ python /soft/datax/bin/datax.py /soft/datax/job/sqlserver_to_mysql_inc.json -p "-Dstart_time='2023-11-28 03:17:46.330' -Dend_time='2023-11-28 03:31:46.330'"

DataX (DATAX-OPENSOURCE-3.0), From Alibaba !
Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved.

2023-11-28 10:29:24.492 [main] INFO  VMInfo - VMInfo# operatingSystem class => sun.management.OperatingSystemImpl

2023-11-28 10:29:24.504 [main] INFO  Engine - the machine info  => 

osInfo: Oracle Corporation 1.8 25.181-b13
jvmInfo: Linux amd64 3.10.0-957.el7.x86_64
cpu num: 8

totalPhysicalMemory: -0.00G
freePhysicalMemory: -0.00G
maxFileDescriptorCount: -1
currentOpenFileDescriptorCount: -1
GC Names [PS MarkSweep, PS Scavenge]
MEMORY_NAME                    | allocation_size                | init_size                      
PS Eden Space                  | 256.00MB                       | 256.00MB                       
Code Cache                     | 240.00MB                       | 2.44MB                         
Compressed Class Space         | 1,024.00MB                     | 0.00MB                         
PS Survivor Space              | 42.50MB                        | 42.50MB                        
PS Old Gen                     | 683.00MB                       | 683.00MB                       
Metaspace                      | -0.00MB                        | 0.00MB                         

2023-11-28 10:29:24.524 [main] INFO  Engine - 
{"content":[{"reader":{"name":"sqlserverreader","parameter":{"column":["*"],
"connection":[{"jdbcUrl":["jdbc:sqlserver://127.0.0.1:1433;DatabaseName=testdb"],
"table":["t2"]}],"password":"*************","username":"SA",
"where":"createtime > '2023-11-28 03:17:46.330' and createtime < '2023-11-28 03:31:46.330'"}},
"writer":{"name":"mysqlwriter","parameter":{"column":["*"],"connection":[{"jdbcUrl":"jdbc:mysql://10..17.139.86:16310/testdb","table":["t2"]}],
"password":"********",
"session":[],
"username":"admin",
"writeMode":"insert"}}}],
"setting":{"speed":{"channel":"5"}}}

2023-11-28 10:29:24.542 [main] WARN  Engine - prioriy set to 0, because NumberFormatException, the value is: null
2023-11-28 10:29:24.544 [main] INFO  PerfTrace - PerfTrace traceId=job_-1, isEnable=false, priority=0
2023-11-28 10:29:24.544 [main] INFO  JobContainer - DataX jobContainer starts job.
2023-11-28 10:29:24.546 [main] INFO  JobContainer - Set jobId = 0
2023-11-28 10:29:24.830 [job-0] INFO  OriginalConfPretreatmentUtil - Available jdbcUrl:jdbc:sqlserver://127.0.0.1:1433;DatabaseName=testdb.
2023-11-28 10:29:24.831 [job-0] WARN  OriginalConfPretreatmentUtil - 您的配置文件中的列配置存在一定的風險. 因為您未配置讀取資料庫表的列，當您的表欄位個數、類型有變動時，可能影響任務正確性甚至會運行出錯。請檢查您的配置並作出修改.
Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.
2023-11-28 10:29:25.113 [job-0] INFO  OriginalConfPretreatmentUtil - table:[t2] all columns:[id,createtime].
2023-11-28 10:29:25.113 [job-0] WARN  OriginalConfPretreatmentUtil - 您的配置文件中的列配置信息存在風險. 因為您配置的寫入資料庫表的列為*，當您的表欄位個數、類型有變動時，可能影響任務正確性甚至會運行出錯。請檢查您的配置並作出修改.
2023-11-28 10:29:25.115 [job-0] INFO  OriginalConfPretreatmentUtil - Write data [
insert INTO %s (id,createtime) VALUES(?,?)
], which jdbcUrl like:[jdbc:mysql://10..17.139.86:16310/testdb?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]
2023-11-28 10:29:25.116 [job-0] INFO  JobContainer - jobContainer starts to do prepare ...
2023-11-28 10:29:25.117 [job-0] INFO  JobContainer - DataX Reader.Job [sqlserverreader] do prepare work .
2023-11-28 10:29:25.117 [job-0] INFO  JobContainer - DataX Writer.Job [mysqlwriter] do prepare work .
2023-11-28 10:29:25.118 [job-0] INFO  JobContainer - jobContainer starts to do split ...
2023-11-28 10:29:25.118 [job-0] INFO  JobContainer - Job set Channel-Number to 5 channels.
2023-11-28 10:29:25.123 [job-0] INFO  JobContainer - DataX Reader.Job [sqlserverreader] splits to [1] tasks.
2023-11-28 10:29:25.124 [job-0] INFO  JobContainer - DataX Writer.Job [mysqlwriter] splits to [1] tasks.
2023-11-28 10:29:25.146 [job-0] INFO  JobContainer - jobContainer starts to do schedule ...
2023-11-28 10:29:25.150 [job-0] INFO  JobContainer - Scheduler starts [1] taskGroups.
2023-11-28 10:29:25.153 [job-0] INFO  JobContainer - Running by standalone Mode.
2023-11-28 10:29:25.159 [taskGroup-0] INFO  TaskGroupContainer - taskGroupId=[0] start [1] channels for [1] tasks.
2023-11-28 10:29:25.165 [taskGroup-0] INFO  Channel - Channel set byte_speed_limit to -1, No bps activated.
2023-11-28 10:29:25.165 [taskGroup-0] INFO  Channel - Channel set record_speed_limit to -1, No tps activated.
2023-11-28 10:29:25.176 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[0] attemptCount[1] is started
2023-11-28 10:29:25.183 [0-0-0-reader] INFO  CommonRdbmsReader$Task - Begin to read record by Sql: [select * from t2 where (createtime > '2023-11-28 03:17:46.330' and createtime < '2023-11-28 03:31:46.330')
] jdbcUrl:[jdbc:sqlserver://127.0.0.1:1433;DatabaseName=testdb].
2023-11-28 10:29:25.344 [0-0-0-reader] INFO  CommonRdbmsReader$Task - Finished read record by Sql: [select * from t2 where (createtime > '2023-11-28 03:17:46.330' and createtime < '2023-11-28 03:31:46.330')
] jdbcUrl:[jdbc:sqlserver://127.0.0.1:1433;DatabaseName=testdb].
2023-11-28 10:29:25.606 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[0] is successed, used[431]ms
2023-11-28 10:29:25.607 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] completed it's tasks.
2023-11-28 10:29:35.173 [job-0] INFO  StandAloneJobContainerCommunicator - Total 4 records, 37 bytes | Speed 3B/s, 0 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.000s |  All Task WaitReaderTime 0.000s | Percentage 100.00%
2023-11-28 10:29:35.173 [job-0] INFO  AbstractScheduler - Scheduler accomplished all tasks.
2023-11-28 10:29:35.174 [job-0] INFO  JobContainer - DataX Writer.Job [mysqlwriter] do post work.
2023-11-28 10:29:35.175 [job-0] INFO  JobContainer - DataX Reader.Job [sqlserverreader] do post work.
2023-11-28 10:29:35.175 [job-0] INFO  JobContainer - DataX jobId [0] completed successfully.
2023-11-28 10:29:35.177 [job-0] INFO  HookInvoker - No hook invoked, because base dir not exists or is a file: /soft/datax/hook
2023-11-28 10:29:35.179 [job-0] INFO  JobContainer - 

 [total cpu info] => 
averageCpu                     | maxDeltaCpu                    | minDeltaCpu                    
-1.00%                         | -1.00%                         | -1.00%

 [total gc info] => 
 NAME                 | totalGCCount       | maxDeltaGCCount    | minDeltaGCCount    | totalGCTime        | maxDeltaGCTime     | minDeltaGCTime     
 PS MarkSweep         | 1                  | 1                  | 1                  | 0.052s             | 0.052s             | 0.052s             
 PS Scavenge          | 1                  | 1                  | 1                  | 0.024s             | 0.024s             | 0.024s             
2023-11-28 10:29:35.180 [job-0] INFO  JobContainer - PerfTrace not enable!
2023-11-28 10:29:35.181 [job-0] INFO  StandAloneJobContainerCommunicator - Total 4 records, 37 bytes | Speed 3B/s, 0 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.000s |  All Task WaitReaderTime 0.000s | Percentage 100.00%

2023-11-28 10:29:35.183 [job-0] INFO  JobContainer - 
任務啟動時刻                    : 2023-11-28 10:29:24
任務結束時刻                    : 2023-11-28 10:29:35
任務總計耗時                    :                 10s
任務平均流量                    :                3B/s
記錄寫入速度                    :              0rec/s
讀出記錄總數                    :                   4
讀寫失敗總數                    :                   0

4.8 到目標端（GreatSQL）驗證增量數據

greatsql> select * from t2;

+----+---------------------+
| id | createtime          |
+----+---------------------+
|  1 | 2023-11-28 02:18:21 |
|  2 | 2023-11-28 02:18:27 |
|  3 | 2023-11-28 02:18:32 |
|  4 | 2023-11-28 02:18:38 |
|  5 | 2023-11-28 02:18:41 |
|  6 | 2023-11-28 02:18:46 |
|  7 | 2023-11-28 03:18:46 |
|  8 | 2023-11-28 03:20:46 |
|  9 | 2023-11-28 03:25:46 |
| 10 | 2023-11-28 03:30:46 |
+----+---------------------+
10 rows in set (0.00 sec)

增量遷移總結：通過增加過濾條件達到增量遷移的目的。主要是通過過濾條件過濾掉了全量遷移的數據，進而變相的完成了增量遷移。

Enjoy GreatSQL

您的分享是我們最大的動力!

-Advertisement-
Play Games
更多相關文章
  • 痞子衡嵌入式：在i.MXRT1xxx系列上用NAND型啟動設備時可用兩級設計縮短啟動時間
    大家好，我是痞子衡，是正經搞技術的痞子。今天痞子衡給大家分享的是在i.MXRT1xxx系列上用NAND型啟動設備時可用兩級設計縮短啟動時間。 去年痞子衡寫過一篇騷操作文章 《藉助i.MXRT10xx系列INIT_VTOR功能可以縮短程式熱重啟時間》，這對於 NAND 型啟動設備上程式熱重啟時間的縮短 ...
  • shell腳本正則表達式
    聲明： 以下內容為個人筆記，內容不完全正確，請謹慎參考。 正則表達式 含義： 正則表達式使用的每個單獨字元串來描述、匹配一些列符合語法規則的字元串。在很多文本編輯器里，正則表達式通常被用來檢索、替換那些符合某個模式的文本。在Linux中，grep，sed,awk等文本處理工具都支持正則表達式進行模式 ...
  • 【ESP32 IDF】用RMT控制 WS2812 彩色燈帶
    在上一篇中，老周用 .NET Nano Framework 給大伙伴們演示了 WS2812 燈帶的控制，包括用 SPI 和 紅外RMT 的方式。利用 RMT 是一個很機靈的方案，不過，可能很多大伙伴對 ESP32 的 RMT 不是很熟悉。除了樂鑫自己的文檔，沒幾個人寫過相關的水文，可見這裡頭空白的水 ...
  • shell腳本while迴圈、read讀取控制台輸入與函數
    聲明： 以下為個人筆記，內容不完全正確，請謹慎參考。 while迴圈 while迴圈結構 while [條件判斷] do 程式 done 腳本示例: 點擊查看代碼 #!/bin/bash while [ $a -le $1] do sum=$[ $sum + $a ] done echo $sum ...
  • 結構體學習
    /******************************************************************************************************** * * 提高可移植性 Copyright (c) 2023-2024 cececlmx@ ...
  • MIT6.S081 - Lecture3: OS Organization and System Calls
    為什麼要使用操作系統 使用操作系統的主要原因是為了實現 CPU 多進程分時復用以及記憶體隔離 如果沒有操作系統，應用程式會直接與硬體進行交互，這時應用程式會直接使用 CPU，比如假設只有一個 CPU 核，一個應用程式在這個 CPU 核上運行，但是同時其他程式也需要運行，因為沒有操作系統來幫助切換，就需 ...
  • shell系統函數和流程式控制制
    系統函數： 1、簡單示例： 點擊查看代碼 #!/bin/bash filename="$1"_log_$(datename +%S) echo $filename basename: 基本語法： basename [string/pathname] [suffix] (功能描述：basename命令 ...
  • EasyMR6.2 全面解讀：四大功能深度優化，解鎖全新大數據處理和計算體驗
    在剛剛過去的2024春季發佈會上，袋鼠雲帶來了數棧產品V6.2版本的全新發佈。其中，EasyMR 作為數棧V6.2中的一項關鍵能力，代表了袋鼠雲對大數據生態的深入理解和持續創新。 EasyMR（後文統稱EMR）是袋鼠雲基於 Hadoop、Hive、Spark、Flink、HBase 等開源組件，構建 ...
一周排行
    -Advertisement-
    Play Games
  • WPF中輕鬆生成動態圖表：實例詳解(MVVM模式)
    概述：本文代碼示例演示瞭如何在WPF中使用LiveCharts庫創建動態條形圖。通過創建數據模型、ViewModel和在XAML中使用`CartesianChart`控制項，你可以輕鬆實現圖表的數據綁定和動態更新。我將通過清晰的步驟指南包括詳細的中文註釋，幫助你快速理解並應用這一功能。 先上效果： 在 ...
  • .NET 中使用 GaussDB C# ORM
    openGauss(GaussDB ) openGauss是一款全面友好開放，攜手伙伴共同打造的企業級開源關係型資料庫。openGauss採用木蘭寬鬆許可證v2發行，提供面向多核架構的極致性能、全鏈路的業務、數據安全、基於AI的調優和高效運維的能力。openGauss深度融合華為在資料庫領域多年的研 ...
  • .NET中使用 openGauss C# ORM
    openGauss(GaussDB ) openGauss是一款全面友好開放，攜手伙伴共同打造的企業級開源關係型資料庫。openGauss採用木蘭寬鬆許可證v2發行，提供面向多核架構的極致性能、全鏈路的業務、數據安全、基於AI的調優和高效運維的能力。openGauss深度融合華為在資料庫領域多年的研 ...
  • 輕鬆使用Aspire rabbitmq framework
    概述：本示例演示了在WPF應用程式中實現多語言支持的詳細步驟。通過資源字典和數據綁定，以及使用語言管理器類，應用程式能夠在運行時動態切換語言。這種方法使得多語言支持更加靈活，便於維護，同時提供清晰的代碼結構。 在WPF中實現多語言的一種常見方法是使用資源字典和數據綁定。以下是一個詳細的步驟和示例源代 ...
  • .net事件
    描述（做一個簡單的記錄）： 事件（event）的本質是一個委托；（聲明一個事件： public event TestDelegate eventTest;） 委托（delegate）可以理解為一個符合某種簽名的方法類型；比如：TestDelegate委托的返回數據類型為string，參數為 int和 ...
  • .NET CORE 完美支持AOT 的 ORM SqlSugar 教程
    1、AOT適合場景 Aot適合工具類型的項目使用，優點禁止反編 ，第一次啟動快，業務型項目或者反射多的項目不適合用AOT AOT更新記錄: 實實在在經過實踐的AOT ORM 5.1.4.117 +支持AOT 5.1.4.123 +支持CodeFirst和非同步方法 5.1.4.129-preview1 ...
  • 【UWP】修改清單脫離沙盒運行
    總說周知，UWP 是運行在沙盒裡面的，所有許可權都有嚴格限制，和沙盒外交互也需要特殊的通道，所以從根本杜絕了 UWP 毒瘤的存在。但是實際上 UWP 只是一個應用模型，本身是沒有什麼許可權管理的，許可權管理全靠 App Container 沙盒控制，如果我們脫離了這個沙盒，UWP 就會放飛自我了。那麼有沒... ...
  • 《Effective C++》第三版-4. 設計與聲明（Design and Declarations）
    目錄條款17：讓介面容易被正確使用，不易被誤用（Make interfaces easy to use correctly and hard to use incorrectly）限制類型和值規定能做和不能做的事提供行為一致的介面條款19：設計class猶如設計type（Treat class de ...
  • 從零開始：Django項目的創建與配置指南
    title: 從零開始：Django項目的創建與配置指南 date: 2024/5/2 18:29:33 updated: 2024/5/2 18:29:33 categories: 後端開發 tags: Django WebDev Python ORM Security Deployment Op ...
  • mORMot 1.18 第18章 使用REST/JSON的客戶端/伺服器
    1、BOM對象 BOM:Broswer object model,即瀏覽器提供我們開發者在javascript用於操作瀏覽器的對象。 1.1、window對象 視窗方法 // BOM Browser object model 瀏覽器對象模型 // js中最大的一個對象.整個瀏覽器視窗出現的所有東西都 ...
所有分類