PostgreSQL部分主要字典信息

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:PostgreSQL部分主要字典信息

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

一、pg_class:

该系统表记录了数据表、索引(仍然需要参阅pg_index)、序列、视图、复合类型和一些特殊关系类型的元数据。注意:不是所有字段对所有对象类型都有意义。

名字 类型 引用 描述
relname name 数据类型名字。
relnamespace oid pg_namespace.oid 包含这个对象的名字空间(模式)的OI。
reltype oid pg_type.oid 对应这个表的行类型的数据类型。
relowner oid pg_authid.oid 对象的所有者。
relam oid pg_am.oid 对于索引对象,表示该索引的类型(B-tree,hash)。
relfilenode oid 对象存储在磁盘上的文件名,如果没有则为0。
reltablespace oid pg_tablespace.oid 对象所在的表空间。如果为零,则表示使用该数据库的缺省表空间。(如果对象在磁盘上没有文件,这个字段就没有什么意义)
relpages int4 该数据表或索引所占用的磁盘页面数量,查询规划器会借助该值选择最优路径。
reltuples float4 表中行的数量,该值只是被规划器使用的一个估计值。
reltoastrelid oid pg_class.oid 与此表关联的TOAST表的OID,如果没有为0。TOAST表在一个从属表里”离线”存储大字段。
reltoastidxid oid pg_class.oid 如果是TOAST表,该字段为它索引的OID,如果不是TOAST表则为0。
relhasindex bool 如果这是一个数据表而且至少有(或者最近有过)一个索引,则为真。它是由CREATE INDEX设置的,但DROP INDEX不会立即将它清除。如果VACUUM发现一个表没有索引,那么它清理 relhasindex。
relisshared bool 如果该表在整个集群中由所有数据库共享,则为真。
relkind char r = 普通表,i = 索引,S = 序列,v = 视图, c = 复合类型,s = 特殊,t = TOAST表
relnatts int2 数据表中用户字段的数量(除了系统字段以外,如oid)。在pg_attribute里肯定有相同数目的数据行。见pg_attribute.attnum.
relchecks int2 表中检查约束的数量,参阅pg_constraint表。
reltriggers int2 表中触发器的数量;参阅pg_trigger表。
relhasoids bool 如果我们为对象中的每行都生成一个OID,则为真。
relhaspkey bool 如果该表存在主键,则为真。
relhasrules bool 如表有规则就为真;参阅pg_rewrite表。
relhassubclass bool 如果该表有子表,则为真。
relacl aclitem[] 访问权限。

二、pg_attribute:

该系统表存储所有表(包括系统表,如pg_class)的字段信息。数据库中的每个表的每个字段在pg_attribute表中都有一行记录。

名字 类型 引用 描述
attrelid oid pg_class.oid 此字段所属的表。
attname name 字段名。
atttypid oid pg_type.oid 字段的数据类型。
attstattarget int4 attstattarget控制ANALYZE为这个字段设置的统计细节的级别。零值表示不收集统计信息,负数表示使用系统缺省的统计对象。正数值的确切信息是和数据类型相关的。
attlen int2 该字段所属类型的长度。(pg_type.typlen的拷贝)
attnum int2 字段的编号,普通字段是从1开始计数的。系统字段,如oid,是任意的负数。
attndims int4 如果该字段是数组,该值表示数组的维数,否则是0。
attcacheoff int4 在磁盘上总是-1,但是如果装载入内存中的行描述器中, 它可能会被更新为缓冲在行中字段的偏移量。
atttypmod int4 表示数据表在创建时提供的类型相关的数据(比如,varchar字段的最大长度)。其值对那些不需要atttypmod的类型而言通常为-1。
attbyval bool pg_type.typbyval字段值的拷贝。
attstorage char pg_type.typstorage字段值的拷贝。
attalign char pg_type.typalign字段值的拷贝。
attnotnull bool 如果该字段带有非空约束,则为真,否则为假。
atthasdef bool 该字段是否存在缺省值,此时它对应pg_attrdef表里实际定义此值的记录。
attisdropped bool 该字段是否已经被删除。如果被删除,该字段在物理上仍然存在表中,但会被分析器忽略,因此不能再通过SQL访问。
attislocal bool 该字段是否局部定义在对象中的。
attinhcount int4 该字段所拥有的直接祖先的个数。如果一个字段的祖先个数非零,那么它就不能被删除或重命名。

三、pg_attrdef:

该系统表主要存储字段默认值,字段中的主要信息存放在pg_attribute系统表中。注意:只有明确声明了缺省值的字段在该表中才会有记录。

名字 类型 引用 描述
adrelid oid pg_class.oid 这个字段所属的表
adnum int2 pg_attribute.attnum 字段编号,其规则等同于pg_attribute.attnum
adbin text 字段缺省值的内部表现形式。
adsrc text 缺省值的人可读的表现形式。

四、pg_constraint:

该系统表存储PostgreSQL中表对象的检查约束、主键、唯一约束和外键约束。

名字 类型 引用 描述
conname name 约束名字(不一定是唯一的)。
connamespace oid pg_namespace.oid 包含这个约束的名字空间(模式)的OID。
contype char c = 检查约束, f = 外键约束, p = 主键约束, u = 唯一约束
condeferrable bool 该约束是否可以推迟。
condeferred bool 缺省时这个约束是否是推迟的?
conrelid oid pg_class.oid 该约束所在的表,如果不是表约束则为0。
contypid oid pg_type.oid 该约束所在的域,如果不是域约束则为0。
confrelid oid pg_class.oid 如果为外键,则指向参照的表,否则为0。
confupdtype char 外键更新动作代码。
confdeltype char 外键删除动作代码。
confmatchtype char 外键匹配类型。
conkey int2[] pg_attribute.attnum 如果是表约束,则是约束控制的字段列表。
confkey int2[] pg_attribute.attnum 如果是外键,则是参照字段的列表。
conbin text 如果是检查约束,则表示表达式的内部形式。
consrc text 如果是检查约束,则是表达式的人可读的形式。

五、pg_tablespace:

该系统表存储表空间的信息。注意:表可以放在特定的表空间里,以帮助管理磁盘布局和解决IO瓶颈。

名字 类型 引用 描述
spcname name 表空间名称。
spcowner oid pg_authid.oid 表空间的所有者,通常是创建它的角色。
spclocation text 表空间的位置(目录路径)。
spcacl aclitem[] 访问权限。

六、pg_namespace:

该系统表存储名字空间(模式)。

名字 类型 引用 描述
nspname name 名字空间(模式)的名称。
nspowner oid pg_authid.oid 名字空间(模式)的所有者
nspacl aclitem[] 访问权限。

七、pg_database:

该系统表存储数据库的信息。和大多数系统表不同的是,在一个集群里该表是所有数据库共享的,即每个集群只有一份pg_database拷贝,而不是每个数据库一份。

名字 类型 引用 描述
datname name 数据库名称。
datdba oid pg_authid.oid 数据库所有者,通常为创建该数据库的角色。
encoding int4 数据库的字符编码方式。
datistemplate bool 如果为真,此数据库可以用于CREATE DATABASE TEMPLATE子句,把新数据库创建为此数据库的克隆。
datallowconn bool 如果为假,则没有人可以联接到这个数据库。
datlastsysoid oid 数据库里最后一个系统OID,此值对pg_dump特别有用。
datvacuumxid xid
datfrozenxid xid
dattablespace text pg_tablespace.oid 该数据库的缺省表空间。在这个数据库里,所有pg_class.reltablespace为零的表都将保存在这个表空间里,特别要指出的是,所有非共享的系统表也都存放在这里。
datconfig text[] 运行时配置变量的会话缺省值。
datacl aclitem[] 访问权限。

八、pg_index:

该系统表存储关于索引的一部分信息。其它的信息大多数存储在pg_class。

名字 类型 引用 描述
indexrelid oid pg_class.oid 该索引在pg_class里的记录的OID。
indrelid oid pg_class.oid 索引所在表在pg_class里的记录的OID。
indnatts int2 索引中的字段数量(拷贝的pg_class.relnatts)。
indisunique bool 如果为真,该索引是唯一索引。
indisprimary bool 如果为真,该索引为该表的主键。
indisclustered bool 如果为真,那么该表在这个索引上建了簇。
indkey int2vector pg_attribute.attnum 该数组的元素数量为indnatts,数组元素值表示建立这个索引时所依赖的字段编号,如1 3,表示第一个字段和第三个字段构成这个索引的键值。如果为0,则表示是表达式索引,而不是基于简单字段的索引。
indclass oidvector pg_opclass.oid 对于构成索引键值的每个字段,这个字段都包含一个指向所使用的操作符表的OID。
indexprs text 表达式树用于那些非简单字段引用的索引属性。它是一个列表,在indkey里面的每个零条目一个元素。如果所有索引属性都是简单的引用,则为空。
indpred text 部分索引断言的表达式树。如果不是部分索引, 则是空字串。

PostgreSQL 16 源码安装

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:PostgreSQL 16 源码安装

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

空闲中尝试在linux 8的版本上,源码安装PostgreSQL 16,参考文章:PostgreSQL16.2源码部署教程

[root@xifenfei tmp]# cat /etc/redhat-release 
Red Hat Enterprise Linux release 8.7 (Ootpa)
[root@xifenfei tmp]#  uname -osr
Linux 5.15.0-3.60.5.1.el8uek.x86_64 GNU/Linux

下载pg源码程序:PostgreSQL 16源码下载
linux配置

--关闭selinux
sed -i "s/SELINUX=enforcing/SELINUX=disabled/" /etc/selinux/config
setenforce 0

--安装相应的包
yum -y install readline readline-devel zlib zlib-devel gettext \
 gettext-devel openssl openssl-devel pam pam-devel libxml2 \
libxml2-devel libxslt libxslt-devel perl perl-devel tcl-devel \
libuuid-devel gcc gcc-c++ make flex bison perl-ExtUtils*  libicu  libicu-devel

--修改内核参数
cat>>/etc/sysctl.conf<<EOF
fs.file-max = 76724200
kernel.sem = 10000 10240000 10000 1024
kernel.shmmni = 4096
kernel.shmall = 253702
kernel.shmmax = 1039163392
net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default = 262144
net.core.wmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_max = 1048576
fs.aio-max-nr = 40960000
vm.dirty_ratio=20
vm.dirty_background_ratio=3
vm.dirty_writeback_centisecs=100
vm.dirty_expire_centisecs=500
vm.swappiness=10
vm.min_free_kbytes=524288
vm.swappiness=0
vm.overcommit_memory=2
vm.overcommit_ratio=75
net.ipv4.ip_local_port_range = 10000 65535
EOF
sysctl -p

--创建组和用户
groupadd postgres -g 3000
useradd postgres -g 3000 -u 3000

--用户 limits 配置
cat>>/etc/security/limits.conf<<EOF
postgres soft nofile 1048576
postgres hard nofile 1048576
postgres soft nproc 131072
postgres hard nproc 131072
postgres soft stack 10240
postgres hard stack 32768
postgres soft core 6291456
postgres hard core 6291456
EOF

--创建相关目录和权限
mkdir -p /pg/database/server
mkdir -p /pg/database/data
mkdir -p /pg/database/wal
mkdir -p /pg/database/archive
chown -R postgres:postgres /pg -R
chmod 0775 /pg
chmod 0700 /pg/database/data

--设置环境变量
vi /home/postgres/.bash_profile
export PGPORT=5432
export PGUSER=postgres
export PGHOME=/pg/database/server
export PGDATA=/pg/database/data
export PATH=$PGHOME/bin:$PATH

--编译pg软件
su - postgres
tar xzvf postgresql-16.2.tar.gz
cd /tmp/postgresql-16.2
./configure --prefix=/pg/database/server --with-pgport=5432
make
make install

--确认软件安装成功(检查编译日志和测试如下命令)
[postgres@xifenfei ~]$ postgres --version
postgres (PostgreSQL) 16.2

创建数据库

[postgres@xifenfei ~]$  /pg/database/server/bin/initdb -D/pg/database/data\
 -X/pg/database/wal -EUTF8 -Upostgres -W
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.UTF-8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

Enter new superuser password: 
Enter it again: 

fixing permissions on existing directory /pg/database/data ... ok
fixing permissions on existing directory /pg/database/wal ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... America/New_York
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

initdb: warning: enabling "trust" authentication for local connections
initdb: hint: You can change this by editing pg_hba.conf or using the option -A,
 or --auth-local and --auth-host, the next time you run initdb.

Success. You can now start the database server using:

    /pg/database/server/bin/pg_ctl -D /pg/database/data -l logfile start

启动数据库

[postgres@xifenfei ~]$ /pg/database/server/bin/pg_ctl -D /pg/database/data -l logfile start
waiting for server to start.... done
server started
[postgres@xifenfei ~]$ psql
psql (16.2)
Type "help" for help.

postgres=# \d pg_class
                     Table "pg_catalog.pg_class"
       Column        |     Type     | Collation | Nullable | Default 
---------------------+--------------+-----------+----------+---------
 oid                 | oid          |           | not null | 
 relname             | name         |           | not null | 
 relnamespace        | oid          |           | not null | 
 reltype             | oid          |           | not null | 
 reloftype           | oid          |           | not null | 
 relowner            | oid          |           | not null | 
 relam               | oid          |           | not null | 
 relfilenode         | oid          |           | not null | 
 reltablespace       | oid          |           | not null | 
 relpages            | integer      |           | not null | 
 reltuples           | real         |           | not null | 
 relallvisible       | integer      |           | not null | 
 reltoastrelid       | oid          |           | not null | 
 relhasindex         | boolean      |           | not null | 
 relisshared         | boolean      |           | not null | 
 relpersistence      | "char"       |           | not null | 
 relkind             | "char"       |           | not null | 
 relnatts            | smallint     |           | not null | 
 relchecks           | smallint     |           | not null | 
 relhasrules         | boolean      |           | not null | 
 relhastriggers      | boolean      |           | not null | 
 relhassubclass      | boolean      |           | not null | 
 relrowsecurity      | boolean      |           | not null | 
 relforcerowsecurity | boolean      |           | not null | 
 relispopulated      | boolean      |           | not null | 
 relreplident        | "char"       |           | not null | 
 relispartition      | boolean      |           | not null | 
 relrewrite          | oid          |           | not null | 
 relfrozenxid        | xid          |           | not null | 
 relminmxid          | xid          |           | not null | 
 relacl              | aclitem[]    |           |          | 
 reloptions          | text[]       | C         |          | 
 relpartbound        | pg_node_tree | C         |          | 
Indexes:
    "pg_class_oid_index" PRIMARY KEY, btree (oid)
    "pg_class_relname_nsp_index" UNIQUE CONSTRAINT, btree (relname, relnamespace)
    "pg_class_tblspc_relfilenode_index" btree (reltablespace, relfilenode)

配置开机自动启动

[root@xifenfei tmp]# cp /tmp/postgresql-16.2/contrib/start-scripts/linux /etc/init.d/PostgreSQL
[root@xifenfei tmp]# vi /etc/init.d/PostgreSQL 
--修改如下值
prefix=/pg/database/server
PGDATA="/pg/database/data"
[root@xifenfei tmp]# chkconfig --add PostgreSQL
[root@xifenfei tmp]# chkconfig --list

Note: This output shows SysV services only and does not include native
      systemd services. SysV configuration data might be overridden by native
      systemd configuration.

      If you want to list systemd services use 'systemctl list-unit-files'.
      To see services enabled on particular target use
      'systemctl list-dependencies [target]'.

PostgreSQL      0:off   1:off   2:on    3:on    4:on    5:on    6:off

PostgreSQL恢复系列:wal日志丢失恢复

联系:手机/微信(+86 17813235971) QQ(107644445)QQ咨询惜分飞

标题:PostgreSQL恢复系列:wal日志丢失恢复

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

WAL是Write Ahead Log的简写,和oracle的redo日志类似,存放在$PGDATA/pg_xlog中,10版本以后在$PGDATA/pg_wal目录.在oracle数据库中,如果redo丢失,分为active/current和inactive的redo,分别有不同的处理方式,对于oracle需要实例恢复的redo丢失,需要屏蔽数据库一致性,强制打开数据库,对于PG数据库这部分日志丢失该如何恢复,主要是通过pg_resetwal/pg_resetxlog(10以前版本)命令来实现,这里通过一个测试来验证
创建测试表并强制kill数据库

-bash-4.2$ psql
psql (14.3)
Type "help" for help.

postgres=# create table t_xifenfei as select * from pg_database;
SELECT 4
postgres=# select count(1) from t_xifenfei;
 count 
-------
     4
(1 row)

postgres=# \q
-bash-4.2$ ps -ef|grep post
root       1819      1  0 May28 ?        00:00:00 /usr/libexec/postfix/master -w
postfix    1838   1819  0 May28 ?        00:00:00 qmgr -l -t unix -u
postgres  11102      1  0 05:49 ?        00:00:00 /usr/pgsql-14/bin/postgres -D /var/lib/pgsql/14/data
postgres  11103  11102  0 05:49 ?        00:00:00 postgres: logger 
postgres  11105  11102  0 05:49 ?        00:00:00 postgres: checkpointer 
postgres  11106  11102  0 05:49 ?        00:00:00 postgres: background writer 
postgres  11107  11102  0 05:49 ?        00:00:00 postgres: walwriter 
postgres  11108  11102  0 05:49 ?        00:00:00 postgres: autovacuum launcher 
postgres  11109  11102  0 05:49 ?        00:00:01 postgres: stats collector 
postgres  11110  11102  0 05:49 ?        00:00:00 postgres: logical replication launcher 
root      22743  22300  0 18:26 pts/3    00:00:00 su - postgres
postgres  22744  22743  0 18:26 pts/3    00:00:00 -bash
postgres  22937  22744  0 18:28 pts/3    00:00:00 psql
postgres  22938  11102  0 18:28 ?        00:00:00 postgres: postgres postgres [local] idle
postfix   32623   1819  0 21:10 ?        00:00:00 pickup -l -t unix -u
root      33032  32912  0 21:15 pts/2    00:00:00 su - postgres
postgres  33033  33032  0 21:15 pts/2    00:00:00 -bash
postgres  35210  33033  0 21:51 pts/2    00:00:00 ps -ef
postgres  35211  33033  0 21:51 pts/2    00:00:00 grep --color=auto post
-bash-4.2$ kill -9 11102

删除wal日志

-bash-4.2$ pwd
/var/lib/pgsql/14/data/pg_wal
-bash-4.2$ ls -ltr
total 311296
drwx------. 2 postgres postgres        6 May 24 02:20 archive_status
-rw-------. 1 postgres postgres 16777216 May 28 21:29 000000010000000000000014
-rw-------. 1 postgres postgres 16777216 May 28 21:29 000000010000000000000015
-rw-------. 1 postgres postgres 16777216 May 28 21:29 000000010000000000000016
-rw-------. 1 postgres postgres 16777216 May 28 21:29 000000010000000000000017
-rw-------. 1 postgres postgres 16777216 May 28 21:29 000000010000000000000018
-rw-------. 1 postgres postgres 16777216 May 28 21:29 000000010000000000000019
-rw-------. 1 postgres postgres 16777216 May 28 21:29 00000001000000000000001A
-rw-------. 1 postgres postgres 16777216 May 28 21:29 00000001000000000000001B
-rw-------. 1 postgres postgres 16777216 May 28 21:29 00000001000000000000001C
-rw-------. 1 postgres postgres 16777216 May 28 21:29 00000001000000000000001D
-rw-------. 1 postgres postgres 16777216 May 28 21:29 00000001000000000000001E
-rw-------. 1 postgres postgres 16777216 May 28 21:29 00000001000000000000001F
-rw-------. 1 postgres postgres 16777216 May 28 21:29 000000010000000000000020
-rw-------. 1 postgres postgres 16777216 May 28 21:29 000000010000000000000021
-rw-------. 1 postgres postgres 16777216 May 28 21:29 000000010000000000000022
-rw-------. 1 postgres postgres 16777216 May 28 21:30 000000010000000000000023
-rw-------. 1 postgres postgres 16777216 May 28 21:30 000000010000000000000024
-rw-------. 1 postgres postgres 16777216 May 28 21:30 000000010000000000000025
-rw-------. 1 postgres postgres 16777216 May 29 21:51 000000010000000000000013
-bash-4.2$ rm -rf 0000000100000000000000*
-bash-4.2$ ls
archive_status

查询当时数据库需要的最小wal记录

-bash-4.2$ pg_controldata 
pg_control version number:            1300
Catalog version number:               202107181
Database system identifier:           7100998319216817119
Database cluster state:               in production
pg_control last modified:             Sat 28 May 2022 09:36:11 PM CST
Latest checkpoint location:           0/13692F80
Latest checkpoint's REDO location:    0/13692F48
Latest checkpoint's REDO WAL file:    000000010000000000000013   <===需要的记录
Latest checkpoint's TimeLineID:       1
Latest checkpoint's PrevTimeLineID:   1
Latest checkpoint's full_page_writes: on
Latest checkpoint's NextXID:          0:17824
Latest checkpoint's NextOID:          32769
Latest checkpoint's NextMultiXactId:  1
Latest checkpoint's NextMultiOffset:  0
Latest checkpoint's oldestXID:        727
Latest checkpoint's oldestXID's DB:   1
Latest checkpoint's oldestActiveXID:  17824
Latest checkpoint's oldestMultiXid:   1
Latest checkpoint's oldestMulti's DB: 1
Latest checkpoint's oldestCommitTsXid:0
Latest checkpoint's newestCommitTsXid:0
Time of latest checkpoint:            Sat 28 May 2022 09:31:41 PM CST

尝试启动PG

-bash-4.2$ pg_ctl start
pg_ctl: another server might be running; trying to start server anyway
waiting for server to start....2022-05-29 21:52:22.926 CST [35270] LOG:  
redirecting log output to logging collector process
2022-05-29 21:52:22.926 CST [35270] HINT:  Future log output will appear in directory "log".
. stopped waiting
pg_ctl: could not start server
Examine the log output.

启动pg失败,查看日志记录

2022-05-29 21:52:22.926 CST [35270] LOG:  starting PostgreSQL 14.3 on x86_64-pc-linux-gnu, 
            compiled by gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-44), 64-bit
2022-05-29 21:52:22.927 CST [35270] LOG:  listening on IPv6 address "::1", port 5432
2022-05-29 21:52:22.927 CST [35270] LOG:  listening on IPv4 address "127.0.0.1", port 5432
2022-05-29 21:52:22.929 CST [35270] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2022-05-29 21:52:22.931 CST [35270] LOG:  listening on Unix socket "/tmp/.s.PGSQL.5432"
2022-05-29 21:52:22.936 CST [35272] LOG:  database system was interrupted; last known up at 2022-05-28 21:36:11 CST
2022-05-29 21:52:23.049 CST [35272] LOG:  invalid primary checkpoint record
2022-05-29 21:52:23.049 CST [35272] PANIC:  could not locate a valid checkpoint record
2022-05-29 21:52:24.211 CST [35270] LOG:  startup process (PID 35272) was terminated by signal 6: Aborted
2022-05-29 21:52:24.211 CST [35270] LOG:  aborting startup due to startup process failure
2022-05-29 21:52:24.218 CST [35270] LOG:  database system is shut down

错误比较明显,无法定位到有效的checkpoint记录,在oracle里面的意思可以理解为无法进行实例恢复,pg启动失败

重设wal
由于数据库为不一致状态,需要使用-f进行强制重设

-bash-4.2$ pg_resetwal $PGDATA
The database server was not shut down cleanly.
Resetting the write-ahead log might cause data to be lost.
If you want to proceed anyway, use -f to force reset.
-bash-4.2$ pg_resetwal -f $PGDATA
Write-ahead log reset

启动PG成功

-bash-4.2$ pg_ctl start -D $PGDATA
waiting for server to start....2022-05-29 22:01:02.647 CST [37178] LOG:  
redirecting log output to logging collector process
2022-05-29 22:01:02.647 CST [37178] HINT:  Future log output will appear in directory "log".
 done
server started

日志记录

2022-05-29 22:01:02.647 CST [37178] LOG:  starting PostgreSQL 14.3 on x86_64-pc-linux-gnu, 
                    compiled by gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-44), 64-bit
2022-05-29 22:01:02.648 CST [37178] LOG:  listening on IPv6 address "::1", port 5432
2022-05-29 22:01:02.648 CST [37178] LOG:  listening on IPv4 address "127.0.0.1", port 5432
2022-05-29 22:01:02.649 CST [37178] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2022-05-29 22:01:02.651 CST [37178] LOG:  listening on Unix socket "/tmp/.s.PGSQL.5432"
2022-05-29 22:01:02.653 CST [37180] LOG:  database system was shut down at 2022-05-29 22:00:47 CST
2022-05-29 22:01:02.661 CST [37178] LOG:  database system is ready to accept connections

查看wal日志,产生新记录

-bash-4.2$ pwd
/var/lib/pgsql/14/data/pg_wal
-bash-4.2$ ls -ltr
total 16384
drwx------. 2 postgres postgres        6 May 24 02:20 archive_status
-rw-------. 1 postgres postgres 16777216 May 29 22:01 000000010000000000000014

验证刚刚创建测试表

-bash-4.2$ psql
psql (14.3)
Type "help" for help.

postgres=# select count(1) from t_xifenfei;
ERROR:  relation "t_xifenfei" does not exist
LINE 1: select count(1) from t_xifenfei;
                             ^

由于需要进行实例恢复的wal日志丢失导致这表记录也丢失.由此可见这类操作可能导致数据丢失风险,对于生产环境,需要慎重,

sql server数据库比特币加密勒索恢复

联系:手机/微信(+86 17813235971) QQ(107644445)

标题:sql server数据库比特币加密勒索恢复

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

对于GANDCRAB病毒加密的Oracle数据库,我们可以提供较为完美的恢复见(GANDCRAB V5.0.4 比特币加密oracle数据库恢复GANDCRAB升级版Oracle恢复,对于被GANDCRAB加密的SQL Server数据库近期我们对其进行了一些研究,现在也可以比较好的恢复.
gandcrab5.2-sql-server


1


而且如果找黑客解密需要费用为10w美元,客户无法接受该费用,系统中主要的是sql server数据库被加密,客户有几个月之前的备份,但是数据丢失严重,无法承受相关损失,请求我们给予恢复支持.通过我们一系列恢复之后,实现较为完美恢复该数据库
gandcrab5.2-sql-server1


gandcrab5.2-sql-server2


如果您的sql server数据库不幸被比特币加密,可以随时联系我们,提供数据库级别恢复支持
Phone:17813235971    Q Q:107644445QQ咨询惜分飞    E-Mail:dba@xifenfei.com

PostgreSQL常见字符串操作函数

联系:手机/微信(+86 17813235971) QQ(107644445)

标题:PostgreSQL常见字符串操作函数

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

转载:pmars blog中关于pg字符串相关函数

函数:string || string
说明:String concatenation 字符串连接操作
例子:select 'Post' || 'greSQL'; = PostgreSQL
函数:string || non-string or non-string || string
说明:String concatenation with one non-string input 字符串与非字符串类型进行连接操作
例子:select 'Value: ' || 42; = Value: 42
函数:bit_length(string)
说明:Number of bits in string 计算字符串的位数
例子:select bit_length('pmars') = 40
函数:char_length(string) or character_length(string)
说明:Number of characters in string 计算字符串中字符个数
例子:select char_length('pmars'); = 5
函数:lower(string)
说明:Convert string to lower case 转换字符串为小写
例子:select lower('PmArS'); = "pmars"
函数:octet_length(string)
说明:Number of bytes in string 计算字符串的字节数
例子:select octet_length('我是pmars'); = 11  select octet_length('我');  = 3
函数:overlay(string placing string from int [for int])
说明:Replace substring 替换字符串中任意长度的子字串为新字符串
例子:select overlay('I am pmars' placing 'ming' from 6 for 5); = "I am ming"
函数:position(substring in string)
说明:Location of specified substring 子串在一字符串中的位置
例子:select position('ma' in 'pmars'); = 2
函数:substring(string [from int] [for int])
说明:Extract substring 截取任意长度的子字符串
例子:select substring('topmars' from 3 for 3); = "pma"
函数:substring(string from pattern)
说明:Extract substring matching POSIX regular expression.
 See Section 9.7 for more information on pattern matching.
  利用正则表达式对一字符串进行任意长度的字串的截取
例子:select substring('topmars' from 'p.*$'); = "pmars"
函数:substring(string from pattern for escape)
说明:Extract substring matching SQL regular expression. See Section 9.7 for more information on pattern matching.
      利于正则表达式对某类字符进行删除,以得到子字符串
例子:select substring('Thomas' from '%#"o_a#"_' for '#'); = "oma"
函数:trim([leading | trailing | both] [characters] from string)
说明:Remove the longest string containing only the characters
     (a space by default) from the start/end/both ends of the string
     去除尽可能长开始,结束或者两边的某类字符,默认为去除空白字符,当然可以自己指定,可同时指定多个要删除的字符串
例子:select trim(leading 'p' from 'pmars'); = "mars"
函数:upper(string)
说明:Convert string to uppercase 将字符串转换为大写
例子:select upper('pmars'); = "PMARS"
函数:ascii(string)
说明:ASCII code of the first character of the argument. For UTF8 returns
 the Unicode code point of the character. For other multibyte encodings.
 the argument must be a strictly ASCII character. 得到某一个字符的Assii值
例子:select ascii('pmars'); = select ascii('p'); = 112
函数:btrim(string text [, characters text])
说明:Remove the longest string consisting only of characters in characters
     (a space by default) from the start and end of string
     去除字符串两边的所有指定的字符,可同时指定多个字符
例子:select btrim('pmars','prs'); = "ma"
函数:chr(int)
说明:Character with the given code. For UTF8 the argument is treated as a Unicode code point.
 For other multibyte encodings the argument must designate a strictly ASCII character.
 The NULL (0) character is not allowed because text data types cannot store such bytes.
  得到某ACSII值对应的字符
例子:select chr(65); = A
函数:convert(string bytea, src_encoding name, dest_encoding name)
说明:Convert string to dest_encoding. The original encoding is specified by src_encoding.
   The string must be valid in this encoding.
  Conversions can be defined by CREATE CONVERSION. Also there are some predefined conversions.
  See Table 9-7 for available conversions. 转换字符串编码,指定源编码与目标编码
例子:select convert('我是pmars_in_utf8', 'UTF8', 'GBK'); = "\316\322\312\307pmars_in_utf8"
函数:convert_from(string bytea, src_encoding name)
说明:Convert string to the database encoding. The original encoding is specified by src_encoding.
  The string must be valid in this encoding. 转换字符串编码,自己要指定源编码,目标编码默认为数据库指定编码,
例子:select convert_from('\316\322\312\307pmars','GBK'); = "我是pmars"
函数:convert_to(string text, dest_encoding name)
说明:Convert string to dest_encoding.转换字符串编码,源编码默认为数据库指定编码,自己要指定目标编码,
例子:select convert_to('我是pmars_in_utf8','GBK'); = "\316\322\312\307pmars_in_utf8"
函数:decode(string text, type text)
说明:Decode binary data from string previously encoded with encode. Parameter type is same as in encode.
     对字符串按指定的类型进行解码
例子:select decode('MTIzAAE=', 'base64'); = "123\000\001"
函数:encode(data bytea, type text)
说明:Encode binary data to different representation. Supported types are: base64, hex, escape.
   Escape merely outputs null bytes as \000 and doubles backslashes. 与decode相反,对字符串按指定类型进行编码
例子:select encode('123\000\001','base64'); = "MTIzAAE="
函数:initcap(string)
说明:Convert the first letter of each word to uppercase and the rest to lowercase.
 Words are sequences of alphanumeric characters separated by non-alphanumeric characters.
  将字符串所有的单词进行格式化,首字母大写,其它为小写
例子:select initcap('I AM PMARs'); = "I Am Pmars"
函数:length(string)
说明:Number of characters in string 讲算字符串长度
例子:select length('我是pmars'); = 7
函数:length(stringbytea, encoding name )
说明:Number of characters in string in the given encoding.
   The string must be valid in this encoding. 计算字符串长度,指定字符串使用的编码
例子:select length('我是pmars','GBK'); = 8
函数:lpad(string text, length int [, fill text])
说明:Fill up the string to length length by prepending the characters fill (a space by default).
If the string is already longer than length then it is truncated (on the right).
 对字符串左边进行某类字符自动填充,即不足某一长度,则在左边自动补上指定的字符串,直至达到指定长度,可同时指定多个自动填充的字符
例子:select lpad('pmars', 10, 'to'); = "tototpmars"
函数:ltrim(string text [, characters text])
说明:Remove the longest string containing only characters from characters (a space by default) from the start of string
删除字符串左边某一些的字符,可以时指定多个要删除的字符
例子:select ltrim('pmars','amp'); = "rs"
函数:md5(string)
说明:Calculates the MD5 hash of string, returning the result in hexadecimal 将字符串进行md5编码
例子:select md5('pmars'); = "1018ceb949f1472f7252f7da1f5eff42"
函数:pg_client_encoding()
说明:Current client encoding name 得到pg客户端编码
例子:select pg_client_encoding(); = "UTF8"
函数:quote_ident(string text)
说明:Return the given string suitably quoted to be used as an identifier in an SQL statement string.
Quotes are added only if necessary (i.e., if the string contains non-identifier characters or would be case-folded).
Embedded quotes are properly doubled. 对某一字符串加上两引号
例子:quote_ident('Foo bar') = "Foo bar"
函数:quote_literal(string text)
说明:Return the given string suitably quoted to be used as a string literal in an SQL statement string.
Embedded single-quotes and backslashes are properly doubled.
对字符串里两边加上单引号,如果字符串里面出现sql编码的单个单引号,则会被表达成两个单引号
例子:quote_literal('O\'Reilly') = 'O''Reilly'
函数:quote_literal(value anyelement)
说明:Coerce the given value to text and then quote it as a literal.
     Embedded single-quotes and backslashes are properly doubled.
    将一数值转换为字符串,并为其两边加上单引号,如果数值中间出现了单引号,也会被表示成两个单引号
例子:quote_literal(42.5) = '42.5'
函数:regexp_matches(string text, pattern text [, flags text])
说明:Return all captured substrings resulting from matching a POSIX regular expression against the string.
See Section 9.7.3 for more information. 对字符串按正则表达式进行匹配,如果存在则会在结果数组中表示出来
例子:regexp_matches('foobarbequebaz', '(bar)(beque)') = {bar,beque}
函数:regexp_replace(string text, pattern text, replacement text [, flags text])
说明:Replace substring(s) matching a POSIX regular expression. See Section 9.7.3 for more information.
    利用正则表达式对字符串进行替换
例子:regexp_replace('Thomas', '.[mN]a.', 'M') = ThM
函数:regexp_split_to_array(string text, pattern text [, flags text ])
说明:Split string using a POSIX regular expression as the delimiter. See Section 9.7.3 for more information.
     利用正则表达式将字符串分割成数组
例子:regexp_split_to_array('hello world', E'\\s+') = {hello,world}
函数:regexp_split_to_table(string text, pattern text [, flags text])
说明:Split string using a POSIX regular expression as the delimiter. See Section 9.7.3 for more information.
      利用正则表达式将字符串分割成表格
例子:regexp_split_to_table('hello world', E'\\s+') =
hello
world
(2 rows)
函数:repeat(string text, number int)
说明:Repeat string the specified number of times 重复字符串一指定次数
例子:repeat('Pg', 4) = PgPgPgPg
函数:replace(string text, from text, to text)
说明:Replace all occurrences in string of substring from with substring to 将字符的某一子串替换成另一子串
例子:('abcdefabcdef', 'cd', 'XX') = abXXefabXXef
函数:rpad(string text, length int [, fill text])
说明:Fill up the string to length length by appending the characters fill (a space by default).
If the string is already longer than length then it is truncated. 对字符串进行填充,填充内容为指定的字符串
例子:rpad('hi', 5, 'xy') = hixyx
函数:rtrim(string text [, characters text])
说明:Remove the longest string containing only characters from characters
(a space by default) from the end of string 去除字符串右边指定的字符
例子:rtrim('trimxxxx', 'x') = trim
函数:split_part(string text, delimiter text, field int)
说明:Split string on delimiter and return the given field (counting from one)
 对字符串按指定子串进行分割,并返回指定的数值位置的值
例子:split_part('abc~@~def~@~ghi', '~@~', 2) = def
函数:strpos(string, substring)
说明:Location of specified substring (same as position(substring in string),
but note the reversed argument order) 指定字符串在目标字符串的位置
例子:strpos('high', 'ig') = 2
函数:substr(string, from [, count])
说明:Extract substring (same as substring(string from from for count)) 截取子串
例子:substr('alphabet', 3, 2) = ph
函数:to_ascii(string text [, encoding text])
说明:Convert string to ASCII from another encoding
(only supports conversion from LATIN1, LATIN2, LATIN9, and WIN1250 encodings)
将字符串转换成ascii编码字符串
例子:to_ascii('Karel') = Karel
函数:to_hex(number int or bigint)
说明:Convert number to its equivalent hexadecimal representation  对数值进行十六进制编码
例子:to_hex(2147483647) = 7fffffff
函数:translate(string text, from text, to text)
说明:Any character in string that matches a
     character in the from set is replaced by the corresponding character in the to set
     将字符串中某些匹配的字符替换成指定字符串,目标字符与源字符都可以同时指定多个
例子:translate('12345', '14', 'ax') = a23x5

转载:https://www.cnblogs.com/pmars/archive/2013/02/17/2914046.html

psql: FATAL: no pg_hba.conf entry for host

联系:手机/微信(+86 17813235971) QQ(107644445)

标题:psql: FATAL: no pg_hba.conf entry for host

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

pg远程登录拒绝

C:\Users\Administrator>psql -h 192.168.222.2 -U u_xifenfei -d db_xifenfei
psql: FATAL:  no pg_hba.conf entry for host "192.168.222.1", user "u_xifenfei", database "db_xifenfei", SSL off

解决方法

vi $PGDATA/pg_hba.conf
--加上
host   all             all           192.168.222.0/24          md5
--重启pg服务
-bash-4.2$ pg_ctl stop
waiting for server to shut down.... done
server stopped
-bash-4.2$ pg_ctl start
waiting for server to start....2018-08-05 23:46:08.237 EDT [44173] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2018-08-05 23:46:08.237 EDT [44173] LOG:  listening on IPv6 address "::", port 5432
2018-08-05 23:46:08.239 EDT [44173] LOG:  listening on Unix socket "/tmp/.s.PGSQL.5432"
2018-08-05 23:46:08.250 EDT [44173] LOG:  redirecting log output to logging collector process
2018-08-05 23:46:08.250 EDT [44173] HINT:  Future log output will appear in directory "log".
 done
server started

测试远程访问

C:\Users\Administrator>psql -h 192.168.222.2 -U u_xifenfei -d db_xifenfei
用户 u_xifenfei 的口令:
psql (10.4)
输入 "help" 来获取帮助信息.
db_xifenfei=# select version();
                                                 version
---------------------------------------------------------------------------------------------------------
 PostgreSQL 10.4 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-18), 64-bit
(1 行记录)

pg_rman 备份还原测试

联系:手机/微信(+86 17813235971) QQ(107644445)

标题:pg_rman 备份还原测试

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

通过试验验证pg_rman的基础使用
创建测试环境
模拟创建新表空间,新数据库,新用户,新schema,并且创建表插入测试数据,这样的环境下,pg_rman 备份还原效果

[root@localhost ~]# psql
Password:
psql.bin (10.4)
Type "help" for help.
postgres=#  CREATE USER u_xifenfei WITH
postgres-#  LOGIN
postgres-#  SUPERUSER
postgres-#  CREATEDB
postgres-#  CREATEROLE
postgres-#  INHERIT
postgres-#  REPLICATION
postgres-#  CONNECTION LIMIT -1
postgres-#  PASSWORD 'xifenfei';
CREATE ROLE
postgres=#  CREATE TABLESPACE tbs_xifenfei
postgres-#    OWNER u_xifenfei
postgres-#    LOCATION '/opt/PostgreSQL/10/tbs_xifenfei';
CREATE TABLESPACE
postgres=#  CREATE DATABASE db_xifenfei
postgres-#      WITH
postgres-#      OWNER = u_xifenfei
postgres-#      ENCODING = 'UTF8'
postgres-#      TABLESPACE = tbs_xifenfei
postgres-#      CONNECTION LIMIT = -1;
CREATE DATABASE
postgres=# \q
[root@localhost ~]# psql -U u_xifenfei
Password for user u_xifenfei:
psql.bin (10.4)
Type "help" for help.
postgres=# \q
[root@localhost ~]# psql -U u_xifenfei -d db_xifenfei
Password for user u_xifenfei:
psql.bin (10.4)
Type "help" for help.
db_xifenfei=#
db_xifenfei=#
db_xifenfei=# create schema u_xifenfei;
CREATE SCHEMA
db_xifenfei=# create table t_xifenfei as select * from pg_tables;
SELECT 69
db_xifenfei=# insert into t_xifenfei select * from t_xifenfei;
INSERT 0 69
db_xifenfei=# insert into t_xifenfei select * from t_xifenfei;
INSERT 0 138
db_xifenfei=# insert into t_xifenfei select * from t_xifenfei;
INSERT 0 276
db_xifenfei=# insert into t_xifenfei select * from t_xifenfei;
INSERT 0 552
db_xifenfei=# insert into t_xifenfei select * from t_xifenfei;
INSERT 0 1104
db_xifenfei=# insert into t_xifenfei select * from t_xifenfei;
INSERT 0 2208
db_xifenfei=# insert into t_xifenfei select * from t_xifenfei;
INSERT 0 4416
db_xifenfei=# insert into t_xifenfei select * from t_xifenfei;
INSERT 0 8832
db_xifenfei=# insert into t_xifenfei select * from t_xifenfei;
INSERT 0 17664
db_xifenfei=# insert into t_xifenfei select * from t_xifenfei;
INSERT 0 35328
db_xifenfei=# insert into t_xifenfei select * from t_xifenfei;
INSERT 0 70656
db_xifenfei=# insert into t_xifenfei select * from t_xifenfei;
INSERT 0 141312
db_xifenfei=# insert into t_xifenfei select * from t_xifenfei;
INSERT 0 282624
db_xifenfei=# insert into t_xifenfei select * from t_xifenfei;
INSERT 0 565248
db_xifenfei=# select count(*) from t_xifenfei;
  count
---------
 1130496
(1 row)
db_xifenfei=#

第一次全备数据库

[root@localhost backup]# pg_rman backup --backup-mode=full \
[root@localhost backup]# --backup-path=/backup -d db_xifenfei -U u_xifenfei -h 127.0.0.1
Password for user u_xifenfei:
INFO: copying database files
INFO: copying archived WAL files
INFO: backup complete
INFO: Please execute 'pg_rman validate' to verify the files are correctly copied.

模拟继续插入数据
t_xifenfei表一共有数据2260992条

b_xifenfei=# insert into t_xifenfei select * from t_xifenfei;
INSERT 0 1130496
db_xifenfei=# select count(*) from t_xifenfei;
  count
---------
 2260992
(1 row)

查看全备情况

[root@localhost 10]#  pg_rman show --backup-path=/backup
=====================================================================
 StartTime           EndTime              Mode    Size   TLI  Status
=====================================================================
2018-08-05 10:47:29  2018-08-05 10:47:43  FULL   611MB     1  DONE

备份归档日志

[root@localhost 10]# pg_rman backup --backup-mode=archive \
[root@localhost 10]# --backup-path=/backup -d db_xifenfei -U u_xifenfei -h 127.0.0.1
Password for user u_xifenfei:
INFO: copying archived WAL files
INFO: backup complete
INFO: Please execute 'pg_rman validate' to verify the files are correctly copied.

检查全备和归档备份情况

[root@localhost 10]#  pg_rman show --backup-path=/backup
=====================================================================
 StartTime           EndTime              Mode    Size   TLI  Status
=====================================================================
2018-08-05 10:54:51  2018-08-05 10:56:07  ARCH   620MB     1  DONE
2018-08-05 10:47:29  2018-08-05 10:47:43  FULL   611MB     1  DONE

停掉pg

[root@localhost data]# su - postgres
Last login: Sun Aug  5 02:19:57 EDT 2018 on pts/1
-bash-4.2$ source pg_env.sh
-bash-4.2$ pg_ctl stop
waiting for server to shut down.... done
server stopped
-bash-4.2$ ps -ef|grep postgres
root      39902  30494  0 11:05 pts/0    00:00:00 su - postgres
postgres  39903  39902  0 11:05 pts/0    00:00:00 -bash
postgres  40021  39903  0 11:06 pts/0    00:00:00 ps -ef
postgres  40022  39903  0 11:06 pts/0    00:00:00 grep --color=auto postgres

删除原库并创建相关目录
注意:对应的空间目录权限为700,所有者和组为postgres

[root@localhost 10]# pwd
/opt/PostgreSQL/10
[root@localhost 10]# mv data data_bak
[root@localhost 10]# mv tbs_xifenfei tbs_xifenfei_bak
[root@localhost 10]# mkdir data
[root@localhost 10]# mkdir tbs_xifenfei
[root@localhost 10]# chmod 700 data
[root@localhost 10]# chmod 700 tbs_xifenfei
[root@localhost 10]# chown postgres:postgres data
[root@localhost 10]# chown postgres:postgres tbs_xifenfei

还原数据库

-bash-4.2$ pg_rman restore --backup-path=/backup
WARNING: pg_controldata file "/opt/PostgreSQL/10/data/global/pg_control" does not exist
INFO: the recovery target timeline ID is not given
INFO: use timeline ID of latest full backup as recovery target: 1
INFO: calculating timeline branches to be used to recovery target point
INFO: searching latest full backup which can be used as restore start point
INFO: found the full backup can be used as base in recovery: "2018-08-05 10:47:29"
INFO: copying online WAL files and server log files
INFO: clearing restore destination
INFO: validate: "2018-08-05 10:47:29" backup and archive log files by SIZE
INFO: backup "2018-08-05 10:47:29" is valid
INFO: restoring database files from the full mode backup "2018-08-05 10:47:29"
INFO: searching incremental backup to be restored
INFO: searching backup which contained archived WAL files to be restored
INFO: backup "2018-08-05 10:47:29" is valid
INFO: restoring WAL files from backup "2018-08-05 10:47:29"
INFO: restoring online WAL files and server log files
INFO: generating recovery.conf
INFO: restore complete
HINT: Recovery will start automatically when the PostgreSQL server is started.

启动pg

-bash-4.2$ pg_ctl start
waiting for server to start....2018-08-05 11:23:40.190 EDT [40855] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2018-08-05 11:23:40.190 EDT [40855] LOG:  listening on IPv6 address "::", port 5432
2018-08-05 11:23:40.193 EDT [40855] LOG:  listening on Unix socket "/tmp/.s.PGSQL.5432"
2018-08-05 11:23:40.200 EDT [40855] LOG:  redirecting log output to logging collector process
2018-08-05 11:23:40.200 EDT [40855] HINT:  Future log output will appear in directory "log".
 done
server started
-bash-4.2$ ps -ef|grep postgres
root      40328  21806  0 11:14 pts/3    00:00:00 su - postgres
postgres  40329  40328  0 11:14 pts/3    00:00:00 -bash
postgres  40855      1  0 11:23 pts/3    00:00:00 /opt/PostgreSQL/10/bin/postgres
postgres  40856  40855  0 11:23 ?        00:00:00 postgres: logger process
postgres  40857  40855 57 11:23 ?        00:00:16 postgres: startup process   waiting for 000000010000000000000025
postgres  40859  40855  0 11:23 ?        00:00:00 postgres: checkpointer process
postgres  40860  40855  7 11:23 ?        00:00:02 postgres: writer process
postgres  40862  40855  0 11:23 ?        00:00:00 postgres: stats collector process
postgres  40892  40329  0 11:24 pts/3    00:00:00 ps -ef
postgres  40893  40329  0 11:24 pts/3    00:00:00 grep --color=auto postgres
-bash-4.2$

验证数据库还原结果

-bash-4.2$ psql -U u_xifenfei -d db_xifenfei
Password for user u_xifenfei:
psql.bin (10.4)
Type "help" for help.
db_xifenfei=#  select count(*) from t_xifenfei;
  count
---------
 2260992
(1 row)
db_xifenfei=#

破坏环境之前表条数和还原之后完全匹配,证pg_rman在功能上备份恢复没有问题

PostgreSQL修改归档模式

联系:手机/微信(+86 17813235971) QQ(107644445)

标题:PostgreSQL修改归档模式

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

pg版本

postgres=# select version();
                                                 version
---------------------------------------------------------------------------------------------------------
 PostgreSQL 10.4 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-18), 64-bit
(1 row)

归档配置参数

--pgdata中的postgresql.conf
wal_level = replica
archive_mode = on
archive_command = 'test ! -f /pg_archivedir/%f && cp %p /pg_archivedir/%f'
--重启pg systemctl restart postgresql-10.service
postgres=# show wal_level;
 wal_level
-----------
 replica
(1 row)
postgres=# show archive_mode;
 archive_mode
--------------
 on
(1 row)
postgres=# show archive_command;
                    archive_command
--------------------------------------------------------
 test ! -f /pg_archivedir/%f && cp %p /pg_archivedir/%f
(1 row)

测试归档效果

[root@localhost pg_wal]# ps -ef|grep "postgres: archiver process"
postgres  35300  35293  0 08:43 ?        00:00:00 postgres: archiver process   last was 000000010000000000000004
root      37504  20853  0 09:55 pts/1    00:00:00 grep --color=auto postgres: archiver process
[root@localhost pg_wal]# pwd
/opt/PostgreSQL/10/data/pg_wal
[root@localhost pg_wal]# ls -ltr
total 49156
-rw------- 1 postgres postgres      309 Aug  5 06:05 000000010000000000000002.00000028.backup
-rw------- 1 postgres postgres 16777216 Aug  5 08:43 000000010000000000000006
-rw------- 1 postgres postgres 16777216 Aug  5 08:50 000000010000000000000004
drwx------ 2 postgres postgres       94 Aug  5 08:50 archive_status
-rw------- 1 postgres postgres 16777216 Aug  5 08:53 000000010000000000000005
[root@localhost pg_wal]# ls -l /pg_archivedir/
total 65540
-rw------- 1 postgres postgres 16777216 Aug  5 08:37 000000010000000000000001
-rw------- 1 postgres postgres 16777216 Aug  5 08:37 000000010000000000000002
-rw------- 1 postgres postgres      309 Aug  5 08:37 000000010000000000000002
-rw------- 1 postgres postgres 16777216 Aug  5 08:43 000000010000000000000003
-rw------- 1 postgres postgres 16777216 Aug  5 08:50 000000010000000000000004
--切换日志
postgres=# select pg_switch_wal();
 pg_switch_wal
---------------
 0/5000158
(1 row)
[root@localhost pg_wal]# ps -ef|grep "postgres: archiver process"
postgres  35300  35293  0 08:43 ?        00:00:00 postgres: archiver process   last was 000000010000000000000005
root      37531  20853  0 09:56 pts/1    00:00:00 grep --color=auto postgres: archiver process
[root@localhost pg_wal]# ls -ltr
total 49156
-rw------- 1 postgres postgres      309 Aug  5 06:05 000000010000000000000002
-rw------- 1 postgres postgres 16777216 Aug  5 08:50 000000010000000000000004
-rw------- 1 postgres postgres 16777216 Aug  5 09:55 000000010000000000000005
drwx------ 2 postgres postgres      130 Aug  5 09:55 archive_status
-rw------- 1 postgres postgres 16777216 Aug  5 09:55 000000010000000000000006
[root@localhost pg_wal]# ls -l /pg_archivedir/
total 81924
-rw------- 1 postgres postgres 16777216 Aug  5 08:37 000000010000000000000001
-rw------- 1 postgres postgres 16777216 Aug  5 08:37 000000010000000000000002
-rw------- 1 postgres postgres      309 Aug  5 08:37 000000010000000000000002
-rw------- 1 postgres postgres 16777216 Aug  5 08:43 000000010000000000000003
-rw------- 1 postgres postgres 16777216 Aug  5 08:50 000000010000000000000004
-rw------- 1 postgres postgres 16777216 Aug  5 09:55 000000010000000000000005

PostgreSQL备份工具pg_rman安装

联系:手机/微信(+86 17813235971) QQ(107644445)

标题:PostgreSQL备份工具pg_rman安装

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

学习oracle的同学都清楚,oracle的rman是最好的备份工具(一般第三方备份软件也基本上是通过调用rman的备份接口实现的)
pg_rman下载地址https://github.com/ossc-db/pg_rman/releases,下载和当前pg版本/操作系统版本对应的pg_rman包
安装pg_rman缺少包
缺少libpq.so.5()和postgresql10-libs相关的包

[root@localhost lib]# rpm -ivh /tmp/pg_rman-1.3.6-1.pg10.rhel7.x86_64.rpm
error: Failed dependencies:
        libpq.so.5()(64bit) is needed by pg_rman-1.3.6-1.pg10.rhel7.x86_64
        postgresql10-libs is needed by pg_rman-1.3.6-1.pg10.rhel7.x86_64

安装pg相关源

[root@localhost tmp]# rpm -ivh pgdg-redhat10-10-2.noarch.rpm
warning: pgdg-redhat10-10-2.noarch.rpm: Header V4 DSA/SHA1 Signature, key ID 442df0f8: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:pgdg-redhat10-10-2               ################################# [100%]
[root@localhost yum.repos.d]# pwd
/etc/yum.repos.d
[root@localhost yum.repos.d]# more pgdg-10-redhat.repo
[pgdg10]
name=PostgreSQL 10 $releasever - $basearch
baseurl=https://download.postgresql.org/pub/repos/yum/10/redhat/rhel-$releasever-$basearch
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-PGDG-10
[pgdg10-source]
name=PostgreSQL 10 $releasever - $basearch - Source
failovermethod=priority
baseurl=https://download.postgresql.org/pub/repos/yum/srpms/10/redhat/rhel-$releasever-$basearch
enabled=0
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-PGDG-10
[pgdg10-updates-testing]
name=PostgreSQL 10 $releasever - $basearch
baseurl=https://download.postgresql.org/pub/repos/yum/testing/10/redhat/rhel-$releasever-$basearch
enabled=0
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-PGDG-10
[pgdg10-source-updates-testing]
name=PostgreSQL 10 $releasever - $basearch - Source
failovermethod=priority
baseurl=https://download.postgresql.org/pub/repos/yum/srpms/testing/10/redhat/rhel-$releasever-$basearch
enabled=0
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-PGDG-10

安装postgresql10-libs包

[root@localhost tmp]# yum install postgresql10-libs
Loaded plugins: langpacks, ulninfo
pgdg10                                         | 4.1 kB  00:00:00
(1/2): pgdg10/7Server/x86_64/group_gz          |  245 B  00:00:01
(2/2): pgdg10/7Server/x86_64/primary_db        | 162 kB  00:00:13
Resolving Dependencies
--> Running transaction check
---> Package postgresql10-libs.x86_64 0:10.4-1PGDG.rhel7 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
==========================================================================
 Package                 Arch     Version               Repository   Size
==========================================================================
Installing:
 postgresql10-libs       x86_64   10.4-1PGDG.rhel7      pgdg10      354 k
Transaction Summary
==========================================================================
Install  1 Package
Total download size: 354 k
Installed size: 1.3 M
Is this ok [y/d/N]: y
Downloading packages:
postgresql10-libs-10.4-1PGDG.rhel7.x86_64.rpm      | 354 kB  00:00:02
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Warning: RPMDB altered outside of yum.
  Installing : postgresql10-libs-10.4-1PGDG.rhel7.x86_64      1/1
  Verifying  : postgresql10-libs-10.4-1PGDG.rhel7.x86_64      1/1
Installed:
  postgresql10-libs.x86_64 0:10.4-1PGDG.rhel7
Complete!

安装pg_rman

[root@localhost tmp]# rpm -ivh pg_rman-1.3.6-1.pg10.rhel7.x86_64.rpm
Preparing...                          ################################# [100%]
Updating / installing...
   1:pg_rman-1.3.6-1.pg10.rhel7       ################################# [100%]

查找默认pg_rman安装路径

[root@localhost bin]# find / -name "pg_rman"
/usr/pgsql-10/bin/pg_rman
[root@localhost bin]# cd /usr/pgsql-10/bin
[root@localhost bin]# ls
pg_rman

迁移到pg安装目录中

[root@localhost bin]# mv pg_rman /opt/PostgreSQL/10/bin

pg_rman命令测试

[root@localhost bin]# pg_rman --help
pg_rman manage backup/recovery of PostgreSQL database.
Usage:
  pg_rman OPTION init
  pg_rman OPTION backup
  pg_rman OPTION restore
  pg_rman OPTION show [DATE]
  pg_rman OPTION show detail [DATE]
  pg_rman OPTION validate [DATE]
  pg_rman OPTION delete DATE
  pg_rman OPTION purge
Common Options:
  -D, --pgdata=PATH         location of the database storage area
  -A, --arclog-path=PATH    location of archive WAL storage area
  -S, --srvlog-path=PATH    location of server log storage area
  -B, --backup-path=PATH    location of the backup storage area
  -c, --check               show what would have been done
  -v, --verbose             show what detail messages
  -P, --progress            show progress of processed files
Backup options:
  -b, --backup-mode=MODE    full, incremental, or archive
  -s, --with-serverlog      also backup server log files
  -Z, --compress-data       compress data backup with zlib
  -C, --smooth-checkpoint   do smooth checkpoint before backup
  -F, --full-backup-on-error   switch to full backup mode
                               if pg_rman cannot find validate full backup
                               on current timeline
      NOTE: this option is only used in --backup-mode=incremental or archive.
  --keep-data-generations=NUM keep NUM generations of full data backup
  --keep-data-days=NUM        keep enough data backup to recover to N days ago
  --keep-arclog-files=NUM   keep NUM of archived WAL
  --keep-arclog-days=DAY    keep archived WAL modified in DAY days
  --keep-srvlog-files=NUM   keep NUM of serverlogs
  --keep-srvlog-days=DAY    keep serverlog modified in DAY days
  --standby-host=HOSTNAME   standby host when taking backup from standby
  --standby-port=PORT       standby port when taking backup from standby
Restore options:
  --recovery-target-time    time stamp up to which recovery will proceed
  --recovery-target-xid     transaction ID up to which recovery will proceed
  --recovery-target-inclusive whether we stop just after the recovery target
  --recovery-target-timeline  recovering into a particular timeline
  --hard-copy                 copying archivelog not symbolic link
Catalog options:
  -a, --show-all            show deleted backup too
Delete options:
  -f, --force               forcibly delete backup older than given DATE
Connection options:
  -d, --dbname=DBNAME       database to connect
  -h, --host=HOSTNAME       database server host or socket directory
  -p, --port=PORT           database server port
  -U, --username=USERNAME   user name to connect as
  -w, --no-password         never prompt for password
  -W, --password            force password prompt
Generic options:
  -q, --quiet               don't show any INFO or DEBUG messages
  --debug                   show DEBUG messages
  --help                    show this help, then exit
  --version                 output version information, then exit
Read the website for details. <http://github.com/ossc-db/pg_rman>
Report bugs to <http://github.com/ossc-db/pg_rman/issues>.

PostgreSQL逻辑导出导入

联系:手机/微信(+86 17813235971) QQ(107644445)

标题:PostgreSQL逻辑导出导入

作者:惜分飞©版权所有[未经本人同意,不得以任何形式转载,否则有进一步追究法律责任的权利.]

pg_dump命令说明

C:\Users\Administrator>pg_dump --help
pg_dump 把一个数据库转储为纯文本文件或者是其它格式.
用法:
  pg_dump [选项]... [数据库名字]
一般选项:
  -f, --file=FILENAME          输出文件或目录名
  -F, --format=c|d|t|p         输出文件格式 (定制, 目录, tar)
                               明文 (默认值))
  -j, --jobs=NUM               执行多个并行任务进行备份转储工作
  -v, --verbose                详细模式
  -V, --version                输出版本信息,然后退出
  -Z, --compress=0-9           被压缩格式的压缩级别
  --lock-wait-timeout=TIMEOUT  在等待表锁超时后操作失败
  --no-sync                    do not wait for changes to be written safely to disk
  -?, --help                   显示此帮助, 然后退出
控制输出内容选项:
  -a, --data-only              只转储数据,不包括模式
  -b, --blobs                  在转储中包括大对象
  -B, --no-blobs               exclude large objects in dump
  -c, --clean                  在重新创建之前,先清除(删除)数据库对象
  -C, --create                 在转储中包括命令,以便创建数据库
  -E, --encoding=ENCODING      转储以ENCODING形式编码的数据
  -n, --schema=SCHEMA          只转储指定名称的模式
  -N, --exclude-schema=SCHEMA  不转储已命名的模式
  -o, --oids                   在转储中包括 OID
  -O, --no-owner               在明文格式中, 忽略恢复对象所属者
  -s, --schema-only            只转储模式, 不包括数据
  -S, --superuser=NAME         在明文格式中使用指定的超级用户名
  -t, --table=TABLE            只转储指定名称的表
  -T, --exclude-table=TABLE    不转储指定名称的表
  -x, --no-privileges          不要转储权限 (grant/revoke)
  --binary-upgrade             只能由升级工具使用
  --column-inserts             以带有列名的INSERT命令形式转储数据
  --disable-dollar-quoting     取消美元 (符号) 引号, 使用 SQL 标准引号
  --disable-triggers           在只恢复数据的过程中禁用触发器
  --enable-row-security        启用行安全性(只转储用户能够访问的内容)
  --exclude-table-data=TABLE   不转储指定名称的表中的数据
  --if-exists              当删除对象时使用IF EXISTS
  --inserts                    以INSERT命令,而不是COPY命令的形式转储数据
  --no-publications            do not dump publications
  --no-security-labels         不转储安全标签的分配
  --no-subscriptions           do not dump subscriptions
  --no-synchronized-snapshots  在并行工作集中不使用同步快照
  --no-tablespaces             不转储表空间分配信息
  --no-unlogged-table-data     不转储没有日志的表数据
  --quote-all-identifiers      所有标识符加引号,即使不是关键字
  --section=SECTION            备份命名的节 (数据前, 数据, 及 数据后)
  --serializable-deferrable   等到备份可以无异常运行
  --snapshot=SNAPSHOT          为转储使用给定的快照
  --strict-names               要求每个表和/或schema包括模式以匹配至少一个实体
  --use-set-session-authorization
                               使用 SESSION AUTHORIZATION 命令代替
                ALTER OWNER 命令来设置所有权
联接选项:
  -d, --dbname=DBNAME       对数据库 DBNAME备份
  -h, --host=主机名        数据库服务器的主机名或套接字目录
  -p, --port=端口号        数据库服务器的端口号
  -U, --username=名字      以指定的数据库用户联接
  -w, --no-password        永远不提示输入口令
  -W, --password           强制口令提示 (自动)
  --role=ROLENAME          在转储前运行SET ROLE
如果没有提供数据库名字, 那么使用 PGDATABASE 环境变量
的数值.
报告错误至 <pgsql-bugs@postgresql.org>.
C:\Users\Administrator>

导出环境准备

C:\Users\Administrator>psql -U u_xifenfei -d db_xifenfei
用户 u_xifenfei 的口令:
psql (10.4)
输入 "help" 来获取帮助信息.
db_xifenfei=# create table t_dump as select * from pg_tables;
SELECT 69
db_xifenfei=# insert into t_dump select * from t_dump;
INSERT 0 69
…………
db_xifenfei=# insert into t_dump select * from t_dump;
INSERT 0 565248
db_xifenfei=# create table public.t_dump_pub as select * from t_dump;
SELECT 1130496
db_xifenfei=# select count(*) from t_dump;
  count
---------
 1130496
(1 行记录)
db_xifenfei=# select count(*) from public.t_dump_pub;
  count
---------
 1130496
(1 行记录)

导出命令

pg_dump  -U u_xifenfei -f e:/u_xifenfei.sql db_xifenfei
pg_dump  -U u_xifenfei -f e:/u_xifenfei2.sql --inserts db_xifenfei

psql导入说明

C:\Users\Administrator>psql -?
psql是PostgreSQL 的交互式客户端工具。
使用方法:
  psql [选项]... [数据库名称 [用户名称]]
通用选项:
  -c,--command=命令        执行单一命令(SQL或内部指令)然后结束
 -d, --dbname=数据库名称   指定要连接的数据库 (默认:"Administrator")
  -f, --file=文件名      从文件中执行命令然后退出
  -l, --list             列出所有可用的数据库,然后退出
  -v, --set=, --variable=NAME=VALUE
                           设置psql变量NAME为VALUE
                           (例如,-v ON_ERROR_STOP=1)
  -V, --version            输出版本信息, 然后退出
  -X, --no-psqlrc         不读取启动文档(~/.psqlrc)
  -1 ("one"), --single-transaction
                          作为一个单一事务来执行命令文件(如果是非交互型的)
  -?, --help[=options]     显示此帮助,然后退出
      --help=commands      列出反斜线命令,然后退出
      --help=variables     列出特殊变量,然后退出
输入和输出选项:
  -a, --echo-all          显示所有来自于脚本的输入
  -b, --echo-errors        回显失败的命令
  -e, --echo-queries      显示发送给服务器的命令
 -E, --echo-hidden        显示内部命令产生的查询
  -L, --log-file=文件名  将会话日志写入文件
  -n, --no-readline       禁用增强命令行编辑功能(readline)
  -o, --output=FILENAME 将查询结果写入文件(或 |管道)
  -q, --quiet             以沉默模式运行(不显示消息,只有查询结果)
  -s, --single-step       单步模式 (确认每个查询)
  -S, --single-line        单行模式 (一行就是一条 SQL 命令)
输出格式选项 :
 -A, --no-align           使用非对齐表格输出模式
  -F, --field-separator=STRING
             为字段设置分隔符,用于不整齐的输出(默认:"|")
  -H, --html             HTML 表格输出模式
  -P, --pset=变量[=参数]    设置将变量打印到参数的选项(查阅 \pset 命令)
  -R, --record-separator=STRING
             为不整齐的输出设置字录的分隔符(默认:换行符号)
  -t, --tuples-only      只打印记录i
  -T, --table-attr=文本   设定 HTML 表格标记属性(例如,宽度,边界)
  -x, --expanded           打开扩展表格输出
  -z, --field-separator-zero
                           为不整齐的输出设置字段分隔符为字节0
  -0, --record-separator-zero
                           为不整齐的输出设置记录分隔符为字节0
联接选项:
  -h, --host=主机名        数据库服务器主机或socket目录(默认:"本地接口")
  -p, --port=端口        数据库服务器的端口(默认:"5432")
  -U, --username=用户名    指定数据库用户名(默认:"Administrator")
  -w, --no-password       永远不提示输入口令
  -W, --password           强制口令提示 (自动)
更多信息,请在psql中输入"\?"(用于内部指令)或者 "\help"(用于SQL命令),
或者参考PostgreSQL文档中的psql章节.
臭虫报告至 <pgsql-bugs@postgresql.org>.

导入操作

C:\Users\Administrator>psql -U u_xifenfei -d db_xifenfei
用户 u_xifenfei 的口令:
psql (10.4)
输入 "help" 来获取帮助信息.
db_xifenfei=# create database test_pump;
CREATE DATABASE
C:\Users\Administrator>psql -d test_pump -U u_xifenfei -f e:/u_xifenfei.sql
用户 u_xifenfei 的口令:
SET
SET
SET
SET
SET
 set_config
------------
(1 行记录)
SET
SET
SET
CREATE SCHEMA
ALTER SCHEMA
CREATE EXTENSION
COMMENT
SET
SET
CREATE TABLE
ALTER TABLE
CREATE TABLE
ALTER TABLE
COPY 1130496
COPY 1130496

导出/导入数据核对

C:\Users\Administrator>psql -d test_pump -U u_xifenfei
用户 u_xifenfei 的口令:
psql (10.4)
输入 "help" 来获取帮助信息.
test_pump=# \d
                   关联列表
  架构模式  |    名称    |  类型  |   拥有者
------------+------------+--------+------------
 public     | t_dump_pub | 数据表 | u_xifenfei
 u_xifenfei | t_dump     | 数据表 | u_xifenfei
(2 行记录)