Podrška #20056
Zatvoren
zvijer-2, zvijer-3: drbd, ocsfs2
Dodano od Ernad Husremović prije više od 15 godina. Izmjenjeno prije više od 14 godina.
0%
Fajlovi
| 323488-001US_secured.pdf (1,39 MB) 323488-001US_secured.pdf | intel+canonical eucalyptus cloud | Ernad Husremović, 05.04.2010 19:58 | |
| UbuntuEnterpriseCloudWP-Architecture-20090820.pdf (257 KB) UbuntuEnterpriseCloudWP-Architecture-20090820.pdf | Ernad Husremović, 06.04.2010 15:21 |
Povezani tiketi 2 (0 otvoreno — 2 zatvorenih)
Izmjenjeno od Ernad Husremović prije više od 15 godina
eucalyptus-cloud
tcp 0 0 192.168.122.1:53 0.0.0.0:* LISTEN 5925/dnsmasq
Izmjenjeno od Ernad Husremović prije više od 15 godina
root@smraka-1b:~# apt-get install eucalyptus-sc eucalyptus-nc
Izmjenjeno od Ernad Husremović prije više od 15 godina
root@smraka-1b:~# sudo apt-get install eucalyptus-walrus
Izmjenjeno od Ernad Husremović prije više od 15 godina
root@smraka-1b:~# apt-get install eucalyptus-cc
Čitam spiskove paketa... Done Gradim stablo zavisnosti Reading state information... Done Slijedeći dodatni paketi će biti instalirani: dhcp3-server powerwake Predloženi paketi: dhcp3-server-ldap apparmor vtun Slijedeći NOVI paketi će biti instalirani: dhcp3-server eucalyptus-cc powerwake 0 upgraded, 3 newly installed, 0 to remove and 0 not upgraded. Need to get 921kB of archives. After this operation, 3453kB of additional disk space will be used.
Izmjenjeno od Ernad Husremović prije više od 15 godina
root@smraka-1b:~# apt-get install eucalyptus-cc
Izmjenjeno od Ernad Husremović prije više od 15 godina
Izmjenjeno od Ernad Husremović prije više od 15 godina
bringout@nvostro-hernad:~$ set | grep EC2
EC2_ACCESS_KEY=csIphK5u5WkSO4dnq8lRLPvxbVG5QaXIHj7ZDA EC2_SECRET_KEY=6zntOVKOMTviGhB8JMo7GUXGHdo1DrzcHroNWA EC2_URL=http://192.168.45.41:8773/services/Eucalyptus
bringout@nvostro-hernad:~$ euca-describe-regions
REGION Eucalyptus http://192.168.45.49:8773/services/Eucalyptus REGION Walrus http://192.168.45.41:8773/services/Walrus
Izmjenjeno od Ernad Husremović prije više od 15 godina
bringout@nvostro-hernad:~$ euca-describe-availability-zones
AVAILABILITYZONE cluster1 192.168.45.41
Izmjenjeno od Ernad Husremović prije više od 15 godina
Izmjenjeno od Ernad Husremović prije više od 15 godina
root@smraka-1b:/etc/init.d# aoe-stat
e11.1 706.153GB br0 up
Izmjenjeno od Ernad Husremović prije više od 15 godina
root@smraka-1b:/etc/init.d# pvcreate /dev/etherd/e11.1
Physical volume "/dev/etherd/e11.1" successfully created
Izmjenjeno od Ernad Husremović prije više od 15 godina
Izmjenjeno od Ernad Husremović prije više od 15 godina
- Fajl 323488-001US_secured.pdf 323488-001US_secured.pdf dodano
Izmjenjeno od Ernad Husremović prije više od 15 godina
root@smraka-1b:/etc/init.d# euca_conf --register-nodes 192.168.45.42
INFO: We expect all nodes to have eucalyptus installed in //var/lib/eucalyptus/keys for key synchronization. Warning: cannot file file node-cert.pem in //var/lib/eucalyptus/keys/ Warning: cannot file file cluster-cert.pem in //var/lib/eucalyptus/keys/ Warning: cannot file file node-pk.pem in //var/lib/eucalyptus/keys/ Trying rsync to sync keys with "192.168.45.42"...rsync: Failed to exec -l: No such file or directory (2) rsync error: error in IPC code (code 14) at pipe.c(84) [sender=3.0.7] rsync: connection unexpectedly closed (0 bytes received so far) [sender] rsync error: error in IPC code (code 14) at io.c(601) [sender=3.0.7] failed. Trying scp to sync keys to: eucalyptus@192.168.45.42://var/lib/eucalyptus/keys/... ssh: connect to host 192.168.45.42 port 22: No route to host lost connection failed. ERROR: could not synchronize keys with 192.168.45.42! The configuration will not have this node. Hint: to setup passwordless login to the nodes as user eucalyptus, you can run the following commands on node 192.168.45.42: sudo -u eucalyptus mkdir -p ~eucalyptus/.ssh sudo -u eucalyptus tee ~eucalyptus/.ssh/authorized_keys > /dev/null <<EOT ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAufTSMBUtu/DoBQq0IY/nVbtBxDuhcVirXUAbeKgEjM0nTocqKc5R25KOogSon60PVWADkf4CZ7uLDqjASXQxg+7cclWbPpwlvxXA46h7KeuIIfNFyQE+zp3cPUsqaGYc7IfFiTcAIqkkdmYBU7HASJj7ilmf4OXig7vCM2eEuso72eIJEfSC4b3ub5JgpH2p//4yLSm9etmpc8etDvsr2zYhAyUwjYgX95QC/buo+Z0TRUWcPYFUdfe7V4ZWAcfQusRlR53ZkXws4lav7N+qJDpjSxUtpXDHxw1/xkGbqwAV7jk1z3WpPyfewETcUriN7h8vmW5WdkfsRX4qCjajqQ== eucalyptus@smraka-1b.bring.out.ba EOT Be sure that authorized_keys is not group/world readable or writable
Izmjenjeno od Ernad Husremović prije više od 15 godina
root@smraka-1b:/etc/init.d# euca_conf --discover-nodes
haystack= 192.168.45.41 192.168.122.1 haystack= 192.168.45.41 192.168.122.1 New node found on 192.168.45.39; add it? [Yn]
Izmjenjeno od Ernad Husremović prije više od 15 godina
troubleshooting¶
http://open.eucalyptus.com/wiki/EucalyptusTroubleshooting_v1.6
Block storage troubleshooting
- Unable to attach volumes when the front end and the NC are running on the same machine. This is a known issue with ATA over Ethernet (AoE). AoE will not export to the same machine that the server is running on. The workaround is to run the front end and the node controller on different hosts.
- Volume ends up in "deleted" state when created, instead of showing up as "available." Look for error messages in $EUCALYPTUS/var/log/eucalyptus/cloud-error.log. A common problem is that ATA-over-Ethernet may not be able to export the created volume (this will appear as a "Could not export..." message in cloud-error.log). Make sure that "VNET_INTERFACE" in eucalyptus.conf on the front end is correct.
- Failure to create volume/snapshot. Make sure you have enough loopback devices. If you are installing from packages, you will get a warning. On most distributions, the loopback driver is installed as a module. The following will increase the number of loopback devices available,
rmmod loop ; modprobe loop max_loop=256 - If block devices do not automatically appear in your VMs, make sure that you have the "udev" package installed.
- If you are running gentoo and you get "which: no vblade in ((null)).", try compiling "su" without pam.
Izmjenjeno od Ernad Husremović prije više od 15 godina
Izmjenjeno od Ernad Husremović prije više od 15 godina
Izmjenjeno od Ernad Husremović prije više od 15 godina
Izmjenjeno od Ernad Husremović prije više od 15 godina
Izmjenjeno od Ernad Husremović prije više od 15 godina
Izmjenjeno od Ernad Husremović prije više od 15 godina
Izmjenjeno od Ernad Husremović prije više od 15 godina
- Naslov promijenjeno iz eucalyptus lucid u eucalyptus lucid, aoe storage
Izmjenjeno od Ernad Husremović prije više od 15 godina
Izmjenjeno od Ernad Husremović prije više od 15 godina
Izmjenjeno od Ernad Husremović prije više od 15 godina
- Naslov promijenjeno iz eucalyptus lucid, aoe storage u eucalyptus lucid, aoe storage, drbd
Izmjenjeno od Ernad Husremović prije više od 15 godina
Izmjenjeno od Ernad Husremović prije više od 15 godina
root@smraka-1b:~# pvresize --setphysicalvolumesize 200G /dev/sda5
Physical volume "/dev/sda5" changed 1 physical volume(s) resized / 0 physical volume(s) not resized
Izmjenjeno od Ernad Husremović prije više od 15 godina
ali nemoguće je napraviti resize particije ... nije ni bitno
Izmjenjeno od Ernad Husremović prije više od 15 godina
Izmjenjeno od Ernad Husremović prije više od 15 godina
Izmjenjeno od Ernad Husremović prije više od 15 godina
zvijer-2¶
pxe install - lucid-server zvijer-2
vg0 - 250 GB LVM particija u njoj swap=24G, root ostatak
Izmjenjeno od Ernad Husremović prije više od 15 godina
eth1 - 10.0.1.1 privatna IP
Izmjenjeno od Ernad Husremović prije više od 15 godina
Izmjenjeno od Ernad Husremović prije više od 15 godina
Izmjenjeno od Ernad Husremović prije više od 15 godina
sda3 za drbd
root@zvijer-3:~# fdisk /dev/sda
WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
switch off the mode (command 'c') and change display units to
sectors (command 'u').
Naredba (m za pomoć): p
Disk /dev/sda: 750.2 GB, 750156374016 bajta
255 glava, 63 sektor/staza, 91201 cilindara
Jedinice = cilindara od 16065 * 512 = 8225280 bajta
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Identifikator diska: 0x00006a03
Uređaj But Početak Kraj Blokova Id Sistem
/dev/sda1 * 1 31 242688 83 Linux
Particija 1 se ne završava na granici cilindara.
/dev/sda2 31 30425 244140033 5 Prošireno
/dev/sda5 31 30425 244140032 8e Linux LVM
Naredba (m za pomoć): n
Komandna akcija
l logička (5 ili više)
p primarna particija (1-4)
p
Broj particije (1-4): 3
Prvi cilindar (30425-91201, podrazumevano 30425):
Koristim podrazumijevane vrijednosti 30425
Last cilindar, +cilindara or +size{K,M,G} (30425-91201, podrazumevano 91201):
Koristim podrazumijevane vrijednosti 91201
Naredba (m za pomoć): w
Tabela particija je zamijenjena!
Pozivam ioctl() da bih ponovo pročito tabelu particija
WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table. The new table will be used at
the next reboot or after you run partprobe(8) or kpartx(8)
Sinhronizujem diskove.
root@zvijer-3:~# partprobe
Izmjenjeno od Ernad Husremović prije više od 15 godina
root@zvijer-3:/etc/drbd.d# cat /etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
# The loopback network interface
auto lo
iface lo inet loopback
auto eth1
# The primary network interface
auto eth0
iface eth0 inet dhcp
iface eth1 inet static
address 10.0.1.2 <<<<<<<<<<<<<<<<<<<<<<<<<<<<
netmask 255.255.255.0
Izmjenjeno od Ernad Husremović prije više od 15 godina
radi privatni interface:
root@zvijer-3:/etc/drbd.d# ping 10.0.1.1
PING 10.0.1.1 (10.0.1.1) 56(84) bytes of data. 64 bytes from 10.0.1.1: icmp_seq=1 ttl=64 time=7.78 ms
Izmjenjeno od Ernad Husremović prije više od 15 godina
http://www.drbd.org/users-guide/ch-ocfs2.html
Although originally designed for use with conventional shared storage devices, OCFS2 is equally well suited to be deployed on dual-Primary DRBD. Applications reading from the filesystem may benefit from reduced read latency due to the fact that DRBD reads from and writes to local storage, as opposed to the SAN devices OCFS2 otherwise normally runs on. In addition, DRBD adds redundancy to OCFS2 by adding an additional copy to every filesystem image, as opposed to just a single filesystem image that is merely shared.
Izmjenjeno od Ernad Husremović prije više od 15 godina
- Naslov promijenjeno iz eucalyptus lucid, aoe storage, drbd u eucalyptus lucid, aoe storage, drbd, ocsfs2
Like other shared cluster file systems such as GFS, OCFS2 allows multiple nodes to access the same storage device, in read/write mode, simultaneously without risking data corruption. It does so by using a Distributed Lock Manager (DLM) which manages concurrent access from cluster nodes. The DLM itself uses a virtual file system (ocfs2_dlmfs) which is separate from the actual OCFS2 file systems present on the system.
Izmjenjeno od Ernad Husremović prije više od 15 godina
root@zvijer-2:~# sudo apt-get install ocfs2-tools
Izmjenjeno od Ernad Husremović prije više od 15 godina
root@zvijer-2:/etc/drbd.d# cat drbd0.res
resource r0 {
device /dev/drbd1;
disk /dev/sda3;
meta-disk internal;
startup {
become-primary-on both;
}
net {
# allow-two-primaries;
after-sb-0pri discard-zero-changes;
after-sb-1pri discard-secondary;
after-sb-2pri disconnect;
}
on zvijer-2 {
address 10.0.1.1:7789;
}
on zvijer-3 {
address 10.0.1.2:7789;
}
}
Izmjenjeno od Ernad Husremović prije više od 15 godina
root@zvijer-2:/etc/drbd.d# drbdadm create-md r0
--== Thank you for participating in the global usage survey ==-- The server's response is: you are the 2220th user to install this version drbd.d/drbd0.res:1: in resource r0: become-primary-on is set to both, but allow-two-primaries is not set.
u net sekciji enableovo ovo
root@zvijer-2:/etc/drbd.d# vi *res
root@zvijer-2:/etc/drbd.d# drbdadm create-md r0
Writing meta data... initializing activity log NOT initialized bitmap New drbd meta data block successfully created. success
Izmjenjeno od Ernad Husremović prije više od 15 godina
http://www.drbd.org/users-guide/s-first-time-up.html
root@zvijer-2:/etc/drbd.d# drbdadm attach r0
root@zvijer-2:/etc/drbd.d# drbdadm syncer r0
root@zvijer-2:/etc/drbd.d# drbdadm connect r0
root@zvijer-2:/etc/drbd.d# cat /proc/drbd
version: 8.3.7 (api:88/proto:86-91)
GIT-hash: ea9e28dbff98e331a62bcbcc63a6135808fe2917 build by root@zvijer-2, 2010-04-07 10:24:06
1: cs:WFConnection ro:Secondary/Unknown ds:Inconsistent/DUnknown C r----
ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:488172328
kaže
The Inconsistent/Inconsistent disk state is expected at this point.
By now, DRBD has successfully allocated both disk and network resources and is ready for operation. What it does not know yet is which of your nodes should be used as the source of the initial device synchronization.
Izmjenjeno od Ernad Husremović prije više od 15 godina
root@zvijer-2:/etc/drbd.d# scp drbd0.res root@192.168.45.168:/etc/drbd.d/
Izmjenjeno od Ernad Husremović prije više od 15 godina
prelazimo na zvijer-3:
root@zvijer-3:/etc/drbd.d# drbdadm create-md r0
--== Thank you for participating in the global usage survey ==-- The server's response is: you are the 2221th user to install this version Writing meta data... initializing activity log NOT initialized bitmap New drbd meta data block successfully created. success root@zvijer-3:/etc/drbd.d# drbdadm attach r0
root@zvijer-3:/etc/drbd.d# drbdadm syncer r0
root@zvijer-3:/etc/drbd.d# drbdadm connect r0
root@zvijer-3:/etc/drbd.d# cat /proc/drbd
version: 8.3.7 (api:88/proto:86-91)
GIT-hash: ea9e28dbff98e331a62bcbcc63a6135808fe2917 build by root@zvijer-3, 2010-04-07 10:24:44
1: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent C r----
ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:488172328
Izmjenjeno od Ernad Husremović prije više od 15 godina
vraćam se na zvijer-2
root@zvijer-2:/etc/drbd.d# drbdadm -- --overwrite-data-of-peer primary r0
root@zvijer-2:/etc/drbd.d# cat /proc/drbd
version: 8.3.7 (api:88/proto:86-91)
GIT-hash: ea9e28dbff98e331a62bcbcc63a6135808fe2917 build by root@zvijer-2, 2010-04-07 10:24:06
1: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r----
ns:7072 nr:0 dw:0 dr:7272 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:488165256
[>....................] sync'ed: 0.1% (476720/476728)M
finish: 169:30:06 speed: 608 (320) K/sec
Izmjenjeno od Ernad Husremović prije više od 15 godina
Select an initial sync source.¶
If you are dealing with newly-initialized, empty disk, this choice is entirely arbitrary. If one of your nodes already has valuable data that you need to preserve, however, it is of crucial importance that you select that node as your synchronization source. If you do initial device synchronization in the wrong direction, you will lose that data. Exercise caution.
Izmjenjeno od Ernad Husremović prije više od 15 godina
drbd.conf
on host-name
Carries the necessary configuration parameters for a DRBD device of
the enclosing resource. host-name is mandatory and must match the
Linux host name (uname -n) of one of the nodes. You may list more
than one host name here, in case you want to use the same
parameters on several hosts (you´d have to move the IP around
usually). Or you may list more than two such sections.
resource r1 {
protocol C;
device minor 1;
meta-disk internal;
on alice bob {
address 10.2.2.100:7801;
disk /dev/mapper/some-san;
}
on charlie {
address 10.2.2.101:7801;
disk /dev/mapper/other-san;
}
Izmjenjeno od Ernad Husremović prije više od 15 godina
međutim resync je očajno spor 320K/sec
http://www.drbd.org/users-guide/re-drbdconf.html
In this example, there is a single DRBD resource (called r0) which uses protocol C for the connection between its devices. The device which runs on host alice uses /dev/drbd1 as devices for its application, and /dev/sda7 as low-level storage for the data. The IP addresses are used to specify the networking interfaces to be used. An eventually running resync process should use about 10MByte/second of IO bandwidth.
stavio sync 10M;
root@zvijer-2:/etc/drbd.d# invoke-rc.d drbd restart
e sad imamo skroz drugu priču
root@zvijer-2:/etc/drbd.d# cat /proc/drbd
version: 8.3.7 (api:88/proto:86-91)
GIT-hash: ea9e28dbff98e331a62bcbcc63a6135808fe2917 build by root@zvijer-2, 2010-04-07 10:24:06
1: cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent C r----
ns:1113088 nr:0 dw:0 dr:1113088 al:0 bm:67 lo:0 pe:22 ua:0 ap:0 ep:1 wo:b oos:486109672
[>....................] sync'ed: 0.3% (474716/475800)M
finish: 12:19:53 speed: 10,944 (10,296) K/sec
Izmjenjeno od Ernad Husremović prije više od 15 godina
međutim ovo je i dalje sporo
1GBit/sec
ruby-1.9.1-p378 > 1000/8.00
=> 125.0
bi trebao dati 125 Mbajta/sec znači cca 100 Mb/sec. ali ni traga od toga
Izmjenjeno od Ernad Husremović prije više od 15 godina
stavim sync 100M ali nema nekog pomaka:
root@zvijer-2:/etc/drbd.d# cat /proc/drbd
version: 8.3.7 (api:88/proto:86-91)
GIT-hash: ea9e28dbff98e331a62bcbcc63a6135808fe2917 build by root@zvijer-2, 2010-04-07 10:24:06
1: cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent C r----
ns:238128 nr:0 dw:0 dr:242464 al:0 bm:14 lo:1 pe:121 ua:136 ap:0 ep:1 wo:b oos:483055624
[>....................] sync'ed: 0.1% (471732/471960)M
finish: 25:48:15 speed: 4,800 (11,712) K/sec
Izmjenjeno od Ernad Husremović prije više od 15 godina
prebacio na eth0 interface i vidi sad:
root@zvijer-2:/etc/drbd.d# cat /proc/drbd
version: 8.3.7 (api:88/proto:86-91)
GIT-hash: ea9e28dbff98e331a62bcbcc63a6135808fe2917 build by root@zvijer-2, 2010-04-07 10:24:06
1: cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent C r----
ns:583680 nr:0 dw:0 dr:583680 al:0 bm:35 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:473908008
[>....................] sync'ed: 0.2% (462800/463368)M
finish: 1:38:43 speed: 79,712 (53,060) K/sec
53 MB/sec
Izmjenjeno od Ernad Husremović prije više od 15 godina
pa fakat ovdje nešto nije uredu
root@zvijer-2:/etc/drbd.d# dmesg | grep eth1
[ 1.199503] sky2 eth1: addr 00:24:1d:ac:8f:ec [ 5.188610] udev: renamed network interface eth0 to eth1 [ 5.203874] udev: renamed network interface eth1_rename to eth0 [ 143.113453] eth1: link up, 100Mbps, full-duplex, lpa 0xC5E1 <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< [ 153.500016] eth1: no IPv6 routers present
root@zvijer-2:/etc/drbd.d# cat /etc/udev/rules.d/70-persistent-net.rules
# PCI device 0x11ab:0x4365 (sky2)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:24:1d:ac:8f:ec", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"
# PCI device 0x1186:0x1300 (8139too)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:1b:11:f3:cd:f1", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"
root@zvijer-3:/etc/drbd.d# cat /etc/udev/rules.d/70-persistent-net.rules
# PCI device 0x11ab:0x4365 (sky2)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:24:1d:ac:a5:a4", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"
# PCI device 0x1186:0x4300 (r8169)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:15:e9:da:69:37", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"
Izmjenjeno od Ernad Husremović prije više od 15 godina
hah jasko je uvalio 100 MBit karticu fakat u zvijer-2
root@zvijer-2:~# cat /etc/udev/rules.d/70-persistent-net.rules
# This file was automatically generated by the /lib/udev/write_net_rules
# program, run by the persistent-net-generator.rules rules file.
#
# You can modify it, as long as you keep each rule on a single
# line, and change only the value of the NAME= key.
# PCI device 0x11ab:0x4365 (sky2)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:24:1d:ac:8f:ec", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"
# PCI device 0x1148:0x4320 (skge)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:0c:46:46:86:c0", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"
interesantan je efekat .... njegova čitava aktivnost je trajala nekih po sahata. a ja sam isto toliko izgubio da skontam o čemu se radi
Izmjenjeno od Ernad Husremović prije više od 15 godina
evo ga sada i preko eth1 ide 1GB link
root@zvijer-2:~# cat /proc/drbd
version: 8.3.7 (api:88/proto:86-91)
GIT-hash: ea9e28dbff98e331a62bcbcc63a6135808fe2917 build by root@zvijer-2, 2010-04-07 10:24:06
1: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r----
ns:789268 nr:0 dw:0 dr:794280 al:0 bm:47 lo:1 pe:106 ua:151 ap:0 ep:1 wo:b oos:447081800
[>....................] sync'ed: 0.2% (436600/437368)M
finish: 2:17:18 speed: 54,200 (49,116) K/sec
Izmjenjeno od Ernad Husremović prije više od 15 godina
pokušavao sam međutim staviti 100M pa nakon toga 70M ali mi se brzina preko ovog linka ne miče od 50 M/sec
Izmjenjeno od Ernad Husremović prije više od 15 godina
dok preko eth0 imam puno bolji bandwidth
root@zvijer-2:/etc/drbd.d# cat /proc/drbd
version: 8.3.7 (api:88/proto:86-91)
GIT-hash: ea9e28dbff98e331a62bcbcc63a6135808fe2917 build by root@zvijer-2, 2010-04-07 10:24:06
1: cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent C r----
ns:1564512 nr:0 dw:0 dr:1566720 al:0 bm:94 lo:188 pe:87 ua:255 ap:0 ep:1 wo:b oos:431511208
[>....................] sync'ed: 0.4% (421396/422920)M
finish: 4:49:59 speed: 24,640 (78,084) K/sec
Izmjenjeno od Ernad Husremović prije više od 15 godina
međutim na 100M parametru konekcija je izgleda nestabilna ... skače do 93 pa pada do 10M ..
Izmjenjeno od Ernad Husremović prije više od 15 godina
ali ko je usko grlo na eth1 to sad treba utvrditi ...
Izmjenjeno od Ernad Husremović prije više od 15 godina
našao dvije alien tellesyn iste kartice (skge linux driver) - testirao i opet imam max 45 M/sec
Izmjenjeno od Ernad Husremović prije više od 15 godina
u p2 ibmu našao dlink istu kao ovu jednu što već sam stavljao pa ću telesyn zamijeniti sa dlinkom
Izmjenjeno od Ernad Husremović prije više od 15 godina
dlinkove su ok
root@zvijer-3:~# cat /proc/drbd
version: 8.3.7 (api:88/proto:86-91)
GIT-hash: ea9e28dbff98e331a62bcbcc63a6135808fe2917 build by root@zvijer-3, 2010-04-07 10:24:44
1: cs:SyncTarget ro:Primary/Primary ds:Inconsistent/UpToDate C r----
ns:0 nr:152580 dw:149792 dr:8 al:0 bm:9 lo:88 pe:4193 ua:87 ap:1 ep:1 wo:b oos:369680904
[>....................] sync'ed: 0.1% (361016/361160)M
finish: 1:22:09 speed: 74,896 (74,896) K/sec
Izmjenjeno od Ernad Husremović prije više od 15 godina
ali je opet znatno lošiji od eth0
root@zvijer-2:/etc/drbd.d# cat /proc/drbd
version: 8.3.7 (api:88/proto:86-91)
GIT-hash: ea9e28dbff98e331a62bcbcc63a6135808fe2917 build by root@zvijer-2, 2010-04-07 10:24:06
1: cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent C r----
ns:6292004 nr:0 dw:0 dr:6300160 al:0 bm:383 lo:1 pe:107 ua:256 ap:0 ep:1 wo:b oos:355972680
[>....................] sync'ed: 1.8% (347628/353768)M
finish: 1:26:03 speed: 68,864 (55,648) K/sec
Izmjenjeno od Ernad Husremović prije više od 15 godina
man drbdsetup
-R, --use-rle
During resync-handshake, the dirty-bitmaps of the nodes are
exchanged and merged (using bit-or), so the nodes will have the
same understanding of which blocks are dirty. On large devices, the
fine grained dirty-bitmap can become large as well, and the bitmap
exchange can take quite some time on low-bandwidth links.
Because the bitmap typically contains compact areas where all bits
are unset (clean) or set (dirty), a simple run-length encoding
scheme can considerably reduce the network traffic necessary for
the bitmap exchange.
For backward compatibilty reasons, and because on fast links this
possibly does not improve transfer time but consumes cpu cycles,
this defaults to off.
stavio ovaj parametar
root@zvijer-2:~# cat /proc/drbd
version: 8.3.7 (api:88/proto:86-91)
GIT-hash: ea9e28dbff98e331a62bcbcc63a6135808fe2917 build by root@zvijer-2, 2010-04-07 10:24:06
1: cs:SyncSource ro:Primary/Primary ds:UpToDate/Inconsistent C r----
ns:2494080 nr:0 dw:0 dr:2501696 al:0 bm:152 lo:2 pe:113 ua:234 ap:0 ep:1 wo:b oos:322775784
[>....................] sync'ed: 0.8% (315208/317640)M
finish: 1:38:16 speed: 54,728 (52,984) K/sec
Izmjenjeno od Ernad Husremović prije više od 15 godina
dokle je stigao
root@zvijer-2:~# cat /proc/drbd
version: 8.3.7 (api:88/proto:86-91)
GIT-hash: ea9e28dbff98e331a62bcbcc63a6135808fe2917 build by root@zvijer-2, 2010-04-07 10:24:06
1: cs:SyncSource ro:Primary/Primary ds:UpToDate/Inconsistent C r----
ns:156834436 nr:0 dw:0 dr:156840704 al:0 bm:9572 lo:67 pe:171 ua:256 ap:0 ep:1 wo:b oos:168437288
[========>...........] sync'ed: 48.3% (164488/317640)M
finish: 0:47:20 speed: 59,252 (54,928) K/sec
Izmjenjeno od Ernad Husremović prije više od 15 godina
mogu li u toku sinhroniziacije raditi sa drbd1 block device-om ?
root@zvijer-2:~# mkfs -t ext4 /dev/drbd1
mke2fs 1.41.11 (14-Mar-2010)
Oznaka sistema datoteka=
Tip operativnog sistema: Linux
Veličina bloka=4096 (log=2)
Veličina fragmenta=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
30515200 i čvorova, 122043082 blokova
6102154 blokova (5.00%) rezervisano za super korisnika
Prvi blok podatka=0
Maksimalan broj blokova sistema datoteka=4294967296
3725 grupe blokova
32768 blokova po grupi, 32768 fragmenata po grupi
8192 i čvorova po grupi
Rezervne kopije superbloka koje su sačuvane na blokovima:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
102400000
Pišem tabele i čvorova: urađeno
Creating journal (32768 blocks): urađeno
Informacije o upisu supperblokova i očitavanja sistema datoteka:
mogu ... istina ovo je trajalo 5-6 minuta
Izmjenjeno od Ernad Husremović prije više od 15 godina
odoh sada na drugi host i mountam ovaj uređaj
root@zvijer-3:~# mount /dev/drbd1 /mnt
ok
Izmjenjeno od Ernad Husremović prije više od 15 godina
mountao na zvijer-2 i tamo dodao u txt
pa se prebacujem ponovo na zvijer-3
root@zvijer-3:~# mount /dev/drbd1 /mnt
root@zvijer-3:~# cat /mnt/1.txt
hello drbd world mountam vamo mountam tamo ...
sporo je ali radi
Izmjenjeno od Ernad Husremović prije više od 15 godina
ocfs2 - dpkg-reconfigure ocfs2-tools¶
root@zvijer-2:~# cat /etc/ocfs2/cluster.conf
node:
ip_port = 7777
ip_address = 10.0.1.1
number = 0
name = zvijer-1
cluster = ocfs2
node:
ip_port = 7777
ip_address = 10.0.1.2
number = 1
name = zvijer-2
cluster = ocfs2
cluster:
node_count = 2
name = ocfs2
The O2CB heartbeat threshold sets up the maximum time in seconds that a │ │ node awaits for an I/O operation. After it, the node "fences" itself, │ │ and you will probably see a crash. │ │ │ │ It is calculated as the result of: (threshold - 1) x 2. │ │ │ │ Its default value is 31 (60 seconds). │ │ │ │ Raise it if you have slow disks and/or crashes with kernel messages │ │ like: │ │ │ │ o2hb_write_timeout: 164 ERROR: heartbeat write timeout to device XXXX │ │ after NNNN milliseconds
oc2b heartbeat treshlold: 31
idle timeout: 30 000 (30 seconds)
keeapalive delay: 2000
reconnect delay: 2000
dpkg-reconfigure ocfs2-tools
root@zvijer-2:~# dpkg-reconfigure ocfs2-tools update-rc.d: warning: o2cb start runlevel arguments (S) do not match LSB Default-Start values (2 3 5) update-rc.d: warning: o2cb stop runlevel arguments (0 6) do not match LSB Default-Stop values (none) Cluster ocfs2 already online update-rc.d: warning: ocfs2 start runlevel arguments (S) do not match LSB Default-Start values (2 3 5) update-rc.d: warning: ocfs2 stop runlevel arguments (0 6) do not match LSB Default-Stop values (none)
Izmjenjeno od Ernad Husremović prije više od 15 godina
root@zvijer-3:~# dpkg-reconfigure ocfs2-tools
update-rc.d: warning: o2cb start runlevel arguments (S) do not match LSB Default-Start values (2 3 5) update-rc.d: warning: o2cb stop runlevel arguments (0 6) do not match LSB Default-Stop values (none) Loading filesystem "configfs": OK Mounting configfs filesystem at /sys/kernel/config: OK Loading stack plugin "o2cb": OK Loading filesystem "ocfs2_dlmfs": OK Creating directory '/dlm': OK Mounting ocfs2_dlmfs filesystem at /dlm: OK Setting cluster stack "o2cb": OK Starting O2CB cluster ocfs2: OK update-rc.d: warning: ocfs2 start runlevel arguments (S) do not match LSB Default-Start values (2 3 5) update-rc.d: warning: ocfs2 stop runlevel arguments (0 6) do not match LSB Default-Stop values (none)
Izmjenjeno od Ernad Husremović prije više od 15 godina
root@zvijer-2:~# /etc/init.d/o2cb status
Driver for "configfs": Loaded Filesystem "configfs": Mounted Stack glue driver: Loaded Stack plugin "o2cb": Loaded Driver for "ocfs2_dlmfs": Loaded Filesystem "ocfs2_dlmfs": Mounted Checking O2CB cluster ocfs2: Online Heartbeat dead threshold = 31 Network idle timeout: 30000 Network keepalive delay: 2000 Network reconnect delay: 2000 Checking O2CB heartbeat: Not active
root@zvijer-3:~# /etc/init.d/o2cb status
Driver for "configfs": Loaded Filesystem "configfs": Mounted Stack glue driver: Loaded Stack plugin "o2cb": Loaded Driver for "ocfs2_dlmfs": Loaded Filesystem "ocfs2_dlmfs": Mounted Checking O2CB cluster ocfs2: Online Heartbeat dead threshold = 31 Network idle timeout: 30000 Network keepalive delay: 2000 Network reconnect delay: 2000 Checking O2CB heartbeat: Not active
Izmjenjeno od Ernad Husremović prije više od 15 godina
root@zvijer-2:~# mkfs -t ocfs2 -N 2 -L ocfs2_drbd1 /dev/drbd1
mkfs.ocfs2 1.4.3 Cluster stack: classic o2cb Filesystem label=ocfs2_drbd1 Block size=4096 (bits=12) Cluster size=4096 (bits=12) Volume size=499888463872 (122043082 clusters) (122043082 blocks) 3784 cluster groups (tail covers 18634 clusters, rest cover 32256 clusters) Journal size=268435456 Initial number of node slots: 2 Creating bitmaps: done Initializing superblock: done Writing system files: done Writing superblock: done Writing backup superblock: 5 block(s) ...
Izmjenjeno od Ernad Husremović prije više od 15 godina
root@zvijer-2:~# time (mkfs -t ocfs2 -N 2 -L ocfs2_drbd1 /dev/drbd1)
mkfs.ocfs2 1.4.3 Cluster stack: classic o2cb Overwriting existing ocfs2 partition. Proceed (y/N): y Filesystem label=ocfs2_drbd1 Block size=4096 (bits=12) Cluster size=4096 (bits=12) Volume size=499888463872 (122043082 clusters) (122043082 blocks) 3784 cluster groups (tail covers 18634 clusters, rest cover 32256 clusters) Journal size=268435456 Initial number of node slots: 2 Creating bitmaps: done Initializing superblock: done Writing system files: done Writing superblock: done Writing backup superblock: 5 block(s) Formatting Journals: done Formatting slot map: done Writing lost+found: done mkfs.ocfs2 successful real 4m5.821s user 0m0.130s sys 0m0.330s
Izmjenjeno od Ernad Husremović prije više od 15 godina
root@zvijer-2:~# mount -t ocfs2 /dev/drbd1 /shared
root@zvijer-3:~# mount -t ocfs2 /dev/drbd1 /shared
Izmjenjeno od Ernad Husremović prije više od 15 godina
Izmjenjeno od Ernad Husremović prije više od 15 godina
zvijer-3: cp shared -> local disk¶
root@zvijer-3:/shared# time (cp 1b.txt /root/test/1c.txt) real 2m30.151s user 0m0.170s sys 0m23.510s
Izmjenjeno od Ernad Husremović prije više od 15 godina
local disk cp¶
krajnje interesantno !
root@zvijer-3:~/test# time (cp 1c.txt 1d.txt) real 2m56.505s user 0m0.130s sys 0m21.770s
lošije ispada nego shared -> local ? da li je lokalni disk spor ili .. ?
Izmjenjeno od Ernad Husremović prije više od 15 godina
sad sam se sjetio ... ja sam na zvijer-2 pustio shared -> shared copy a to se opet dešava na drbd particiji koja stalno se sinhronizira između zvijer-2 i zvijer-3 ...
tako da je to moralo usporiti i ovaj lokalni cp test
Izmjenjeno od Ernad Husremović prije više od 15 godina
upravo tako, na zvijer-2 imamo puno lošiji rezultat
root@zvijer-2:/shared# time (cp 1b.txt 1c.txt)
real 6m11.968s user 0m0.070s sys 0m42.730s
ponavljam test ali ovaj put ne radim ništa na zvijer-3
Izmjenjeno od Ernad Husremović prije više od 15 godina
root@zvijer-2:/shared# time (cp 1b.txt 1c.txt)
real 5m4.599s user 0m0.200s sys 0m41.900s
Izmjenjeno od Ernad Husremović prije više od 15 godina
hm i nema nekih značajnih promjena u brzini:
root@zvijer-3:~/test# time (cp 1c.txt 1d.txt)
real 2m25.883s user 0m0.150s sys 0m26.890s
Izmjenjeno od Ernad Husremović prije više od 15 godina
zezno sam što nisamo ovo na poseban ticket stavio
Izmjenjeno od Ernad Husremović prije više od 15 godina
zvijer-2 shared -> local¶
root@zvijer-2:/shared# time (cp 1b.txt /root/test/1c.txt)
real 2m37.951s user 0m0.090s sys 0m23.320s
nema bitne razlike u odnosu na zvijer-3
Izmjenjeno od Ernad Husremović prije više od 15 godina
zvjer-2 local¶
root@zvijer-2:~/test# time (cp 1c.txt 1d.txt)
real 2m22.664s user 0m0.130s sys 0m20.480s
beznačajna razlika u 3 sekunde
Izmjenjeno od Ernad Husremović prije više od 15 godina
drbd status¶
root@zvijer-2:~/test# cat /proc/drbd
version: 8.3.7 (api:88/proto:86-91)
GIT-hash: ea9e28dbff98e331a62bcbcc63a6135808fe2917 build by root@zvijer-2, 2010-04-07 10:24:06
1: cs:Connected ro:Primary/Primary ds:UpToDate/UpToDate C r----
ns:362962809 nr:7886218 dw:48067491 dr:353799266 al:18624 bm:19853 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0
Izmjenjeno od Ernad Husremović prije više od 15 godina
zvijer-2 je down¶
root@zvijer-3:/shared# cat /proc/drbd
version: 8.3.7 (api:88/proto:86-91)
GIT-hash: ea9e28dbff98e331a62bcbcc63a6135808fe2917 build by root@zvijer-3, 2010-04-07 10:24:44
1: cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown r----
ns:7886255 nr:362962846 dw:370849552 dr:16033274 al:2120 bm:19855 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:176
na ocfs2 share-u normalno radim
Izmjenjeno od Ernad Husremović prije više od 15 godina
i sve što je trebalo uraditi je na ispravnom node-u reći
root@zvijer-3:/shared# drbdadm connect r0
Izmjenjeno od Ernad Husremović prije više od 15 godina
odmah nakon toga
root@zvijer-2:~# cat /proc/drbd
version: 8.3.7 (api:88/proto:86-91)
GIT-hash: ea9e28dbff98e331a62bcbcc63a6135808fe2917 build by root@zvijer-2, 2010-04-07 10:24:06
1: cs:Connected ro:Primary/Primary ds:UpToDate/UpToDate C r----
ns:0 nr:320 dw:320 dr:200 al:0 bm:7 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0
Izmjenjeno od Ernad Husremović prije oko 15 godina
- Prioritet promijenjeno iz Normalan u Visok
Izmjenjeno od Ernad Husremović prije oko 15 godina
Izmjenjeno od Ernad Husremović prije više od 14 godina
- Naslov promijenjeno iz eucalyptus lucid, aoe storage, drbd, ocsfs2 u zvijer-2, zvijer-3: drbd, ocsfs2
- Status promijenjeno iz Dodijeljeno u Zatvoreno