system toolbox

eucalyptus-cloud

tcp        0      0 192.168.122.1:53        0.0.0.0:*               LISTEN      5925/dnsmasq

root@smraka-1b:~# apt-get install eucalyptus-sc eucalyptus-nc

root@smraka-1b:~# sudo apt-get install eucalyptus-walrus

root@smraka-1b:~# apt-get install eucalyptus-cc

Čitam spiskove paketa... Done
Gradim stablo zavisnosti       
Reading state information... Done
Slijedeći dodatni paketi će biti instalirani:
  dhcp3-server powerwake
Predloženi paketi:
  dhcp3-server-ldap apparmor vtun
Slijedeći NOVI paketi će biti instalirani:
  dhcp3-server eucalyptus-cc powerwake
0 upgraded, 3 newly installed, 0 to remove and 0 not upgraded.
Need to get 921kB of archives.
After this operation, 3453kB of additional disk space will be used.

root@smraka-1b:~# apt-get install eucalyptus-cc

root@smraka-1b:/etc/init.d# start eucalyptus

https://192.168.45.41:8443/#login

admin / admin

bringout@nvostro-hernad:~$ set | grep EC2

EC2_ACCESS_KEY=csIphK5u5WkSO4dnq8lRLPvxbVG5QaXIHj7ZDA
EC2_SECRET_KEY=6zntOVKOMTviGhB8JMo7GUXGHdo1DrzcHroNWA
EC2_URL=http://192.168.45.41:8773/services/Eucalyptus

bringout@nvostro-hernad:~$ euca-describe-regions

REGION    Eucalyptus    http://192.168.45.49:8773/services/Eucalyptus
REGION    Walrus    http://192.168.45.41:8773/services/Walrus

bringout@nvostro-hernad:~$ euca-describe-availability-zones

AVAILABILITYZONE    cluster1    192.168.45.41

aoe volumen¶

zvijer 1:

root@zvijer-1:~# vbladed 11 1 br0 /dev/sda3

root@smraka-1b:/etc/init.d# aoe-stat

     e11.1       706.153GB    br0 up 

root@smraka-1b:/etc/init.d# pvcreate /dev/etherd/e11.1

  Physical volume "/dev/etherd/e11.1" successfully created

http://virtually-a-machine.blogspot.com/search/label/eucalyptus

root@smraka-1b:/etc/init.d# euca_conf --register-nodes 192.168.45.42

INFO: We expect all nodes to have eucalyptus installed in //var/lib/eucalyptus/keys for key synchronization.
Warning: cannot file file node-cert.pem in //var/lib/eucalyptus/keys/
Warning: cannot file file cluster-cert.pem in //var/lib/eucalyptus/keys/
Warning: cannot file file node-pk.pem in //var/lib/eucalyptus/keys/

Trying rsync to sync keys with "192.168.45.42"...rsync: Failed to exec -l: No such file or directory (2)
rsync error: error in IPC code (code 14) at pipe.c(84) [sender=3.0.7]
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in IPC code (code 14) at io.c(601) [sender=3.0.7]
failed.

Trying scp to sync keys to: eucalyptus@192.168.45.42://var/lib/eucalyptus/keys/...
ssh: connect to host 192.168.45.42 port 22: No route to host
lost connection
failed.

ERROR: could not synchronize keys with 192.168.45.42!
The configuration will not have this node.
Hint: to setup passwordless login to the nodes as user eucalyptus, you can
run the following commands on node 192.168.45.42:
sudo -u eucalyptus mkdir -p ~eucalyptus/.ssh
sudo -u eucalyptus tee ~eucalyptus/.ssh/authorized_keys > /dev/null <<EOT
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAufTSMBUtu/DoBQq0IY/nVbtBxDuhcVirXUAbeKgEjM0nTocqKc5R25KOogSon60PVWADkf4CZ7uLDqjASXQxg+7cclWbPpwlvxXA46h7KeuIIfNFyQE+zp3cPUsqaGYc7IfFiTcAIqkkdmYBU7HASJj7ilmf4OXig7vCM2eEuso72eIJEfSC4b3ub5JgpH2p//4yLSm9etmpc8etDvsr2zYhAyUwjYgX95QC/buo+Z0TRUWcPYFUdfe7V4ZWAcfQusRlR53ZkXws4lav7N+qJDpjSxUtpXDHxw1/xkGbqwAV7jk1z3WpPyfewETcUriN7h8vmW5WdkfsRX4qCjajqQ== eucalyptus@smraka-1b.bring.out.ba
EOT

Be sure that authorized_keys is not group/world readable or writable

root@smraka-1b:/etc/init.d# euca_conf --discover-nodes

haystack=  192.168.45.41 192.168.122.1 
haystack=  192.168.45.41 192.168.122.1 
New node found on 192.168.45.39; add it? [Yn] 

troubleshooting¶

http://open.eucalyptus.com/wiki/EucalyptusTroubleshooting_v1.6

Block storage troubleshooting

• Unable to attach volumes when the front end and the NC are running on the same machine. This is a known issue with ATA over Ethernet (AoE). AoE will not export to the same machine that the server is running on. The workaround is to run the front end and the node controller on different hosts.
• Volume ends up in "deleted" state when created, instead of showing up as "available." Look for error messages in $EUCALYPTUS/var/log/eucalyptus/cloud-error.log. A common problem is that ATA-over-Ethernet may not be able to export the created volume (this will appear as a "Could not export..." message in cloud-error.log). Make sure that "VNET_INTERFACE" in eucalyptus.conf on the front end is correct.
• Failure to create volume/snapshot. Make sure you have enough loopback devices. If you are installing from packages, you will get a warning. On most distributions, the loopback driver is installed as a module. The following will increase the number of loopback devices available,
  

        rmmod loop ; modprobe loop max_loop=256
  

• If block devices do not automatically appear in your VMs, make sure that you have the "udev" package installed.
• If you are running gentoo and you get "which: no vblade in ((null)).", try compiling "su" without pam.

https://help.ubuntu.com/community/UEC/PackageInstall

https://help.ubuntu.com/community/UEC/Topologies

http://uec-images.ubuntu.com/lucid/current/

http://kiranmurari.wordpress.com/2010/03/19/22/

http://code.google.com/p/hybridfox/

https://wiki.ubuntu.com/UECTesting

http://sysadvent.blogspot.com/2009/12/storage-availability-drbd.html

http://www.drbd.org/download/drbd-mc/

root@smraka-1b:~# pvresize --setphysicalvolumesize 200G /dev/sda5

  Physical volume "/dev/sda5" changed
  1 physical volume(s) resized / 0 physical volume(s) not resized

ali nemoguće je napraviti resize particije ... nije ni bitno

http://probablyinteractive.com/2009/3/29/Amazon%20EC2%20+%20Chef%20=%20Mmmmm.html

http://www.opscode.com/blog/2010/03/24/amazon-ec2-amis-for-ubuntu-9-10-and-chef-0-8-8/

http://cookbooks.opscode.com/cookbooks/bootstrap

http://wiki.opscode.com/display/chef/Amazon+EC2+AMIs+and+Chef+0.8.x

zvijer-2¶

pxe install - lucid-server zvijer-2

vg0 - 250 GB LVM particija u njoj swap=24G, root ostatak

eth0 - public 45.105

eth1 - 10.0.1.1 privatna IP

zvijer-3¶

public ip .45.168

drbd¶

root@zvijer-2:~# apt-get install drbd8-utils

sda3 za drbd

root@zvijer-3:~# fdisk /dev/sda

WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
         switch off the mode (command 'c') and change display units to
         sectors (command 'u').

Naredba (m za pomoć): p

Disk /dev/sda: 750.2 GB, 750156374016 bajta
255 glava, 63 sektor/staza, 91201 cilindara
Jedinice = cilindara od 16065 * 512 = 8225280 bajta
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Identifikator diska: 0x00006a03

  Uređaj But      Početak         Kraj      Blokova   Id  Sistem
/dev/sda1   *           1          31      242688   83  Linux
Particija 1 se ne završava na granici cilindara.
/dev/sda2              31       30425   244140033    5  Prošireno
/dev/sda5              31       30425   244140032   8e  Linux LVM

Naredba (m za pomoć): n
Komandna akcija
   l   logička (5 ili više)
   p   primarna particija (1-4)
p
Broj particije (1-4): 3
Prvi cilindar (30425-91201, podrazumevano 30425): 
Koristim podrazumijevane vrijednosti 30425
Last cilindar, +cilindara or +size{K,M,G} (30425-91201, podrazumevano 91201): 
Koristim podrazumijevane vrijednosti 91201

Naredba (m za pomoć): w
Tabela particija je zamijenjena!

Pozivam ioctl() da bih ponovo pročito tabelu particija

WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table. The new table will be used at
the next reboot or after you run partprobe(8) or kpartx(8)
Sinhronizujem diskove.
root@zvijer-3:~# partprobe

root@zvijer-3:/etc/drbd.d# cat /etc/network/interfaces

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

auto eth1

# The primary network interface
auto eth0
iface eth0 inet dhcp

iface eth1 inet static
        address 10.0.1.2 <<<<<<<<<<<<<<<<<<<<<<<<<<<<
        netmask 255.255.255.0

radi privatni interface:

root@zvijer-3:/etc/drbd.d# ping 10.0.1.1

PING 10.0.1.1 (10.0.1.1) 56(84) bytes of data.
64 bytes from 10.0.1.1: icmp_seq=1 ttl=64 time=7.78 ms

http://www.drbd.org/users-guide/ch-ocfs2.html

Although originally designed for use with conventional shared storage devices, OCFS2 is equally well suited to be deployed on dual-Primary DRBD. Applications reading from the filesystem may benefit from reduced read latency due to the fact that DRBD reads from and writes to local storage, as opposed to the SAN devices OCFS2 otherwise normally runs on. In addition, DRBD adds redundancy to OCFS2 by adding an additional copy to every filesystem image, as opposed to just a single filesystem image that is merely shared.

root@zvijer-2:~# sudo apt-get install ocfs2-tools

root@zvijer-2:/etc/drbd.d# cat drbd0.res

resource r0 {
  device /dev/drbd1;
  disk   /dev/sda3;
  meta-disk internal;

  startup {
    become-primary-on both;
  }
  net {
    # allow-two-primaries;
    after-sb-0pri discard-zero-changes;
    after-sb-1pri discard-secondary;
    after-sb-2pri disconnect;
  }

  on zvijer-2 {
     address 10.0.1.1:7789;
  }

  on zvijer-3 {
     address 10.0.1.2:7789;
  }
}

root@zvijer-2:/etc/drbd.d# drbdadm create-md r0

 --==  Thank you for participating in the global usage survey  ==--
The server's response is:

you are the 2220th user to install this version
drbd.d/drbd0.res:1: in resource r0:
become-primary-on is set to both, but allow-two-primaries is not set.

u net sekciji enableovo ovo

root@zvijer-2:/etc/drbd.d# vi *res

root@zvijer-2:/etc/drbd.d# drbdadm create-md r0

Writing meta data...
initializing activity log
NOT initialized bitmap
New drbd meta data block successfully created.
success

http://www.drbd.org/users-guide/s-first-time-up.html

root@zvijer-2:/etc/drbd.d# drbdadm attach r0
root@zvijer-2:/etc/drbd.d# drbdadm syncer r0
root@zvijer-2:/etc/drbd.d# drbdadm connect r0
root@zvijer-2:/etc/drbd.d# cat /proc/drbd

version: 8.3.7 (api:88/proto:86-91)
GIT-hash: ea9e28dbff98e331a62bcbcc63a6135808fe2917 build by root@zvijer-2, 2010-04-07 10:24:06

 1: cs:WFConnection ro:Secondary/Unknown ds:Inconsistent/DUnknown C r----
    ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:488172328

kaže

The Inconsistent/Inconsistent disk state is expected at this point.

By now, DRBD has successfully allocated both disk and network resources and is ready for operation. What it does not know yet is which of your nodes should be used as the source of the initial device synchronization.

root@zvijer-2:/etc/drbd.d# scp drbd0.res root@192.168.45.168:/etc/drbd.d/

prelazimo na zvijer-3:

root@zvijer-3:/etc/drbd.d# drbdadm create-md r0

  --==  Thank you for participating in the global usage survey  ==--
The server's response is:

you are the 2221th user to install this version
Writing meta data...
initializing activity log
NOT initialized bitmap
New drbd meta data block successfully created.
success
root@zvijer-3:/etc/drbd.d# drbdadm attach r0

root@zvijer-3:/etc/drbd.d# drbdadm syncer r0
root@zvijer-3:/etc/drbd.d# drbdadm connect r0

root@zvijer-3:/etc/drbd.d# cat /proc/drbd

version: 8.3.7 (api:88/proto:86-91)
GIT-hash: ea9e28dbff98e331a62bcbcc63a6135808fe2917 build by root@zvijer-3, 2010-04-07 10:24:44

 1: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent C r----
    ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:488172328

vraćam se na zvijer-2

root@zvijer-2:/etc/drbd.d# drbdadm -- --overwrite-data-of-peer primary r0

root@zvijer-2:/etc/drbd.d# cat /proc/drbd

version: 8.3.7 (api:88/proto:86-91)
GIT-hash: ea9e28dbff98e331a62bcbcc63a6135808fe2917 build by root@zvijer-2, 2010-04-07 10:24:06

 1: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r----
    ns:7072 nr:0 dw:0 dr:7272 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:488165256
    [>....................] sync'ed:  0.1% (476720/476728)M
    finish: 169:30:06 speed: 608 (320) K/sec

Select an initial sync source.¶

If you are dealing with newly-initialized, empty disk, this choice is entirely arbitrary. If one of your nodes already has valuable data that you need to preserve, however, it is of crucial importance that you select that node as your synchronization source. If you do initial device synchronization in the wrong direction, you will lose that data. Exercise caution.

drbd.conf

   on host-name
           Carries the necessary configuration parameters for a DRBD device of
           the enclosing resource.  host-name is mandatory and must match the
           Linux host name (uname -n) of one of the nodes. You may list more
           than one host name here, in case you want to use the same
           parameters on several hosts (you´d have to move the IP around
           usually). Or you may list more than two such sections.

                    resource r1 {
                         protocol C;
                         device minor 1;
                         meta-disk internal;

                         on alice bob {
                              address 10.2.2.100:7801;
                              disk /dev/mapper/some-san;
                         }
                         on charlie {
                              address 10.2.2.101:7801;
                              disk /dev/mapper/other-san;
                         }

međutim resync je očajno spor 320K/sec

http://www.drbd.org/users-guide/re-drbdconf.html

In this example, there is a single DRBD resource (called r0) which uses protocol C for the connection between its devices. The device which runs on host alice uses /dev/drbd1 as devices for its application, and /dev/sda7 as low-level storage for the data. The IP addresses are used to specify the networking interfaces to be used. An eventually running resync process should use about 10MByte/second of IO bandwidth.

stavio sync 10M;

root@zvijer-2:/etc/drbd.d# invoke-rc.d drbd restart

e sad imamo skroz drugu priču

root@zvijer-2:/etc/drbd.d# cat /proc/drbd

version: 8.3.7 (api:88/proto:86-91)
GIT-hash: ea9e28dbff98e331a62bcbcc63a6135808fe2917 build by root@zvijer-2, 2010-04-07 10:24:06

 1: cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent C r----
    ns:1113088 nr:0 dw:0 dr:1113088 al:0 bm:67 lo:0 pe:22 ua:0 ap:0 ep:1 wo:b oos:486109672
    [>....................] sync'ed:  0.3% (474716/475800)M
    finish: 12:19:53 speed: 10,944 (10,296) K/sec

međutim ovo je i dalje sporo

1GBit/sec

ruby-1.9.1-p378 > 1000/8.00

 => 125.0

bi trebao dati 125 Mbajta/sec znači cca 100 Mb/sec. ali ni traga od toga

stavim sync 100M ali nema nekog pomaka:

root@zvijer-2:/etc/drbd.d# cat /proc/drbd

version: 8.3.7 (api:88/proto:86-91)
GIT-hash: ea9e28dbff98e331a62bcbcc63a6135808fe2917 build by root@zvijer-2, 2010-04-07 10:24:06

 1: cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent C r----
    ns:238128 nr:0 dw:0 dr:242464 al:0 bm:14 lo:1 pe:121 ua:136 ap:0 ep:1 wo:b oos:483055624
    [>....................] sync'ed:  0.1% (471732/471960)M
    finish: 25:48:15 speed: 4,800 (11,712) K/sec

prebacio na eth0 interface i vidi sad:

root@zvijer-2:/etc/drbd.d# cat /proc/drbd

version: 8.3.7 (api:88/proto:86-91)
GIT-hash: ea9e28dbff98e331a62bcbcc63a6135808fe2917 build by root@zvijer-2, 2010-04-07 10:24:06

 1: cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent C r----
    ns:583680 nr:0 dw:0 dr:583680 al:0 bm:35 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:473908008
    [>....................] sync'ed:  0.2% (462800/463368)M
    finish: 1:38:43 speed: 79,712 (53,060) K/sec

53 MB/sec

pa fakat ovdje nešto nije uredu

root@zvijer-2:/etc/drbd.d# dmesg | grep eth1

[    1.199503] sky2 eth1: addr 00:24:1d:ac:8f:ec
[    5.188610] udev: renamed network interface eth0 to eth1
[    5.203874] udev: renamed network interface eth1_rename to eth0
[  143.113453] eth1: link up, 100Mbps, full-duplex, lpa 0xC5E1 <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
[  153.500016] eth1: no IPv6 routers present

root@zvijer-2:/etc/drbd.d# cat /etc/udev/rules.d/70-persistent-net.rules

# PCI device 0x11ab:0x4365 (sky2)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:24:1d:ac:8f:ec", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0" 

# PCI device 0x1186:0x1300 (8139too)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:1b:11:f3:cd:f1", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1" 

root@zvijer-3:/etc/drbd.d# cat /etc/udev/rules.d/70-persistent-net.rules

# PCI device 0x11ab:0x4365 (sky2)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:24:1d:ac:a5:a4", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0" 

# PCI device 0x1186:0x4300 (r8169)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:15:e9:da:69:37", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1" 

hah jasko je uvalio 100 MBit karticu fakat u zvijer-2

root@zvijer-2:~# cat /etc/udev/rules.d/70-persistent-net.rules

# This file was automatically generated by the /lib/udev/write_net_rules
# program, run by the persistent-net-generator.rules rules file.
#
# You can modify it, as long as you keep each rule on a single
# line, and change only the value of the NAME= key.

# PCI device 0x11ab:0x4365 (sky2)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:24:1d:ac:8f:ec", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0" 

# PCI device 0x1148:0x4320 (skge)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:0c:46:46:86:c0", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1" 

interesantan je efekat .... njegova čitava aktivnost je trajala nekih po sahata. a ja sam isto toliko izgubio da skontam o čemu se radi

evo ga sada i preko eth1 ide 1GB link

root@zvijer-2:~# cat /proc/drbd

version: 8.3.7 (api:88/proto:86-91)
GIT-hash: ea9e28dbff98e331a62bcbcc63a6135808fe2917 build by root@zvijer-2, 2010-04-07 10:24:06

 1: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r----
    ns:789268 nr:0 dw:0 dr:794280 al:0 bm:47 lo:1 pe:106 ua:151 ap:0 ep:1 wo:b oos:447081800
    [>....................] sync'ed:  0.2% (436600/437368)M
    finish: 2:17:18 speed: 54,200 (49,116) K/sec

pokušavao sam međutim staviti 100M pa nakon toga 70M ali mi se brzina preko ovog linka ne miče od 50 M/sec

dok preko eth0 imam puno bolji bandwidth

root@zvijer-2:/etc/drbd.d# cat /proc/drbd

version: 8.3.7 (api:88/proto:86-91)
GIT-hash: ea9e28dbff98e331a62bcbcc63a6135808fe2917 build by root@zvijer-2, 2010-04-07 10:24:06

 1: cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent C r----
    ns:1564512 nr:0 dw:0 dr:1566720 al:0 bm:94 lo:188 pe:87 ua:255 ap:0 ep:1 wo:b oos:431511208
    [>....................] sync'ed:  0.4% (421396/422920)M
    finish: 4:49:59 speed: 24,640 (78,084) K/sec

međutim na 100M parametru konekcija je izgleda nestabilna ... skače do 93 pa pada do 10M ..

ali ko je usko grlo na eth1 to sad treba utvrditi ...

našao dvije alien tellesyn iste kartice (skge linux driver) - testirao i opet imam max 45 M/sec

u p2 ibmu našao dlink istu kao ovu jednu što već sam stavljao pa ću telesyn zamijeniti sa dlinkom

dlinkove su ok

root@zvijer-3:~# cat /proc/drbd

version: 8.3.7 (api:88/proto:86-91)
GIT-hash: ea9e28dbff98e331a62bcbcc63a6135808fe2917 build by root@zvijer-3, 2010-04-07 10:24:44

 1: cs:SyncTarget ro:Primary/Primary ds:Inconsistent/UpToDate C r----
    ns:0 nr:152580 dw:149792 dr:8 al:0 bm:9 lo:88 pe:4193 ua:87 ap:1 ep:1 wo:b oos:369680904
    [>....................] sync'ed:  0.1% (361016/361160)M
    finish: 1:22:09 speed: 74,896 (74,896) K/sec

ali je opet znatno lošiji od eth0

root@zvijer-2:/etc/drbd.d# cat /proc/drbd

version: 8.3.7 (api:88/proto:86-91)
GIT-hash: ea9e28dbff98e331a62bcbcc63a6135808fe2917 build by root@zvijer-2, 2010-04-07 10:24:06

 1: cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent C r----
    ns:6292004 nr:0 dw:0 dr:6300160 al:0 bm:383 lo:1 pe:107 ua:256 ap:0 ep:1 wo:b oos:355972680
    [>....................] sync'ed:  1.8% (347628/353768)M
    finish: 1:26:03 speed: 68,864 (55,648) K/sec

man drbdsetup

      -R, --use-rle
           During resync-handshake, the dirty-bitmaps of the nodes are
           exchanged and merged (using bit-or), so the nodes will have the
           same understanding of which blocks are dirty. On large devices, the
           fine grained dirty-bitmap can become large as well, and the bitmap
           exchange can take quite some time on low-bandwidth links.

           Because the bitmap typically contains compact areas where all bits
           are unset (clean) or set (dirty), a simple run-length encoding
           scheme can considerably reduce the network traffic necessary for
           the bitmap exchange.

           For backward compatibilty reasons, and because on fast links this
           possibly does not improve transfer time but consumes cpu cycles,
           this defaults to off.

stavio ovaj parametar

root@zvijer-2:~# cat /proc/drbd

version: 8.3.7 (api:88/proto:86-91)
GIT-hash: ea9e28dbff98e331a62bcbcc63a6135808fe2917 build by root@zvijer-2, 2010-04-07 10:24:06

 1: cs:SyncSource ro:Primary/Primary ds:UpToDate/Inconsistent C r----
    ns:2494080 nr:0 dw:0 dr:2501696 al:0 bm:152 lo:2 pe:113 ua:234 ap:0 ep:1 wo:b oos:322775784
    [>....................] sync'ed:  0.8% (315208/317640)M
    finish: 1:38:16 speed: 54,728 (52,984) K/sec

dokle je stigao

root@zvijer-2:~# cat /proc/drbd

version: 8.3.7 (api:88/proto:86-91)
GIT-hash: ea9e28dbff98e331a62bcbcc63a6135808fe2917 build by root@zvijer-2, 2010-04-07 10:24:06

 1: cs:SyncSource ro:Primary/Primary ds:UpToDate/Inconsistent C r----
    ns:156834436 nr:0 dw:0 dr:156840704 al:0 bm:9572 lo:67 pe:171 ua:256 ap:0 ep:1 wo:b oos:168437288
    [========>...........] sync'ed: 48.3% (164488/317640)M
    finish: 0:47:20 speed: 59,252 (54,928) K/sec

mogu li u toku sinhroniziacije raditi sa drbd1 block device-om ?

root@zvijer-2:~# mkfs -t ext4 /dev/drbd1

mke2fs 1.41.11 (14-Mar-2010)
Oznaka sistema datoteka=
Tip operativnog sistema: Linux
Veličina bloka=4096 (log=2)
Veličina fragmenta=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
30515200 i čvorova, 122043082 blokova
6102154 blokova (5.00%) rezervisano za super korisnika
Prvi blok podatka=0
Maksimalan broj blokova sistema datoteka=4294967296
3725 grupe blokova
32768 blokova po grupi, 32768 fragmenata po grupi
8192 i čvorova po grupi
Rezervne kopije superbloka koje su sačuvane na blokovima: 
    32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
    4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968, 
    102400000

Pišem tabele i čvorova: urađeno                            
Creating journal (32768 blocks): urađeno
Informacije o upisu supperblokova i očitavanja sistema datoteka:

mogu ... istina ovo je trajalo 5-6 minuta

odoh sada na drugi host i mountam ovaj uređaj

root@zvijer-3:~# mount /dev/drbd1 /mnt

ok

mountao na zvijer-2 i tamo dodao u txt

pa se prebacujem ponovo na zvijer-3

root@zvijer-3:~# mount /dev/drbd1 /mnt

root@zvijer-3:~# cat /mnt/1.txt

hello drbd world

mountam vamo mountam tamo ...

sporo je ali radi

ocfs2 - dpkg-reconfigure ocfs2-tools¶

root@zvijer-2:~# cat /etc/ocfs2/cluster.conf

node:
    ip_port = 7777
    ip_address = 10.0.1.1
    number = 0
    name = zvijer-1
    cluster = ocfs2

node:
    ip_port = 7777
    ip_address = 10.0.1.2
    number = 1
    name = zvijer-2
    cluster = ocfs2

cluster:
    node_count = 2
    name = ocfs2

   The O2CB heartbeat threshold sets up the maximum time in seconds that a   │ 
 │ node awaits for an I/O operation. After it, the node "fences" itself,     │ 
 │ and you will probably see a crash.                                        │ 
 │                                                                           │ 
 │ It is calculated as the result of: (threshold - 1) x 2.                   │ 
 │                                                                           │ 
 │ Its default value is 31 (60 seconds).                                     │ 
 │                                                                           │ 
 │ Raise it if you have slow disks and/or crashes with kernel messages       │ 
 │ like:                                                                     │ 
 │                                                                           │ 
 │ o2hb_write_timeout: 164 ERROR: heartbeat write timeout to device XXXX     │ 
 │ after NNNN milliseconds

oc2b heartbeat treshlold: 31

idle timeout: 30 000 (30 seconds)

keeapalive delay: 2000

reconnect delay: 2000

dpkg-reconfigure ocfs2-tools

root@zvijer-2:~# dpkg-reconfigure ocfs2-tools
update-rc.d: warning: o2cb start runlevel arguments (S) do not match LSB Default-Start values (2 3 5)
update-rc.d: warning: o2cb stop runlevel arguments (0 6) do not match LSB Default-Stop values (none)
Cluster ocfs2 already online
update-rc.d: warning: ocfs2 start runlevel arguments (S) do not match LSB Default-Start values (2 3 5)
update-rc.d: warning: ocfs2 stop runlevel arguments (0 6) do not match LSB Default-Stop values (none)

root@zvijer-3:~# dpkg-reconfigure ocfs2-tools

update-rc.d: warning: o2cb start runlevel arguments (S) do not match LSB Default-Start values (2 3 5)
update-rc.d: warning: o2cb stop runlevel arguments (0 6) do not match LSB Default-Stop values (none)
Loading filesystem "configfs": OK
Mounting configfs filesystem at /sys/kernel/config: OK
Loading stack plugin "o2cb": OK
Loading filesystem "ocfs2_dlmfs": OK
Creating directory '/dlm': OK
Mounting ocfs2_dlmfs filesystem at /dlm: OK
Setting cluster stack "o2cb": OK
Starting O2CB cluster ocfs2: OK
update-rc.d: warning: ocfs2 start runlevel arguments (S) do not match LSB Default-Start values (2 3 5)
update-rc.d: warning: ocfs2 stop runlevel arguments (0 6) do not match LSB Default-Stop values (none)

root@zvijer-2:~# /etc/init.d/o2cb status

Driver for "configfs": Loaded
Filesystem "configfs": Mounted
Stack glue driver: Loaded
Stack plugin "o2cb": Loaded
Driver for "ocfs2_dlmfs": Loaded
Filesystem "ocfs2_dlmfs": Mounted
Checking O2CB cluster ocfs2: Online
Heartbeat dead threshold = 31
  Network idle timeout: 30000
  Network keepalive delay: 2000
  Network reconnect delay: 2000
Checking O2CB heartbeat: Not active

root@zvijer-3:~# /etc/init.d/o2cb status

Driver for "configfs": Loaded
Filesystem "configfs": Mounted
Stack glue driver: Loaded
Stack plugin "o2cb": Loaded
Driver for "ocfs2_dlmfs": Loaded
Filesystem "ocfs2_dlmfs": Mounted
Checking O2CB cluster ocfs2: Online
Heartbeat dead threshold = 31
  Network idle timeout: 30000
  Network keepalive delay: 2000
  Network reconnect delay: 2000
Checking O2CB heartbeat: Not active

root@zvijer-2:~# mkfs -t ocfs2 -N 2 -L ocfs2_drbd1 /dev/drbd1

mkfs.ocfs2 1.4.3
Cluster stack: classic o2cb
Filesystem label=ocfs2_drbd1
Block size=4096 (bits=12)
Cluster size=4096 (bits=12)
Volume size=499888463872 (122043082 clusters) (122043082 blocks)
3784 cluster groups (tail covers 18634 clusters, rest cover 32256 clusters)
Journal size=268435456
Initial number of node slots: 2
Creating bitmaps: done
Initializing superblock: done
Writing system files: 

done
Writing superblock: done
Writing backup superblock: 5 block(s)
...

root@zvijer-2:~# time (mkfs -t ocfs2 -N 2 -L ocfs2_drbd1 /dev/drbd1)

mkfs.ocfs2 1.4.3
Cluster stack: classic o2cb
Overwriting existing ocfs2 partition.
Proceed (y/N): y
Filesystem label=ocfs2_drbd1
Block size=4096 (bits=12)
Cluster size=4096 (bits=12)
Volume size=499888463872 (122043082 clusters) (122043082 blocks)
3784 cluster groups (tail covers 18634 clusters, rest cover 32256 clusters)
Journal size=268435456
Initial number of node slots: 2
Creating bitmaps: done
Initializing superblock: done
Writing system files: done
Writing superblock: done
Writing backup superblock: 5 block(s)
Formatting Journals: done
Formatting slot map: done
Writing lost+found: done
mkfs.ocfs2 successful

real    4m5.821s
user    0m0.130s
sys    0m0.330s

root@zvijer-2:~# mount -t ocfs2 /dev/drbd1 /shared
root@zvijer-3:~# mount -t ocfs2 /dev/drbd1 /shared

testovi brzine ocfs2, local disk¶

root@zvijer-3:/shared# ls 1c.txt -l -h

-rw-r--r-- 1 root root 7,6G 2010-04-07 15:23 1c.txt

cp shared -> shared¶

root@zvijer-3:/shared# time (cp 1b.txt 1c.txt)

real    4m58.684s
user    0m0.100s
sys    0m41.860s

zvijer-3: cp shared -> local disk¶

root@zvijer-3:/shared# time (cp 1b.txt /root/test/1c.txt)

real    2m30.151s
user    0m0.170s
sys    0m23.510s

local disk cp¶

krajnje interesantno !

root@zvijer-3:~/test# time (cp 1c.txt 1d.txt)

real    2m56.505s
user    0m0.130s
sys    0m21.770s

lošije ispada nego shared -> local ? da li je lokalni disk spor ili .. ?

sad sam se sjetio ... ja sam na zvijer-2 pustio shared -> shared copy a to se opet dešava na drbd particiji koja stalno se sinhronizira između zvijer-2 i zvijer-3 ...

tako da je to moralo usporiti i ovaj lokalni cp test

upravo tako, na zvijer-2 imamo puno lošiji rezultat

root@zvijer-2:/shared# time (cp 1b.txt 1c.txt)

real    6m11.968s
user    0m0.070s
sys    0m42.730s

ponavljam test ali ovaj put ne radim ništa na zvijer-3

root@zvijer-2:/shared# time (cp 1b.txt 1c.txt)

real    5m4.599s
user    0m0.200s
sys    0m41.900s

hm i nema nekih značajnih promjena u brzini:

root@zvijer-3:~/test# time (cp 1c.txt 1d.txt)

real    2m25.883s
user    0m0.150s
sys    0m26.890s

zezno sam što nisamo ovo na poseban ticket stavio

zvijer-2 shared -> local¶

root@zvijer-2:/shared# time (cp 1b.txt /root/test/1c.txt)

real    2m37.951s
user    0m0.090s
sys    0m23.320s

nema bitne razlike u odnosu na zvijer-3

zvjer-2 local¶

root@zvijer-2:~/test# time (cp 1c.txt 1d.txt)

real    2m22.664s
user    0m0.130s
sys    0m20.480s

beznačajna razlika u 3 sekunde

drbd status¶

root@zvijer-2:~/test# cat /proc/drbd

version: 8.3.7 (api:88/proto:86-91)
GIT-hash: ea9e28dbff98e331a62bcbcc63a6135808fe2917 build by root@zvijer-2, 2010-04-07 10:24:06

 1: cs:Connected ro:Primary/Primary ds:UpToDate/UpToDate C r----
    ns:362962809 nr:7886218 dw:48067491 dr:353799266 al:18624 bm:19853 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0

zvijer-2 je down¶

root@zvijer-3:/shared# cat /proc/drbd

version: 8.3.7 (api:88/proto:86-91)
GIT-hash: ea9e28dbff98e331a62bcbcc63a6135808fe2917 build by root@zvijer-3, 2010-04-07 10:24:44

 1: cs:StandAlone ro:Primary/Unknown ds:UpToDate/DUnknown   r----
    ns:7886255 nr:362962846 dw:370849552 dr:16033274 al:2120 bm:19855 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:176

na ocfs2 share-u normalno radim

i sve što je trebalo uraditi je na ispravnom node-u reći

root@zvijer-3:/shared# drbdadm connect r0

odmah nakon toga

root@zvijer-2:~# cat /proc/drbd

version: 8.3.7 (api:88/proto:86-91)
GIT-hash: ea9e28dbff98e331a62bcbcc63a6135808fe2917 build by root@zvijer-2, 2010-04-07 10:24:06

 1: cs:Connected ro:Primary/Primary ds:UpToDate/UpToDate C r----
    ns:0 nr:320 dw:320 dr:200 al:0 bm:7 lo:0 pe:0 ua:0 ap:0 ep:1 wo:b oos:0

zvijer-2 ip=192.168.45.44¶

zvijer-3 ip=192.168.45.42¶