system toolbox

opet je ruby frcio

 3298 redmine   25   0  322m 194m 2684 R 16.7  3.3   0:54.62 ruby1.8                                                                                                    
 3305 redmine   25   0  417m 293m 2416 R 16.7  4.9   1:40.10 ruby1.8                                                                                                    
13688 redmine   25   0  506m 382m 2432 R 16.7  6.4   3:35.28 ruby1.8                                                                                                    
28139 redmine   25   0  352m 224m 2700 R 16.7  3.8   2:33.57 ruby1.8                                                                                                    
 7245 redmine   25   0  297m 170m 2684 R 16.3  2.9   0:36.99 ruby1.8                                                                                                    
 1596 redmine   25   0  438m 314m 2400 R 15.7  5.3   2:02.50 ruby1.8            

root@redmine-1:/# cat /proc/user_beancounters ...nema fail cnt-a

restart sesije vratio situaciju u normalu

evo kada je puklo

root@redmine-1:/home/redmine/redmine/log# tail --lines=2000 production.log | more

Processing IssueRelationsController#new (for 77.239.26.4 at 2011-10-12 08:48:59) [POST]
  Parameters: {"commit"=>"Dodaj", "issue_id"=>"24776", "action"=>"new", "authenticity_token"=>"ntMzYjP7wH5iYMZ8FgYnCiDbIUU4CpStZ9gTxuLIJT8=", "relation"=>{"delay"=>"", 
"issue_to_id"=>"24408", "relation_type"=>"relates"}, "id"=>"24776", "controller"=>"issue_relations"}

SignalException (SIGTERM):
  app/models/issue.rb:419:in `all_dependent_issues'
  app/models/issue.rb:421:in `all_dependent_issues'
  app/models/issue.rb:419:in `all_dependent_issues'
  ...
  app/models/issue.rb:421:in `all_dependent_issues'
  app/models/issue.rb:419:in `all_dependent_issues'
  app/models/issue.rb:421:in `all_dependent_issues'
  app/models/issue.rb:419:in `all_dependent_issues'
  app/models/issue.rb:421:in `all_dependent_issues'
  app/models/issue.rb:419:in `all_dependent_issues'
  app/models/issue_relation.rb:50:in `validate'
  app/controllers/issue_relations_controller.rb:27:in `new'
  passenger (3.0.0) lib/phusion_passenger/rack/request_handler.rb:96:in `process_request'
  passenger (3.0.0) lib/phusion_passenger/abstract_request_handler.rb:513:in `accept_and_process_next_request'
  passenger (3.0.0) lib/phusion_passenger/abstract_request_handler.rb:274:in `main_loop'
  passenger (3.0.0) lib/phusion_passenger/classic_rails/application_spawner.rb:321:in `start_request_handler'
  passenger (3.0.0) lib/phusion_passenger/classic_rails/application_spawner.rb:275:in `send'
  passenger (3.0.0) lib/phusion_passenger/classic_rails/application_spawner.rb:275:in `handle_spawn_application'
  passenger (3.0.0) lib/phusion_passenger/utils.rb:479:in `safe_fork'
  passenger (3.0.0) lib/phusion_passenger/classic_rails/application_spawner.rb:270:in `handle_spawn_application'
  passenger (3.0.0) lib/phusion_passenger/abstract_server.rb:357:in `__send__'
  passenger (3.0.0) lib/phusion_passenger/abstract_server.rb:357:in `server_main_loop'
  passenger (3.0.0) lib/phusion_passenger/abstract_server.rb:206:in `start_synchronously'
  passenger (3.0.0) lib/phusion_passenger/abstract_server.rb:180:in `start'
  passenger (3.0.0) lib/phusion_passenger/classic_rails/application_spawner.rb:149:in `start'
  passenger (3.0.0) lib/phusion_passenger/spawn_manager.rb:219:in `spawn_rails_application'
  passenger (3.0.0) lib/phusion_passenger/abstract_server_collection.rb:132:in `lookup_or_add'
  passenger (3.0.0) lib/phusion_passenger/spawn_manager.rb:214:in `spawn_rails_application'
  passenger (3.0.0) lib/phusion_passenger/abstract_server_collection.rb:82:in `synchronize'
  passenger (3.0.0) lib/phusion_passenger/abstract_server_collection.rb:79:in `synchronize'
  passenger (3.0.0) lib/phusion_passenger/spawn_manager.rb:213:in `spawn_rails_application'
  passenger (3.0.0) lib/phusion_passenger/spawn_manager.rb:132:in `spawn_application'
  passenger (3.0.0) lib/phusion_passenger/spawn_manager.rb:275:in `handle_spawn_application'
  passenger (3.0.0) lib/phusion_passenger/abstract_server.rb:357:in `__send__'
  passenger (3.0.0) lib/phusion_passenger/abstract_server.rb:357:in `server_main_loop'
  passenger (3.0.0) lib/phusion_passenger/abstract_server.rb:206:in `start_synchronously'
  passenger (3.0.0) helper-scripts/passenger-spawn-server:99

Rendering /home/redmine/redmine/public/500.html (500 Internal Server Error)

hehe fakat jest žeki urokala, da ovo nije neka nasty manifestacije MTU problema ??? jer ne vidim neki problem

dobro sad sam u frci, naknadno ću detaljno pregledati

otvori ticket na public redmine području. naravno izbaci detalje koji su specifični za našu infrrastrukturu

ovaj metod očigledn ima bug koji se na našim podacima manifestuje:

app/models/issue.rb:419:in `all_dependent_issues'

naravno, poveži se sa ovim ticketom

hm da nas nisu roboti napali pa izazivaju mini ddos napadi
vidim ove razne IP adrese

Processing BoardsController#show (for 92.240.57.182 at 2011-10-12 09:25:24) [GET]
  Parameters: {"project_id"=>"klijenti", "action"=>"show", "id"=>"2", "controller"=>"boards"}
Rendering template within layouts/base
Rendering boards/show
Completed in 1174ms (View: 121, DB: 1033) | 200 OK [http://redmine.bring.out.ba/projects/klijenti/boards/2]

Processing IssuesController#index (for 66.249.71.18 at 2011-10-12 09:25:29) [GET]
  Parameters: {"action"=>"index", "page"=>"5", "controller"=>"issues", "sort"=>"id:desc,tracker:desc,priority:desc"}
Rendering template within layouts/base
Rendering issues/index.rhtml
Completed in 2940ms (View: 92, DB: 2619) | 200 OK [http://redmine.bring.out.ba/issues?page=5&sort=id%3Adesc%2Ctracker%3Adesc%2Cpriority%3Adesc]

Processing MessagesController#show (for 92.240.57.182 at 2011-10-12 09:25:32) [GET]
  Parameters: {"board_id"=>"2", "action"=>"show", "id"=>"5643", "controller"=>"messages"}
Rendering template within layouts/base
Rendering messages/show
Completed in 193ms (View: 179, DB: 1) | 200 OK [http://redmine.bring.out.ba/boards/2/topics/5643]

Processing IssuesController#index (for 95.108.155.251 at 2011-10-12 09:25:40) [GET]
  Parameters: {"action"=>"index", "page"=>"2", "query_id"=>"27", "controller"=>"issues", "per_page"=>"50", "sort"=>"assigned_to,subject:desc,id:desc"}
Rendering template within layouts/base
Rendering issues/index.rhtml
Completed in 6186ms (View: 522, DB: 5625) | 200 OK [http://redmine.bring.out.ba/issues?page=2&per_page=50&query_id=27&sort=assigned_to%2Csubject%3Adesc%2Cid%3Adesc]

Processing IssuesController#show (for 92.240.57.182 at 2011-10-12 09:25:48) [GET]
  Parameters: {"action"=>"show", "id"=>"24777", "controller"=>"issues"}
Rendering template within layouts/base
Rendering common/403 (403)
Filter chain halted as [:authorize] rendered_or_redirected.

ili nas raja na public projektima posjećuje :)

npr ova zahtjev za openwrt-b-out-ba, ili neki roboti indexiraju ili neko gleda, moguće da si i oni automatski spammeri što poruke ostavljaju na forumima i stranicama

Processing WikiController#annotate (for 95.108.155.251 at 2011-10-12 09:33:59) [GET]
  Parameters: {"action"=>"annotate", "id"=>"openwrt-b-out-ba", "page"=>"Webif", "version"=>"4", "controller"=>"wiki"}
Rendering template within layouts/base
Rendering wiki/annotate
Completed in 82ms (View: 63, DB: 3) | 200 OK [http://redmine.bring.out.ba/projects/openwrt-b-out-ba/wiki/Webif/annotate/4]

ma da neko nas redom napada

Processing WikiController#annotate (for 95.108.155.251 at 2011-10-12 09:38:26) [GET]
  Parameters: {"action"=>"annotate", "id"=>"openwrt-b-out-ba", "page"=>"FwBuilder", "version"=>"8", "controller"=>"wiki"}
Rendering template within layouts/base
Rendering wiki/annotate
Completed in 54ms (View: 30, DB: 5) | 200 OK [http://redmine.bring.out.ba/projects/openwrt-b-out-ba/wiki/FwBuilder/annotate/8]

09:37:09 AM) semir: kod mene je redmine užasno spor
(09:38:05 AM) bjasko: kod svih je
(09:39:29 AM) bjasko: neko nas napada redom po redmine-u 
(09:39:40 AM) hernad entered the room.
(09:40:08 AM) hernad: odakle ? sa koje ip-a
(09:40:31 AM) hernad: stavi na nginx-u ograničenje broja zahtjeva
(09:40:52 AM) hernad: imaju ti limiti koji se za jednog korisnika mogu postaviit
(09:40:56 AM) bjasko: ma da moramo to nekakko po kontrolu staviti  
(09:41:32 AM) hernad: odakle dolaze ti napadi de odmah to uprati
(09:42:10 AM) bjasko: :( neda mi se , ubiše mi dan 
(09:42:35 AM) hernad: bjasko
(09:42:41 AM) hernad: anonimni posjetilac
(09:42:55 AM) bjasko: ja ...
(09:42:57 AM) hernad: daj da mu ograničimo privilegije
(09:43:09 AM) hernad: recimo vidim da gleda history wikija a to on ne treba
(09:43:24 AM) hernad: to recimo uzima procesorsko vrijeme dok se takvi requesti handliraju
(09:43:34 AM) hernad: bitno je samo da ima pregled sadržaja
(09:43:50 AM) bjasko: sad ću pogledati u redmine šta se može limitirati  kombinacija nginx+privilegije bi trebalo da rastereti server

za Anonymous korisnike ostavio samo view issues i wiki privilegije

evo ih traže li traže

rocessing IssuesController#index (for 95.108.155.251 at 2011-10-12 09:49:50) [GET]
  Parameters: {"action"=>"index", "page"=>"1", "query_id"=>"27", "controller"=>"issues", "per_page"=>"50", "sort"=>"priority,project:desc,subject:desc"}
Rendering template within layouts/base
Rendering issues/index.rhtml
Completed in 3486ms (View: 518, DB: 2926) | 200 OK [http://redmine.bring.out.ba/issues?page=1&per_page=50&query_id=27&sort=priority%2Cproject%3Adesc%2Csubject%3Adesc]

ovo su neki rusi, idem ja ovo na firewallu blokirati to definitivno nisu legitimni zahtjevi

source:         RIPE # Filtered
abuse-mailbox:  abuse@yandex.ru

% Information related to '95.108.128.0/17AS13238'

route:          95.108.128.0/17
descr:          Yandex enterprise network
origin:         AS13238
mnt-by:         YANDEX-MNT
source:         RIPE # Filtered

% Information related to '95.108.152.0/22AS13238'

route:          95.108.152.0/22
descr:          Yandex network
origin:         AS13238
mnt-by:         YANDEX-MNT
source:         RIPE # Filtered

% Information related to '95.108.154.0/23AS13238'

route:          95.108.154.0/23
descr:          Yandex network
origin:         AS13238
mnt-by:         YANDEX-MNT

ukinuo si search je li tako ? to je bitno ukinuti.

može li se search ukinuti skroz ? ne može ?

bitno je tu limit na proxy-ju staviti pa kad neko počne gomilati reqeuest-e da ga odbije

Ernad Husremović je napisao/la:

može li se search ukinuti skroz ? ne može ?

koliko vidim nema, tražim tražim

ima i ova IP 72.14.199.102

to je google njega ostavljamo

NetRange:       72.14.192.0 - 72.14.255.255
CIDR:           72.14.192.0/18
OriginAS:       
NetName:        GOOGLE
NetHandle:      NET-72-14-192-0-1
Parent:         NET-72-0-0-0-0
NetType:        Direct Allocation
RegDate:        2004-11-10
Updated:        2007-04-10
Ref:            http://whois.arin.net/rest/net/NET-72-14-192-0-1

google je i 66.249.71.18

google nas pravo često indexira

Processing IssuesController#index (for 66.249.71.18 at 2011-10-12 10:18:16) [GET]
Processing IssuesController#index (for 66.249.71.18 at 2011-10-12 10:20:51) [GET]
Processing FilesController#index (for 66.249.71.18 at 2011-10-12 10:20:58) [GET]
Processing AccountController#login (for 66.249.71.18 at 2011-10-12 10:21:00) [GET]
Processing IssuesController#index (for 66.249.71.18 at 2011-10-12 10:23:04) [GET]
Processing IssuesController#index (for 66.249.71.18 at 2011-10-12 10:23:24) [GET]
Processing IssuesController#index (for 66.249.71.18 at 2011-10-12 10:25:18) [GET]
Processing IssuesController#index (for 66.249.71.18 at 2011-10-12 10:25:59) [GET]
Processing WikiController#annotate (for 66.249.71.18 at 2011-10-12 10:26:12) [GET]
Processing AccountController#login (for 66.249.71.18 at 2011-10-12 10:26:14) [GET]
Processing IssuesController#index (for 66.249.71.18 at 2011-10-12 10:27:22) [GET]

imamo i neke kineze 123.125.68.125

bjasko@bjasko-laptop:~/vagrant/oneiric$ whois 123.125.68.125

% [whois.apnic.net node-4]
% Whois data copyright terms    http://www.apnic.net/db/dbcopyright.html

inetnum:      123.112.0.0 - 123.127.255.255
netname:      UNICOM-BJ
descr:        China Unicom Beijing province network
descr:        China Unicom
country:      CN
admin-c:      CH1302-AP
tech-c:       SY21-AP
mnt-by:       APNIC-HM
mnt-lower:    MAINT-CNCGROUP-BJ
mnt-routes:   MAINT-CNCGROUP-RR
status:       ALLOCATED PORTABLE

ovo je neki pretraživač nije uporan kao rus

ovdje nam nginx baš i nebi pomogao, radi se u jednom zahtjevu a ne multi tako da to ograničenje ne bi pomoglo, kvaka je što je naporan, koristi pretragu, nakon blokade je redmine prodisao

ima nginx dosta opcija ograničenja ... neka bi nam svakako trebalo postaviti.

recimo može i po url-u uraditi blokiranja. masa je "rampi" koje bi se mogle preventivno postaviti

mora da ima neka caka, treba to istražiti

danas je google navalio 66.249.72.38, mora da ima interesantnih tema pa je malo naporniji

(01:14:47 PM) hernad: http://www.modrails.com/documentation/Users%20guide%20Apache.html#PassengerMaxPoolSize
(01:15:11 PM) hernad: ovdje imaš ovu passenger_max_pool_size
(01:15:18 PM) hernad: za nginx
(01:15:37 PM) hernad: passenger_pool_idle_time
(01:16:57 PM) hernad: ima i ovo:
(01:16:58 PM) hernad: The maximum number of requests an application instance will process. After serving that many requests, the application instance will be shut down and Phusion Passenger will restart it. A value of 0 means that there is no maximum: an application instance will thus be shut down when its idle timeout has been reached.

(01:17:11 PM) hernad: ali ne znam kako ide za nginx
(01:17:17 PM) hernad: vjerovatno passenger_max_requests

stanje kod zakucanog servera

root@redmine-1:/opt/nginx/conf# passenger-memory-stats

------------- Apache processes -------------
*** WARNING: The Apache executable cannot be found.
Please set the APXS2 environment variable to your 'apxs2' executable's filename, or set the HTTPD environment variable to your 'httpd' or 'apache2' executable's filename.

---------- Nginx processes ----------
PID    PPID   VMSize   Private  Name
-------------------------------------
11778  1      35.2 MB  0.2 MB   nginx: master process /opt/nginx/sbin/nginx
11779  11778  36.1 MB  1.1 MB   nginx: worker process
### Processes: 2
### Total private dirty RSS: 1.23 MB

----- Passenger processes ------
PID    VMSize    Private   Name
--------------------------------
11538  630.1 MB  487.4 MB  Rails: /home/redmine/redmine
11734  22.7 MB   0.3 MB    PassengerWatchdog
11744  102.3 MB  1.8 MB    PassengerHelperAgent
11745  60.0 MB   10.0 MB   Passenger spawn server
11748  70.3 MB   0.8 MB    PassengerLoggingAgent
13693  627.4 MB  496.9 MB  Rails: /home/redmine/redmine
18149  616.4 MB  469.5 MB  Rails: /home/redmine/redmine
19506  599.0 MB  452.0 MB  Rails: /home/redmine/redmine
21557  547.7 MB  404.7 MB  Rails: /home/redmine/redmine
21859  495.9 MB  349.0 MB  Rails: /home/redmine/redmine
### Processes: 10
### Total private dirty RSS: 2672.31 MB

i nakon restarta

root@redmine-1:/# passenger-memory-stats

------------- Apache processes -------------
*** WARNING: The Apache executable cannot be found.
Please set the APXS2 environment variable to your 'apxs2' executable's filename, or set the HTTPD environment variable to your 'httpd' or 'apache2' executable's filename.

---------- Nginx processes ----------
PID    PPID   VMSize   Private  Name
-------------------------------------
20361  1      35.2 MB  0.2 MB   nginx: master process /opt/nginx/sbin/nginx
20362  20361  35.7 MB  0.6 MB   nginx: worker process
### Processes: 2
### Total private dirty RSS: 0.76 MB

----- Passenger processes ------
PID    VMSize    Private   Name
--------------------------------
20320  22.7 MB   0.3 MB    PassengerWatchdog
20322  38.1 MB   1.4 MB    PassengerHelperAgent
20323  58.5 MB   8.5 MB    Passenger spawn server
20336  70.3 MB   0.8 MB    PassengerLoggingAgent
21889  223.2 MB  20.1 MB   Passenger ApplicationSpawner: /home/redmine/redmine
21898  248.8 MB  102.3 MB  Rails: /home/redmine/redmine
26171  223.3 MB  20.2 MB   Rails: /home/redmine/redmine
### Processes: 7
### Total private dirty RSS: 153.53 MB

nginx izmjene

root@redmine-1:/# cat /opt/nginx/conf/nginx.conf

    passenger_max_pool_size 15;
    passenger_pool_idle_time 10;

   server {
      listen 80;
      server_name redmine.bring.out.ba;
      root /home/redmine/redmine/public;
      passenger_enabled on;
      passenger_min_instances 3;

  }

opet oko 11:10

root@redmine-1:/# passenger-status

*** Cleaning stale folder /tmp/passenger.1.0.30501
*** Cleaning stale folder /tmp/passenger.1.0.20283
*** Cleaning stale folder /tmp/passenger.1.0.20301
----------- General information -----------
max      = 15
count    = 10
active   = 10
inactive = 0
Waiting on global queue: 0

----------- Application groups -----------
/home/redmine/redmine:
  App root: /home/redmine/redmine
  * PID: 23832   Sessions: 1    Processed: 4903    Uptime: 41h 23m 56s
  * PID: 32303   Sessions: 1    Processed: 189     Uptime: 2h 1m 18s
  * PID: 18176   Sessions: 1    Processed: 508     Uptime: 13h 8m 43s
  * PID: 22127   Sessions: 1    Processed: 1       Uptime: 1h 6m 56s
  * PID: 30612   Sessions: 1    Processed: 10      Uptime: 58m 6s
  * PID: 32510   Sessions: 1    Processed: 2       Uptime: 56m 17s
  * PID: 3316    Sessions: 1    Processed: 0       Uptime: 53m 10s
  * PID: 11315   Sessions: 1    Processed: 3       Uptime: 45m 29s
  * PID: 14263   Sessions: 1    Processed: 1       Uptime: 41m 42s
  * PID: 19546   Sessions: 1    Processed: 0       Uptime: 37m 22s

root@redmine-1:/# passenger-memory-stats

------------- Apache processes -------------
*** WARNING: The Apache executable cannot be found.
Please set the APXS2 environment variable to your 'apxs2' executable's filename, or set the HTTPD environment variable to your 'httpd' or 'apache2' executable's filename.

--------- Nginx processes ---------
PID   PPID  VMSize   Private  Name
-----------------------------------
7564  1     35.4 MB  0.0 MB   nginx: master process /opt/nginx/sbin/nginx
7565  7564  36.3 MB  0.7 MB   nginx: worker process
### Processes: 2
### Total private dirty RSS: 0.68 MB

----- Passenger processes ------
PID    VMSize    Private   Name
--------------------------------
3316   538.7 MB  355.1 MB  Rails: /home/redmine/redmine
7522   23.0 MB   0.1 MB    PassengerWatchdog
7525   109.6 MB  2.8 MB    PassengerHelperAgent
7527   62.2 MB   8.9 MB    Passenger spawn server
7537   70.8 MB   0.0 MB    PassengerLoggingAgent
11315  513.4 MB  333.0 MB  Rails: /home/redmine/redmine
14263  502.2 MB  319.8 MB  Rails: /home/redmine/redmine
18007  228.8 MB  43.7 MB   Passenger ApplicationSpawner: /home/redmine/redmine
18176  651.3 MB  510.4 MB  Rails: /home/redmine/redmine
19546  488.0 MB  304.1 MB  Rails: /home/redmine/redmine
22127  610.7 MB  433.8 MB  Rails: /home/redmine/redmine
23832  707.0 MB  548.3 MB  Rails: /home/redmine/redmine
30612  558.0 MB  392.0 MB  Rails: /home/redmine/redmine
32303  672.7 MB  531.3 MB  Rails: /home/redmine/redmine
32510  553.8 MB  371.2 MB  Rails: /home/redmine/redmine

najverovatniji uzrok:

internet puca > zenoss šalje hifte notifikacija i pocuga resurse > zimbra hendlira notifikacije i pocuga ostatak resursa > redmine ostaje zakucan.

zimbra i zenoss se znaju vratiti u normalu

amavis na zimbri kod hendliranja velikog broja mailova zauzme cup 100%

 PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                   
11003 1000      15   0  207m  60m 3928 S   16  0.8   0:02.64 amavisd                                                                                   
30885 1000      15   0  207m  60m 3928 S   16  0.8   0:01.99 amavisd                                                                                   
11008 1000      17   0  207m  60m 3928 R   14  0.8   0:01.97 amavisd                                                                                   
11235 1000      16   0  208m  61m 4116 S   14  0.8   0:03.69 amavisd                                                                                   
11007 1000      16   0  207m  60m 3928 S    6  0.8   0:01.94 amavisd                                                                                   
11009 1000      15   0  208m  61m 4116 S    6  0.8   0:03.81 amavisd                                                                                   
10919 1000      16   0  207m  59m 3928 S    4  0.8   0:00.73 amavisd                                                                                   
11000 1000      15   0  209m  62m 4112 S    4  0.8   0:02.29 amavisd