Битые блоки в system.dbf

pravednik
Дата: 08.12.2007 02:04:02
Доброго времени суток.
Досталась мне база со следующей картиной

Oracle 10.2.0.1
Произошли проблемы с питанием, в следствии чего упала БД и покарежились несколько дисков
База моунтится и открывается, но в открытом состоянии живет не долго.
Содержание алерта следующее
Fri Dec  7 19:26:04 2007
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Picked latch-free SCN scheme 2
Autotune of undo retention is turned on. 
IMODE=BR
ILAT =48
LICENSE_MAX_USERS = 0
SYS auditing is disabled
ksdpec: called for event 13740 prior to event group initialization
Starting up ORACLE RDBMS Version: 10.2.0.1.0.
System parameters with non-default values:
  processes                = 400
  sessions                 = 445
  __shared_pool_size       = 184549376
  __large_pool_size        = 16777216
  __java_pool_size         = 33554432
  __streams_pool_size      = 33554432
  nls_language             = ENGLISH
  nls_territory            = RUSSIA
  filesystemio_options     = SETALL
  sga_target               = 2147483648
  control_files            = /oradata1/KPINET/control01.ctl, /oradata1/KPINET/control02.ctl, /oradata1/KPINET/control03.ctl
  db_block_size            = 16384
  __db_cache_size          = 1862270976
  compatible               = 10.2.0.1.0
  log_archive_dest_1       = LOCATION=/oradata2/arch/KPINET
  log_archive_format       = %t_%s_%r.dbf
  db_file_multiblock_read_count= 16
  cluster_database         = FALSE
  db_create_online_log_dest_1= /oradata1
  db_create_online_log_dest_2= /oradata2
  undo_management          = AUTO
  undo_tablespace          = UNDOTBS1
  undo_retention           = 1200
  remote_login_passwordfile= EXCLUSIVE
  db_domain                = 
  session_cached_cursors   = 100
  smtp_out_server          = relay1.ntu-kpi.kiev.ua
  plsql_warnings           = ENABLE:ALL
  job_queue_processes      = 10
  background_dump_dest     = /oracle/admin/KPINET/bdump
  user_dump_dest           = /oracle/admin/KPINET/udump
  core_dump_dest           = /oracle/admin/KPINET/cdump
  audit_file_dest          = /oracle/admin/KPINET/adump
  db_name                  = KPINET
  open_cursors             = 300
  optimizer_mode           = CHOOSE
  pga_aggregate_target     = 838860800
  _awr_flush_threshold_metrics= FALSE
PMON started with pid=2, OS id=18583
PSP0 started with pid=3, OS id=18585
MMAN started with pid=4, OS id=18587
LGWR started with pid=6, OS id=18592
DBW0 started with pid=5, OS id=18589
CKPT started with pid=7, OS id=18594
SMON started with pid=8, OS id=18596
RECO started with pid=9, OS id=18598
CJQ0 started with pid=10, OS id=18600
MMON started with pid=11, OS id=18602
MMNL started with pid=12, OS id=18604
Fri Dec  7 19:26:05 2007
ALTER DATABASE   MOUNT
Fri Dec  7 19:26:09 2007
Setting recovery target incarnation to 2
Fri Dec  7 19:26:09 2007
Successful mount of redo thread 1, with mount id 698604877
Fri Dec  7 19:26:09 2007
Database mounted in Exclusive Mode
Completed: ALTER DATABASE   MOUNT
Fri Dec  7 19:26:09 2007
ALTER DATABASE OPEN
Block change tracking file is current.
Fri Dec  7 19:26:14 2007
Beginning crash recovery of 1 threads
 parallel recovery started with 2 processes
Fri Dec  7 19:26:15 2007
Started redo scan
Fri Dec  7 19:26:15 2007
Completed redo scan
 244 redo blocks read, 71 data blocks need recovery
Fri Dec  7 19:26:15 2007
Started redo application at
 Thread 1: logseq 104722, block 2
Fri Dec  7 19:26:16 2007
Recovery of Online Redo Log: Thread 1 Group 1 Seq 104722 Reading mem 0
  Mem# 0 errs 0: /oradata1/KPINET/redo01.log
Fri Dec  7 19:26:16 2007
Completed redo application
Fri Dec  7 19:26:16 2007
Completed crash recovery at
 Thread 1: logseq 104722, block 246, scn 552480083
 71 data blocks read, 70 data blocks written, 244 redo blocks read
Fri Dec  7 19:26:16 2007
LGWR: STARTING ARCH PROCESSES
ARC0 started with pid=16, OS id=18615
Fri Dec  7 19:26:16 2007
ARC0: Archival started
ARC1 started with pid=17, OS id=18617
Fri Dec  7 19:26:16 2007
ARC1: Archival started
LGWR: STARTING ARCH PROCESSES COMPLETE
Thread 1 advanced to log sequence 104723
Thread 1 opened at log sequence 104723
  Current log# 2 seq# 104723 mem# 0: /oradata1/KPINET/redo02.log
Successful open of redo thread 1
Fri Dec  7 19:26:17 2007
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Fri Dec  7 19:26:17 2007
ARC0: STARTING ARCH PROCESSES
Fri Dec  7 19:26:17 2007
ARC1: Becoming the 'no FAL' ARCH
ARC1: Becoming the 'no SRL' ARCH
Fri Dec  7 19:26:17 2007
ARC2: Archival started
ARC0: STARTING ARCH PROCESSES COMPLETE
ARC0: Becoming the heartbeat ARCH
ARC2 started with pid=18, OS id=18620
Fri Dec  7 19:26:17 2007
Starting background process CTWR
CTWR started with pid=19, OS id=18622
Block change tracking service is active.
Fri Dec  7 19:26:18 2007
SMON: enabling cache recovery
Fri Dec  7 19:26:20 2007
Successfully onlined Undo Tablespace 1.
Fri Dec  7 19:26:20 2007
SMON: enabling tx recovery
Fri Dec  7 19:26:20 2007
Database Characterset is AL32UTF8
Fri Dec  7 19:26:20 2007
Errors in file /oracle/admin/KPINET/bdump/kpinet_smon_18596.trc:
ORA-00600: Message 600 not found; No message file for product=RDBMS, facility=ORA; arguments: [ktfbhget-1]
Fri Dec  7 19:26:22 2007
Non-fatal internal error happenned while SMON was doing temporary segment drop.
SMON encountered 1 out of maximum 100 non-fatal internal errors.
Fri Dec  7 19:26:22 2007
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
QMNC started with pid=20, OS id=18625
Fri Dec  7 19:26:23 2007
Errors in file /oracle/admin/KPINET/bdump/kpinet_smon_18596.trc:
ORA-00600: Message 600 not found; No message file for product=RDBMS, facility=ORA; arguments: [ktfbhget-1]
Fri Dec  7 19:26:24 2007
Non-fatal internal error happenned while SMON was doing temporary segment drop.
SMON encountered 2 out of maximum 100 non-fatal internal errors.
Fri Dec  7 19:26:25 2007
Errors in file /oracle/admin/KPINET/bdump/kpinet_smon_18596.trc:
ORA-00600: Message 600 not found; No message file for product=RDBMS, facility=ORA; arguments: [ktfbhget-1]
Non-fatal internal error happenned while SMON was doing temporary segment drop.
SMON encountered 3 out of maximum 100 non-fatal internal errors.
Fri Dec  7 19:26:27 2007
Errors in file /oracle/admin/KPINET/bdump/kpinet_smon_18596.trc:
ORA-00600: Message 600 not found; No message file for product=RDBMS, facility=ORA; arguments: [ktfbhget-1]
Non-fatal internal error happenned while SMON was doing temporary segment drop.
SMON encountered 4 out of maximum 100 non-fatal internal errors.
Fri Dec  7 19:26:29 2007
Errors in file /oracle/admin/KPINET/bdump/kpinet_smon_18596.trc:
ORA-00600: Message 600 not found; No message file for product=RDBMS, facility=ORA; arguments: [ktfbhget-1]
Fri Dec  7 19:26:29 2007
Completed: ALTER DATABASE OPEN
Fri Dec  7 19:26:29 2007
Non-fatal internal error happenned while SMON was doing temporary segment drop.
SMON encountered 5 out of maximum 100 non-fatal internal errors.
Fri Dec  7 19:26:38 2007
Errors in file /oracle/admin/KPINET/bdump/kpinet_j000_18638.trc:
ORA-00600: internal error code, arguments: [ktfbhget-4], [31], [30], [], [], [], [], []
Fri Dec  7 19:26:39 2007
Errors in file /oracle/admin/KPINET/bdump/kpinet_smon_18596.trc:
ORA-00600: Message 600 not found; No message file for product=RDBMS, facility=ORA; arguments: [ktfbhget-1]
Fri Dec  7 19:26:40 2007
Non-fatal internal error happenned while SMON was doing temporary segment drop.
SMON encountered 6 out of maximum 100 non-fatal internal errors.
Fri Dec  7 19:26:50 2007
Errors in file /oracle/admin/KPINET/bdump/kpinet_smon_18596.trc:
ORA-00600: Message 600 not found; No message file for product=RDBMS, facility=ORA; arguments: [ktfbhget-1]
Fri Dec  7 19:26:51 2007
Non-fatal internal error happenned while SMON was doing temporary segment drop.
SMON encountered 7 out of maximum 100 non-fatal internal errors.
Fri Dec  7 19:27:01 2007
Errors in file /oracle/admin/KPINET/bdump/kpinet_smon_18596.trc:
ORA-00600: Message 600 not found; No message file for product=RDBMS, facility=ORA; arguments: [ktfbhget-1]
Fri Dec  7 19:27:01 2007
Non-fatal internal error happenned while SMON was doing temporary segment drop.
SMON encountered 8 out of maximum 100 non-fatal internal errors.
Fri Dec  7 19:27:11 2007
Errors in file /oracle/admin/KPINET/bdump/kpinet_smon_18596.trc:
ORA-00600: Message 600 not found; No message file for product=RDBMS, facility=ORA; arguments: [ktfbhget-1]
Fri Dec  7 19:27:14 2007
Errors in file /oracle/admin/KPINET/bdump/kpinet_pmon_18583.trc:
ORA-00474: Message 474 not found; No message file for product=RDBMS, facility=ORA
Fri Dec  7 19:27:14 2007
PMON: terminating instance due to error 474
Instance terminated by PMON, pid = 18583 

Все переменные среды установленны корректно.....
Прогнал dbv и получил следующее
dbv file=.../system01.dbf logfile=/home/oracle/log.txt  blocksize=16384

DBVERIFY: Release 10.2.0.1.0 - Production on Fri Dec 7 23:43:32 2007

Copyright (c) 1982, 2005, Oracle.  All rights reserved.

DBVERIFY - Verification complete

Total Pages Examined         : 33920
Total Pages Processed (Data) : 20620
Total Pages Failing   (Data) : 0
Total Pages Processed (Index): 3923
Total Pages Failing   (Index): 0
Total Pages Processed (Other): 1834
Total Pages Processed (Seg)  : 0
Total Pages Failing   (Seg)  : 0
Total Pages Empty            : 7479
Total Pages Marked Corrupt   : 64
Total Pages Influx           : 0
Highest block SCN            : 552460077 (0.552460077)
SQL> select * from v$database_block_corruption;

     FILE#     BLOCK#     BLOCKS CORRUPTION_CHANGE# CORRUPTIO
---------- ---------- ---------- ------------------ ---------
         1      33281         63                  0 ALL ZERO
         1      33407          1                  0 ALL ZERO

С бекапами ситуция следующая.....

последний бекап(...отссутстиве арков с 104616 по 104671) (арки с 104671 и до сегодня )
архивлоги с 104616 по 104671 были на одном из збойнутых дисков и им глина...

доступа на металинк пока, вернее уже, нет....и не известно, когда будет...
До того, как обнаружил битые в system, пересоздавал файлы Темп ТП и пробывал запустить бд с UNDO_MANAGMENT=MANUAL....не помогло

kpinet_j000_18638.rar
kpinet_smon_18596.rar

буду рад любой помощи и советам.
Заранее спасибо
.....
Дата: 08.12.2007 10:46:17
попробовать убрать чистку smon - в конце инишника прописать

event="10061 trace name context forever, level 10"
pravednik
Дата: 08.12.2007 14:14:46
.....
попробовать убрать чистку smon - в конце инишника прописать

event="10061 trace name context forever, level 10"


ОГРОМНОЕ человеческое спасибо.....оно открылося....
алерт
Sat Dec  8 12:57:13 2007
ALTER DATABASE OPEN
Block change tracking file is current.
Sat Dec  8 12:57:14 2007
Beginning crash recovery of 1 threads
 parallel recovery started with 2 processes
Sat Dec  8 12:57:15 2007
Started redo scan
Sat Dec  8 12:57:15 2007
Completed redo scan
 267 redo blocks read, 86 data blocks need recovery
Sat Dec  8 12:57:16 2007
Started redo application at
 Thread 1: logseq 104723, block 2
Sat Dec  8 12:57:16 2007
Recovery of Online Redo Log: Thread 1 Group 2 Seq 104723 Reading mem 0
  Mem# 0 errs 0: /oradata1/KPINET/redo02.log
Sat Dec  8 12:57:16 2007
Completed redo application
Sat Dec  8 12:57:16 2007
Completed crash recovery at
 Thread 1: logseq 104723, block 269, scn 552500292
 86 data blocks read, 85 data blocks written, 267 redo blocks read
Sat Dec  8 12:57:16 2007
LGWR: STARTING ARCH PROCESSES
ARC0 started with pid=16, OS id=30524
Sat Dec  8 12:57:16 2007
ARC0: Archival started
ARC1: Archival started
LGWR: STARTING ARCH PROCESSES COMPLETE
ARC1 started with pid=17, OS id=30526
Sat Dec  8 12:57:16 2007
Thread 1 advanced to log sequence 104724
Thread 1 opened at log sequence 104724
  Current log# 3 seq# 104724 mem# 0: /oradata1/KPINET/redo03.log
Successful open of redo thread 1
Sat Dec  8 12:57:16 2007
MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set
Sat Dec  8 12:57:16 2007
ARC0: STARTING ARCH PROCESSES
Sat Dec  8 12:57:16 2007
ARC1: Becoming the 'no FAL' ARCH
ARC1: Becoming the 'no SRL' ARCH
Sat Dec  8 12:57:16 2007
ARC2: Archival started
ARC0: STARTING ARCH PROCESSES COMPLETE
ARC0: Becoming the heartbeat ARCH
ARC2 started with pid=18, OS id=30528
Sat Dec  8 12:57:17 2007
Starting background process CTWR
CTWR started with pid=19, OS id=30530
Block change tracking service is active.
Sat Dec  8 12:57:17 2007
SMON: enabling cache recovery
Sat Dec  8 12:57:18 2007
Successfully onlined Undo Tablespace 1.
Sat Dec  8 12:57:18 2007
SMON: enabling tx recovery
Sat Dec  8 12:57:18 2007
Database Characterset is AL32UTF8
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
QMNC started with pid=20, OS id=30533
Sat Dec  8 12:57:23 2007
Completed: ALTER DATABASE OPEN
Sat Dec  8 12:57:34 2007
Errors in file /oracle/admin/KPINET/bdump/kpinet_j000_30536.trc:
ORA-00600: internal error code, arguments: [ktfbhget-4], [31], [30], [], [], [], [], []
Sat Dec  8 12:57:36 2007
Errors in file /oracle/admin/KPINET/bdump/kpinet_j001_30545.trc:
ORA-12012: error on auto execute of job 8887
ORA-00376: file ORA-00376: file 31 cannot be read at this time
ORA-01110: data file 31: '/oradata1/KPINET/VPN_2007_10.idx'
ORA-06512: at "SYS.PRVT_ADVISOR", line 1624
ORA-06512: at "SYS.DBMS_ADVISOR", line 186
ORA-06512: at "SYS.DBMS_SPACE", line 1488
ORA-06512: at "SYS.DBMS_SPACE", line 1554
 cannot be read at this time
Sat Dec  8 12:57:38 2007
Errors in file /oracle/admin/KPINET/bdump/kpinet_j003_30551.trc:
ORA-12012: error on auto execute of job 8886
ORA-08103: object no longer exists
Sat Dec  8 13:03:16 2007
Shutting down archive processes
Sat Dec  8 13:03:21 2007
ARCH shutting down
ARC2: Archival stopped
.....
Дата: 08.12.2007 15:02:44
pravednik
[quot .....].....оно открылося....


это не избавляет от проблемы. надо смотреть дальше, на чем сбоит, что-за транзация, с чем связана.
pravednik
Дата: 08.12.2007 15:42:14
.....
pravednik
[quot .....].....оно открылося....


это не избавляет от проблемы. надо смотреть дальше, на чем сбоит, что-за транзация, с чем связана.


я понимаю, что не избавляет......смысл задачи в том, чтоб вытянуть из бд по возможности все данные и перенести на новую базу.....
juks@gala.net
Дата: 09.12.2007 06:27:20
pravednik ,ты бы спросил, что делает этот ивент, где про него прочитать )
juks@gala.net
Дата: 09.12.2007 06:29:17
pravednik
.....
pravednik
[quot .....].....оно открылося....


это не избавляет от проблемы. надо смотреть дальше, на чем сбоит, что-за транзация, с чем связана.


я понимаю, что не избавляет......смысл задачи в том, чтоб вытянуть из бд по возможности все данные и перенести на новую базу.....

У тебя два битых блока, найди что за обьекты в ТС
pravednik
Дата: 09.12.2007 07:51:06
juks@gala.net
pravednik ,ты бы спросил, что делает этот ивент, где про него прочитать )


ну гуглем то пользоваться я умею...:) ...вроде....

попользованый гугль
pravednik
Дата: 09.12.2007 08:12:34
juks@gala.net

У тебя два битых блока, найди что за обьекты в ТС


64.....но не важно, этим и занимаюся....тоже..:)
juks@gala.net
Дата: 09.12.2007 08:26:30
pravednik
juks@gala.net

У тебя два битых блока, найди что за обьекты в ТС


64.....но не важно, этим и занимаюся....тоже..:)

На этом Дальнем Востоке уже все мозги отморозил (