Solaris mon amour

Mais que s'est-il passé Mercredi dernier ? Vous ne suivez pas l'actualité de l'association Guses !? C'est vraiment dommage pour vous...

En collaboration avec Oracle, le Guses a participé activement au TechDay Solaris 2013. Je ne vais pas vous faire un résumé de la soirée car Eric Bezille et Axel Paratre ont déjà écrit à ce sujet. Je vais juste profiter de cet article pour publier les deux présentations du Guses (ainsi que celles des précédentes éditions déjà disponibles sur mon blog).

Je remercie particulièrement René Garcia (Ingénieur Unix chez PSA Peugeot Citroen) pour sa présentation ZFS. Merci aussi à vous tous de votre présence, en espérant vous croiser lors d'une prochaine soirée que nous organisons.

Les présentations du TechDay 2013 :

• ZFS Internal (par René Garcia)
• Consolidation avec Solaris 11 (par moi-même - membre du Guses)

Si jamais cela vous a échappé, le Guses a aussi participé aux précédents TechDay Solaris... Petit rappel des présentations...

Les présentations du TechDay 2011 :

• Intro
• Présentation du Guses (par Jean-Christophe Martin - membre du Guses)
• Crossbow
• IPS
• Performance
• Retour d'expérience (par Cyril Galibern - membre du Guses)

La présentation du TechDay 2012 :

• Automated-Installer (par moi-même - membre du Guses)

Maintenant plus d'excuses suivez le Guses...

• Twitter @AssocGUSES
• guses@googlegroups.com
• ug-fosug@yahoogroups.com
• www.guses.org

I describe only migration P2V of a physical server in a ldom, the installation and the configuration of Oracle VM Server for Sparc are not specified in this article.

Some details:

• The name of physical server is ldom-guest (Solaris 10u3 – kernel 118833-33)
• The name of crontol domain is ldom-crtl (Solaris 11.1 SRU 5.5)

There are a 3 phases to migrate from a physical system to a virtual system:

• Collection Phase: A filesystem source is created based on the configuration information that it collects about the source system.
• Preparation Phase: A logical domain is created
• Conversion Phase: The filesystem source is converted into a logical domain (ex: conversion from sun4u to sun4v)

To execute this procedure, you must use tool ldmp2v (download this path p15880570 to obtain the tool - In Solaris 11, this tool is directly available).

Before starting, let us look at the configuration available on control domain:

ldom-crtl # ldm –V

Logical Domains Manager (v 3.0.0.2)
        Hypervisor control protocol v 1.7
        Using Hypervisor MD v 1.3

System PROM:
        Hypervisor v. 1.10.0. @(#)Hypervisor 1.10.0.a 2011/07/15 11:51\015
        OpenBoot   v. 4.33.0. @(#)OpenBoot 4.33.0.b 2011/05/16 16:28

ldom-crtl # ldm ls –o console,network,disk primary
[…]

VCC
    NAME           PORT-RANGE
    primary-vcc0   5000-5100

VSW
    NAME           MAC          […]
    primary-vsw0   x:x:x:x:x:x  […]

VDS
    NAME           VOLUME       […]
    primary-vds0

[…]

A traditional configuration !?, no ?

Fisrt step: Collection phase (runs on the physical source system)

To create a consistent file system image, I suggest you to boot the server in “single mode”. To save a file system image, I often use a NFS share.

ldom-guest # mount –F nfs myshare:/tempo /mnt

By default, the cmd ldmp2v creates a flar image.

ldom-guest # /usr/sbin/ldmp2v collect -d /mnt/ldom-guest
Collecting system configuration ...
Archiving file systems ...
Full Flash
Checking integrity...
Integrity OK.
Running precreation scripts...
Precreation scripts done.
Creating the archive...
136740734 blocks
Archive creation complete.

ldom-guest # init 0

Second step: Preparation phase (runs on the control domain)

I start by creating a ZFS pool which will contain the data of the logical domain.

ldmon-crtl # zpool create -m none ldom-guest cXtYdZ

I prefer to use the manual mode to create a logical domain (so I edit the following file ldmp2v.conf).

ldmon-crtl # cat /etc/ldmp2v.conf
# Virtual switch to use
VSW="primary-vsw0"
# Virtual disk service to use
VDS="primary-vds0"
# Virtual console concentrator to use
VCC="primary-vcc0"
# Location where vdisk backend devices are stored
BACKEND_PREFIX=""
# Default backend type: "zvol" or "file".
BACKEND_TYPE="zvol"
# Create sparse backend devices: "yes" or "no"
BACKEND_SPARSE="yes"
# Timeout for Solaris boot in seconds
BOOT_TIMEOUT=60

Just after mounted the share NFS, I create a logical domain by indicating the following informations: cpu, mem and prefix (here it is name of ZFS pool)

ldom-crtl # mount –F nfs myshare:/tempo /mnt
ldom-crtl # ldmp2v prepare -c 16 -M 16g –p ldom-guest -d /mnt/ldom-guest ldom-guest
Creating vdisks ...
Creating file systems ...
Populating file systems ...
136740734 blocks
Modifying guest OS image ...
Modifying SVM configuration ...
Unmounting file systems ...
Creating domain ...
Attaching vdisks to domain ldom-guest ...

For this example, the ldom guest is configured with 16 vcpu and 16 Go (options –c and –M).

Final step: Conversion phase (runs on the control domain)

In the conversion phase, the logical domain uses the Oracle Solaris upgrade process to upgrade to the Oracle Solaris 10 OS. The upgrade operation removes all existing packages and installs the Oracle Solaris 10 sun4v packages, which automatically performs a sun4u-to-sun4v conversion. The convert phase can use an Oracle Solaris DVD ISO image or a network installation image. On Oracle Solaris 10 systems, you can also use the Oracle Solaris JumpStart feature to perform a fully automated upgrade operation.

On “jumpstart server” (do you known jet ?), I edit the jumpstart profile to add the following lines

install_type    upgrade
root_device c0d0s0

Ready for conversation !! The last command to convert the Sparc architecture and to start the guest domain.

ldom-crtl # ldmp2v convert -j -n vnet0 -d /mnt/ldom-guest ldmon-guest
Testing original system status ...
LDom ldom-guest started
Waiting for Solaris to come up ...
Using Custom JumpStart
Trying 0.0.0.0...
Connected to 0.
Escape character is '^]'.

Connecting to console "server" in group "server" ....
Press ~? for control options ..
Configuring devices.
Using RPC Bootparams for network configuration information.
Attempting to configure interface vnet0...
Configured interface vnet0
Setting up Java. Please wait...
Extracting windowing system. Please wait...
Beginning system identification...
Searching for configuration file(s)...
Using sysid configuration file 10.x.x.x:/opt/SUNWjet/Clients/ldom-guest/sysidcfg
Search complete.
Discovering additional network configuration...
Completing system identification...
Starting remote procedure call (RPC) services: done.
System identification complete.
Starting Solaris installation program...
Searching for JumpStart directory...
Using rules.ok from 10.x.x.x:/opt/SUNWjet.
Checking rules.ok file...
Using begin script: Utils/begin
Using derived profile: Utils/begin
Using finish script: Utils/finish
Executing JumpStart preinstall phase...
Executing begin script "Utils/begin"...
Installation of ldom-guest at 00:41 on 10-May-2013
Loading JumpStart Server variables
Loading JumpStart Server variables
Loading Client configuration file
Loading Client configuration file
Running base_config begin script....
Running base_config begin script....
Begin script Utils/begin execution completed.
Searching for SolStart directory...
Checking rules.ok file...
Using begin script: install_begin
Using finish script: patch_finish
Executing SolStart preinstall phase...
Executing begin script "install_begin"...
Begin script install_begin execution completed.

WARNING: Backup media not specified.  A backup media (backup_media) keyword must be specified if an upgrade with disk space reallocation is required

Processing default locales
       - Specifying default locale (en_US.ISO8859-1)

Processing profile

Loading local environment and services

Generating upgrade actions
       - Selecting locale (en_US.ISO8859-1)

Checking file system space: 100% completed
Space check complete.

Building upgrade script

Preparing system for Solaris upgrade

Upgrading Solaris: 101% completed
       - Environment variables (/etc/default/init)

Installation log location
       - /a/var/sadm/system/logs/upgrade_log (before reboot)
       - /var/sadm/system/logs/upgrade_log (after reboot)

Please examine the file:
       - /a/var/sadm/system/data/upgrade_cleanup

It contains a list of actions that may need to be performed to complete the upgrade. After this system is rebooted, this file can be found at:
       - /var/sadm/system/data/upgrade_cleanup

Upgrade complete
Executing SolStart postinstall phase...
Executing finish script "patch_finish"...

Finish script patch_finish execution completed.
Executing JumpStart postinstall phase...
Executing finish script "Utils/finish"...
[…]
Terminated

Finish script Utils/finish execution completed.
The begin script log 'begin.log'
is located in /var/sadm/system/logs after reboot.

The finish script log 'finish.log'
is located in /var/sadm/system/logs after reboot.

syncing file systems... done
rebooting...
Resetting...

T5240, No Keyboard
Copyright (c) 1998, 2011, Oracle and/or its affiliates. All rights reserved.
OpenBoot 4.33.0.b, 16384 MB memory available, Serial #83470255.
Ethernet address 0:x:x:x:x:x, Host ID: 84f9a7af.

Boot device: disk0:a  File and args:
SunOS Release 5.10 Version Generic_118833-33 64-bit
Copyright 1983-2006 Sun Microsystems, Inc.  All rights reserved.
Use is subject to license terms.
Hostname: ldom-guest
Loading smf(5) service descriptions: 1/1
checking ufs filesystems
/dev/rdsk/c0d1s0: is logging.

ldom-guest console login:

It’s already finishes. Is it simple !? You do not have any more excuses not to use ldoms.

See Also

• Oracle VM for Sparc (Technologie)
• Oracle VM for Sparc (Oracle Documentation)
• MAN for ldmp2v(Oracle Documentation)

Everyone knows that one of the major problem for consolidating Solaris 10 is network. if each Solaris Zones use a different network (vlan), the configuration of the Global Zone becomes a real headache.

In Solaris 11, Crosbow effectively addresses this problem. This article explains how to create several Solaris Zone an emphasis on network configuration (several vlan).

In this example, there are 3 Solaris Zone running on dedicated systems that are exposed to the external networks. Each Solaris Zone runs a different vlan.

• The Global Zone running in vlan id 1 (Address: 192.168.1.10/24 - Router: 192.168.1.1)
• The Solaris Zone zone1 running in vlan id 1 (Address: 192.168.1.11/24 - Router: 192.168.1.1)
• The Solaris Zone zone2 running in vlan id 2 (Address: 192.168.2.10/24 - Router: 192.168.2.1)
• The Solaris Zone zone3 running in vlan id 3 (Address: 192.168.3.10/24 - Router: 192.168.3.1)
• Each port of NIC used by aggregation is configured in different vlans (vlan id 1, 2 and 3)

Let's go... Just wait... The network configuration of switch must be already configured (please contact network team !?)

Step 1: Create link aggregation

My system (Sparc M5000) includes 4 NICs. There is no network configuration yet (I connect on XSCF).

# dladm show-phys
LINK       MEDIA         STATE      SPEED  DUPLEX    DEVICE
net1       Ethernet      unknown    0      unknown   bge1
net0       Ethernet      unknown    0      unknown   bge0
net3       Ethernet      unknown    0      unknown   bge3
net2       Ethernet      unknown    0      unknown   bge2

I create a basic link aggregation (I don't use LACP) with 4 NICs.

# dladm create-aggr -P L2,L3 -l net0 -l net1 -l net2 -l net3 default0

I check quicly the status of the aggregation.

# dladm show-link
LINK          CLASS     MTU    STATE    OVER
net1          phys      1500   up       --
net0          phys      1500   up       --
net3          phys      1500   up       --
net2          phys      1500   up       --
default0      aggr      1500   up       net0 net1 net2 net3

Yet, I configure address on this aggregation.

# ipadm create-ip default0
# ipadm create-addr -T static -a local=192.168.1.10/24 default0/v4

Don't forget, the configuration of router.

# route add -p default 192.168.1.1 -ifp default0

Step 2: Create Solaris Zone for Cloning

It is much faster to clone Solaris Zone than to create one from scratch, because building an image from packages takes longer than, in essence, copying an existing zone. I use the cloning technique in this example to first create one Solaris Zone and then clone it three times.

# zfs create -o mountpoint=/zones -o dedup=on rpool/zones
# zfs create -o mountpoint=/zones/zclone rpool/zones/zclone
# chmod 700 /zones/zclone

# zonecfg -z zclone
Use 'create' to begin configuring a new zone.
zonecfg:zclone> create
create: Using system default template 'SYSdefault'
zonecfg:zclone> set zonepath=/zones/zclone
zonecfg:zclone> set ip-type=exclusive
zonecfg:zclone> exit

# zoneadm -z zclone install
Progress being logged to /var/log/zones/zoneadm.20130329T161207Z.zclone.install
       Image: Preparing at /zones/zclone/root. 
[...] 
  Next Steps: Boot the zone, then log into the zone console (zlogin -C)
              to complete the configuration process.
Log saved in non-global zone as /zones/zclone/root/var/log/zones/zoneadm.20130329T161207Z.zclone.install

# zoneadm -z zclone boot ; zlogin -C zclone
[Connected to zone 'zclone' console]
Loading smf(5) service descriptions: 115/115

When I obtain the screen to configure this Solaris Zone, I halt this zone.

# zoneadm -z zclone halt

Step 3: Create Solaris Zones zone1

Remimber, Solaris Zone zone1 use a same vlan that Global Zone. First, I create a vlan link over a datalink (default0).

# dladm create-vnic -v 1 -l default0 vnic1

Next, I create zone1 from the zclone zone (don't forget a profile creation - new sysidcfg).

# zonecfg -z zone1 "create -t zclone"
# zonecfg -z zone1
zonecfg:zone1> set zonepath=/zones/zone1
zonecfg:zone1> select anet linkname=net0
zonecfg:zone1:anet> set linkname=vnic1
zonecfg:zone1:anet> set lower-link=default0
zonecfg:zone1:anet> end
zonecfg:zone1> commit
zonecfg:zone1> exit

# zoneadm -z zone1 clone -c /tmp/sc_profile1.xml zclone
The following ZFS file system(s) have been created:
    rpool/zones/zone1
Progress being logged to /var/log/zones/zoneadm.20130329T172124Z.zone1.clone
Log saved in non-global zone as /zones/zone1/root/var/log/zones/zoneadm.20130329T172124Z.zone1.clone

Step 4: Create Solaris Zones zone2

Solaris Zone zone2 use a vlan id 2. First, I create a vlan link over a datalink (default0).

# dladm create-vnic -v 2 -l default0 vnic2

Next, I create zone2 from the zclone zone (don't forget a profile creation - new sysidcfg). Beware, I use the following paramater to configure the vlan id: vlan-id.

# zonecfg -z zone2 "create -t zclone"
# zonecfg -z zone2
zonecfg:zone2> set zonepath=/zones/zone2
zonecfg:zone2> select anet linkname=net0
zonecfg:zone2:anet> set linkname=vnic2
zonecfg:zone2:anet> set lower-link=default0
zonecfg:zone2:anet> set vlan-id=2
zonecfg:zone2:anet> end
zonecfg:zone2> commit
zonecfg:zone2> exit

# zoneadm -z zone2 clone -c /tmp/sc_profile2.xml zclone
The following ZFS file system(s) have been created:
    rpool/zones/zone2
Progress being logged to /var/log/zones/zoneadm.20130329T174913Z.zone2.clone
Log saved in non-global zone as /zones/zone2/root/var/log/zones/zoneadm.20130329T174913Z.zone2.clone

Step 5: Create Solaris Zones zone3

It's the same configuration than zone2, the only change comes from vlan id. This zone uses a vlan id 3.

# dladm create-vnic -v 3 -l default0 vnic3

# zonecfg -z zone3 "create -t zclone"
# zonecfg -z zone3
zonecfg:zone3> set zonepath=/zones/zone3
zonecfg:zone3> select anet linkname=net0
zonecfg:zone3:anet> set linkname=vnic3
zonecfg:zone3:anet> set lower-link=default0
zonecfg:zone3:anet> set vlan-id=3
zonecfg:zone3:anet> end
zonecfg:zone3> commit
zonecfg:zone3> exit

# zoneadm -z zone3 clone -c /tmp/sc_profile3.xml zclone
The following ZFS file system(s) have been created:
    rpool/zones/zone3
Progress being logged to /var/log/zones/zoneadm.20130329T175707Z.zone3.clone
Log saved in non-global zone as /zones/zone3/root/var/log/zones/zoneadm.20130329T175707Z.zone3.clone

Step 6: Start all Solaris Zone

My configuration is finished. I just start all zone.

# zoneadm list -cv
  ID NAME      STATUS     PATH               BRAND    IP   
   0 global    running    /                  solaris  shared
   - zclone    installed  /zones/zclone      solaris  excl 
   - zone1     installed  /zones/zone1       solaris  excl 
   - zone2     installed  /zones/zone2       solaris  excl 
   - zone3     installed  /zones/zone3       solaris  excl 

# zoneadm –z zone1 boot ; zoneadm –z zone2 boot ; zoneadm –z zone3 boot

Conclusion: We hope this step-by-step guide will give you some ideas for future consolidation. With Oracle Solaris 11 capabilities, you can easily set up fairly complex environments.

See Also

• Network Interfaces and Network Virtualization (Oracle Documentation)
• Oracle Solaris Zones, Oracle Solaris 10 Zones, and Resource Management (Oracle Documentation)
• Oracle Technology Network (All Solaris articles)

Petite observation lors d’une recette cluster d’un Oracle RAC 11gR2 sur Solaris 10 Sparc. Pendant le test suivant « perte d’une baie SAN », nous avons observé un petit problème lors de la resynchronisation des diskgroups ASM utilisant des volumes sous ACFS. Nous nous attendions à utiliser la fonctionnalité fastresync et pourtant…

Observons tout d’abord le contexte initial : « perte d'une baie SAN sur les deux nodes du clusters Oracle RAC ».

Perte de la baie sur les deux nodes du cluster RAC (observation système).

Feb 12 18:08:55 rac01 fctl: [ID 517869 kern.warning] WARNING: fp(6)::GPN_ID for D_ID=150300 failed
Feb 12 18:08:55 rac01 fctl: [ID 517869 kern.warning] WARNING: fp(6)::N_x Port with D_ID=150300, PWWN=5000097408231998 disappeared from fabric
Feb 12 18:09:52 rac01 fctl: [ID 517869 kern.warning] WARNING: fp(5)::GPN_ID for D_ID=b0300 failed
Feb 12 18:09:52 rac01 fctl: [ID 517869 kern.warning] WARNING: fp(5)::N_x Port with D_ID=b0300, PWWN=50000974082319a4 disappeared from fabric
[…]

Feb 12 18:08:55 rac02 fctl: [ID 517869 kern.warning] WARNING: fp(5)::GPN_ID for D_ID=150300 failed
Feb 12 18:08:55 rac02 fctl: [ID 517869 kern.warning] WARNING: fp(5)::N_x Port with D_ID=150300, PWWN=5000097408231998 disappeared from fabric
Feb 12 18:09:52 rac02 fctl: [ID 517869 kern.warning] WARNING: fp(6)::GPN_ID for D_ID=b0300 failed
Feb 12 18:09:52 rac02 fctl: [ID 517869 kern.warning] WARNING: fp(6)::N_x Port with D_ID=b0300, PWWN=50000974082319a4 disappeared from fabric
[…]

Confirmation des  « path dead » sous powerpath (observation multiptah).

rac01 # /etc/powermt display
Symmetrix logical device count=30
CLARiiON logical device count=0
Hitachi logical device count=0
Invista logical device count=0
HP xp logical device count=0
Ess logical device count=0
HP HSx logical device count=0

==============================================================================

----- Host Bus Adapters ---------  ------ I/O Paths -----  ------ Stats ------
###  HW Path                       Summary   Total   Dead  IO/Sec Q-IOs Errors

==============================================================================
3077 pci@11,700000/SUNW,qlc@0/fp@0,0 degraded    30     14      -     0     16
3078 pci@1,700000/SUNW,qlc@0/fp@0,0  degraded    30     14      -     0     20

rac02 # /etc/powermt display
Symmetrix logical device count=30
CLARiiON logical device count=0
Hitachi logical device count=0
Invista logical device count=0
HP xp logical device count=0
Ess logical device count=0
HP HSx logical device count=0

==============================================================================

----- Host Bus Adapters ---------  ------ I/O Paths -----  ------ Stats ------
###  HW Path                       Summary   Total   Dead  IO/Sec Q-IOs Errors

==============================================================================
3077 pci@1,700000/SUNW,qlc@0/fp@0,0  degraded    30     14      -     0     17
3078 pci@11,700000/SUNW,qlc@0/fp@0,0 degraded    30     14      -     0     17

Etat des diskgroups ASM (observation couche ASM).

$ asmcmd lsdsk -pk
 Name           Failgroup     … Mount_Stat … Mode_Stat State  Path
 DATA_DG1_0000  DATA_DG1_FG1  … MISSING    …   OFFLINE NORMAL 
 DATA_DG1_0001  DATA_DG1_FG1  … MISSING    …   OFFLINE NORMAL 
 DUMP_DG_0002   DUMP_DG_FG1   … MISSING    …   OFFLINE NORMAL 
 DUMP_DG_0009   DUMP_DG_FG1   … MISSING    …   OFFLINE NORMAL 
 SYSTEMDG_0000  SYSTEMDG_0000 … MISSING    …   OFFLINE NORMAL 
 DUMP_DG_0000   DUMP_DG_FG1   … MISSING    …   OFFLINE NORMAL 
 DUMP_DG_0001   DUMP_DG_FG1   … MISSING    …   OFFLINE NORMAL 
 DUMP_DG_0008   DUMP_DG_FG1   … MISSING    …   OFFLINE NORMAL 
 DUMP_DG_0003   DUMP_DG_FG1   … MISSING    …   OFFLINE NORMAL 
 REDO_DG1_0000  REDO_DG1_FG1  … MISSING    …   OFFLINE NORMAL 
 FLASH_DG1_0000 FLASH_DG1_FG1 … MISSING    …   OFFLINE NORMAL 
 DATA_DG1_0002  DATA_DG1_FG2  … CACHED     …   ONLINE  NORMAL  …
 DATA_DG1_0003  DATA_DG1_FG2  … CACHED     …   ONLINE  NORMAL  …
 DUMP_DG_0004   DUMP_DG_FG2   … CACHED     …   ONLINE  NORMAL  …
 DUMP_DG_0005   DUMP_DG_FG2   … CACHED     …   ONLINE  NORMAL  …
 DUMP_DG_0006   DUMP_DG_FG2   … CACHED     …   ONLINE  NORMAL  …
 DUMP_DG_0007   DUMP_DG_FG2   … CACHED     …   ONLINE  NORMAL  …
 DUMP_DG_0010   DUMP_DG_FG2   … CACHED     …   ONLINE  NORMAL  …
 DUMP_DG_0011   DUMP_DG_FG2   … CACHED     …   ONLINE  NORMAL  …
 FLASH_DG1_0001 FLASH_DG1_FG2 … CACHED     …   ONLINE  NORMAL  …
 REDO_DG1_0001  REDO_DG1_FG2  … CACHED     …   ONLINE  NORMAL  …
 SYSTEMDG_0004  SYSTEMDG_0004 … CACHED     …   ONLINE  NORMAL  …
 SYSTEMDG_0001  SYSTEMDG_000  … CACHED     …   ONLINE  NORMAL  …

La perte de la baie est confirmée par les différentes couches (serveurs, multipath, lvm). Le cluster RAC lui fonctionne correctement (un peu normal). Le test est concluant, passons maintenant à la « redécouverte » de la baie et la resynchronisation (fastresync) des diskgroups ASM.

Baie online sur les deux nodes du cluster RAC (observation système).

Feb 12 18:23:16 rac01 fctl: [ID 517869 kern.warning] WARNING: fp(6)::N_x Port with D_ID=150300, PWWN=5000097408231998 reappeared in fabric
Feb 12 18:23:34 rac01 fctl: [ID 517869 kern.warning] WARNING: fp(5)::N_x Port with D_ID=b0300, PWWN=50000974082319a4 reappeared in fabric
[…]

Feb 12 18:23:16 rac02 fctl: [ID 517869 kern.warning] WARNING: fp(5)::N_x Port with D_ID=150300, PWWN=5000097408231998 reappeared in fabric
Feb 12 18:23:34 rac02 fctl: [ID 517869 kern.warning] WARNING: fp(6)::N_x Port with D_ID=b0300, PWWN=50000974082319a4 reappeared in fabric
[…]

Confirmation des  « path dead » sous powerpath (observation multiptah)

rac01 # /etc/powermt display    
Symmetrix logical device count=30
CLARiiON logical device count=0
Hitachi logical device count=0
Invista logical device count=0
HP xp logical device count=0
Ess logical device count=0
HP HSx logical device count=0

==============================================================================

----- Host Bus Adapters ---------  ------ I/O Paths -----  ------ Stats ------
###  HW Path                       Summary   Total   Dead  IO/Sec Q-IOs Errors

==============================================================================
3077 pci@11,700000/SUNW,qlc@0/fp@0,0 optimal     30     0      -     0     16
3078 pci@1,700000/SUNW,qlc@0/fp@0,0  optimal     30     0      -     0     20

rac02 # /etc/powermt display    
Symmetrix logical device count=30
CLARiiON logical device count=0
Hitachi logical device count=0
Invista logical device count=0
HP xp logical device count=0
Ess logical device count=0
HP HSx logical device count=0

==============================================================================

----- Host Bus Adapters ---------  ------ I/O Paths -----  ------ Stats ------
###  HW Path                       Summary   Total   Dead  IO/Sec Q-IOs Errors

==============================================================================
3077 pci@1,700000/SUNW,qlc@0/fp@0,0  optimal     30     0      -     0     17
3078 pci@11,700000/SUNW,qlc@0/fp@0,0 optimal     30     0      -     0     17

La baie est de nouveau visible sur le serveur. Il est donc possible de relancer la resynchronisation des diskgroups ASM.

$  sqlplus / as sysasm

SQL*Plus: Release 11.2.0.3.0 Production on Tue Feb 12 18:23:36 2013
Copyright (c) 1982, 2011, Oracle.  All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options

SQL> alter diskgroup data_dg1 online all;
Diskgroup altered.
SQL> alter diskgroup systemdg online all;
Diskgroup altered.
SQL> alter diskgroup redo_dg1 online all;
Diskgroup altered.
SQL> alter diskgroup flash_dg1 online all;
Diskgroup altered.
SQL> alter diskgroup dump_dg online all;
Diskgroup altered.

SQL> set lines 120
SQL> select * from v$asm_operation;

GROUP_NUMBER OPERA STAT POWER ACTUAL SOFAR EST_WORK EST_RATE EST_MINUTES
------------ ----- ---- ----- ------ ----- -------- -------- -----------
ERROR_CODE
--------------------------------------------
           2 ONLIN RUN      1      0     0        0        0           0

La resynchronisation semble être terminée et pourtant.

$ asmcmd lsdsk -pk
 Name           Failgroup     … Mount_Stat … Mode_Stat State  Path
 DATA_DG1_0000  DATA_DG1_FG1  … CACHED     …   ONLINE  NORMAL 
 DATA_DG1_0001  DATA_DG1_FG1  … CACHED     …   ONLINE  NORMAL 
 DUMP_DG_0002   DUMP_DG_FG1   … CACHED     …   SYNCING NORMAL 
 DUMP_DG_0009   DUMP_DG_FG1   … CACHED     …   SYNCING NORMAL 
 SYSTEMDG_0000  SYSTEMDG_0000 … CACHED     …   ONLINE  NORMAL 
 DUMP_DG_0000   DUMP_DG_FG1   … CACHED     …   SYNCING NORMAL 
 DUMP_DG_0001   DUMP_DG_FG1   … CACHED     …   SYNCING NORMAL 
 DUMP_DG_0008   DUMP_DG_FG1   … CACHED     …   SYNCING NORMAL 
 DUMP_DG_0003   DUMP_DG_FG1   … CACHED     …   SYNCING NORMAL 
 REDO_DG1_0000  REDO_DG1_FG1  … CACHED     …   ONLINE  NORMAL 
 FLASH_DG1_0000 FLASH_DG1_FG1 … CACHED     …   ONLINE  NORMAL 
 DATA_DG1_0002  DATA_DG1_FG2  … CACHED     …   ONLINE  NORMAL  …
 DATA_DG1_0003  DATA_DG1_FG2  … CACHED     …   ONLINE  NORMAL  …
 DUMP_DG_0004   DUMP_DG_FG2   … CACHED     …   ONLINE  NORMAL  …
 DUMP_DG_0005   DUMP_DG_FG2   … CACHED     …   ONLINE  NORMAL  …
 DUMP_DG_0006   DUMP_DG_FG2   … CACHED     …   ONLINE  NORMAL  …
 DUMP_DG_0007   DUMP_DG_FG2   … CACHED     …   ONLINE  NORMAL  …
 DUMP_DG_0010   DUMP_DG_FG2   … CACHED     …   ONLINE  NORMAL  …
 DUMP_DG_0011   DUMP_DG_FG2   … CACHED     …   ONLINE  NORMAL  …
 FLASH_DG1_0001 FLASH_DG1_FG2 … CACHED     …   ONLINE  NORMAL  …
 REDO_DG1_0001  REDO_DG1_FG2  … CACHED     …   ONLINE  NORMAL  …
 SYSTEMDG_0004  SYSTEMDG_0004 … CACHED     …   ONLINE  NORMAL  …
 SYSTEMDG_0001  SYSTEMDG_000  … CACHED     …   ONLINE  NORMAL  … 

Bizarrement le diskgroup ASM dump_dg est encore en cours de synchronisation !? Que se passe-t-il ? Vu qu'il s'agit d'une recette, il n'y a eu aucune activité utilisateurs, ce qui implique aucune modification des données. La resynchronisation entre les deux baies doit être immédiate avec le fasresync. Fonctionne-t-il correctement ?

Regardons ce qui se passe dans les logs de l’ASM.

$ cd /oracle/base/diag/asm/+asm/+ASM1/trace
$ tail -f alert_+ASM1.log
Tue Feb 12 18:38:55 2013
NOTE: successfully read ACD block gn=2 blk=10752 via retry read
Errors in file /oracle/base/diag/asm/+asm/+ASM1/trace/+ASM1_lgwr_3524.trc:
ORA-15062: ASM disk is globally closed
NOTE: successfully read ACD block gn=2 blk=10752 via retry read
Errors in file /oracle/base/diag/asm/+asm/+ASM1/trace/+ASM1_lgwr_3524.trc:
ORA-15062: ASM disk is globally closed
NOTE: successfully read ACD block gn=2 blk=10752 via retry read
Errors in file /oracle/base/diag/asm/+asm/+ASM1/trace/+ASM1_lgwr_3524.trc:
ORA-15062: ASM disk is globally closed
NOTE: successfully read ACD block gn=2 blk=10752 via retry read
Errors in file /oracle/base/diag/asm/+asm/+ASM1/trace/+ASM1_lgwr_3524.trc:
ORA-15062: ASM disk is globally closed
[…]

$ cat /oracle/base/diag/asm/+asm/+ASM1/trace/+ASM1_lgwr_3524.trc
Trace file /oracle/base/diag/asm/+asm/+ASM1/trace/+ASM1_lgwr_3524.trc
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
ORACLE_HOME = /oracle/product/11.2.0/grid
System name:    SunOS
Node name:      rac01
Release:        5.10
Version:        Generic_147440-26
Machine:        sun4u
Instance name: +ASM1
Redo thread mounted by this instance: 0 <none>
Oracle process number: 15
Unix process pid: 3524, image: oracle@rac01 (LGWR)

*** 2013-02-12 18:10:12.384
*** SESSION ID:(931.1) 2013-02-12 18:10:12.385
*** CLIENT ID:() 2013-02-12 18:10:12.385
*** SERVICE NAME:() 2013-02-12 18:10:12.385
*** MODULE NAME:() 2013-02-12 18:10:12.385
*** ACTION NAME:() 2013-02-12 18:10:12.385
[…]
*** 2013-02-12 18:10:20.264
NOTE: successfully read ACD block gn=1 blk=10752 via retry read
ORA-15062: ASM disk is globally closed
*** 2013-02-12 18:11:11.444
NOTE: successfully read ACD block gn=1 blk=10752 via retry read
ORA-15062: ASM disk is globally closed
NOTE: successfully read ACD block gn=2 blk=10752 via retry read
ORA-15062: ASM disk is globally closed
NOTE: successfully read ACD block gn=4 blk=10752 via retry read
ORA-15062: ASM disk is globally closed
NOTE: successfully read ACD block gn=5 blk=0 via retry read
ORA-15062: ASM disk is globally closed
[…]

D'après les logs Oracle, il semble toujours y avoir des opérations ASM. Vérifions les propriétés de ce diskgroup ASM, peut-être y-a-t-il des différences.

$ asmcmd
ASMCMD> lsattr -l -G dump_dg
Name                     Value      
access_control.enabled   FALSE      
access_control.umask     066        
au_size                  1048576    
cell.smart_scan_capable  FALSE      
compatible.advm          11.2.0.0.0 
compatible.asm           11.2.0.0.0 
compatible.rdbms         11.2.0.0.0 
disk_repair_time         24h        
sector_size              512

En la comparant à un autre diskgroup ASM, il n’y a aucune différence. La seule différence est l’utilisation du volume en ACFS. Cherchons différemment...

Lancement d’un débat entre les DBA Oracle et moi : pourquoi le diskgroup se synchronise toujours au bout de 20 minutes ?

• Soit le diskgroup ASM n’utilise pas la fonctionnalité fastresync (si ACFS) ?
• Soit un bug ASM – ACFS ?
• Y-a-t-il encore des I/O ?

On va mettre tout le monde d’accord avec Dtrace ! Un petit on-liners Dtrace pour trouver le pid effectuant les demandes d’écriture.

# dtrace –n 'syscall::write:entry /execname == “oracle”/ \
{ @[pid] = count(); }’
^C
[…]
   3572       1080

Effectuons un petit oradebug sur ce PID

$ sqlplus / as sysasm

SQL*Plus: Release 11.2.0.3.0 Production on Tue Feb 12 18:23:36 2013
Copyright (c) 1982, 2011, Oracle.  All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options

SQL> oradebug setospid 3572
Oracle pid: 27, Unix process pid: 3572, image: oracle@rac01 (ASMB)
SQL> oradebug unlimit
Statement processed.
SQL> oradebug EVENT 10046 TRACE NAME CONTEXT FOREVER, LEVEL 12
Statement processed.
SQL> oradebug tracefile_name
/oracle/base/diag/asm/+asm/+ASM1/trace/+ASM1_asmb_3572.trc
SQL> ! tail -f /oracle/base/diag/asm/+asm/+ASM1/trace/+ASM1_asmb_3572.trc
WAIT #0: nam='ASM file metadata operation' ela= 62 msgop=4 locn=0 p3=0 obj#=-1 tim=443471434898
WAIT #0: nam='ASM file metadata operation' ela= 61 msgop=4 locn=0 p3=0 obj#=-1 tim=443471435027
WAIT #0: nam='ASM file metadata operation' ela= 109 msgop=4 locn=0 p3=0 obj#=-1 tim=443471435205
WAIT #0: nam='ASM file metadata operation' ela= 64 msgop=4 locn=0 p3=0 obj#=-1 tim=443471435338
WAIT #0: nam='ASM background timer' ela= 4175 p1=0 p2=0 p3=0 obj#=-1 tim=443471439572
WAIT #0: nam='ASM file metadata operation' ela= 1 msgop=0 locn=3 p3=0 obj#=-1 tim=443471439636
WAIT #0: nam='ASM file metadata operation' ela= 64 msgop=4 locn=0 p3=0 obj#=-1 tim=443471439751
WAIT #0: nam='ASM file metadata operation' ela= 61 msgop=4 locn=0 p3=0 obj#=-1 tim=443471439967
WAIT #0: nam='ASM file metadata operation' ela= 59 msgop=4 locn=0 p3=0 obj#=-1 tim=443471440102
WAIT #0: nam='ASM file metadata operation' ela= 62 msgop=4 locn=0 p3=0 obj#=-1 tim=443471440231
WAIT #0: nam='ASM background timer' ela= 81 p1=0 p2=0 p3=0 obj#=-1 tim=443471440388
WAIT #0: nam='ASM file metadata operation' ela= 1 msgop=0 locn=3 p3=0 obj#=-1 tim=443471440444
WAIT #0: nam='ASM file metadata operation' ela= 63 msgop=4 locn=0 p3=0 obj#=-1 tim=443471440552
WAIT #0: nam='ASM file metadata operation' ela= 60 msgop=4 locn=0 p3=0 obj#=-1 tim=443471440681
WAIT #0: nam='ASM file metadata operation' ela= 60 msgop=4 locn=0 p3=0 obj#=-1 tim=443471440801
WAIT #0: nam='ASM file metadata operation' ela= 60 msgop=4 locn=0 p3=0 obj#=-1 tim=443471440937
WAIT #0: nam='ASM background timer' ela= 74 p1=0 p2=0 p3=0 obj#=-1 tim=443471441070
WAIT #0: nam='ASM file metadata operation' ela= 0 msgop=0 locn=3 p3=0 obj#=-1 tim=443471441119
WAIT #0: nam='ASM file metadata operation' ela= 60 msgop=4 locn=0 p3=0 obj#=-1 tim=443471441223
WAIT #0: nam='ASM file metadata operation' ela= 60 msgop=4 locn=0 p3=0 obj#=-1 tim=443471441347
WAIT #0: nam='ASM file metadata operation' ela= 60 msgop=4 locn=0 p3=0 obj#=-1 tim=443471441472
WAIT #0: nam='ASM file metadata operation' ela= 76 msgop=4 locn=0 p3=0 obj#=-1 tim=443471441614
[…]

SQL> oradebug EVENT 10046 TRACE NAME CONTEXT OFF
Statement processed.

Nouvelle discussion : il s’agit d’opérations uniquement sur les metadas (synchronisation avec « fastresync »). Alors pourquoi le statut dans ASM est toujours en « SYNCING ». Il n’y a eu aucune mise jours pendant le test !!

$ asmcmd lsdsk –pk | grep SYNCING 
 DUMP_DG_0002   DUMP_DG_FG1   … CACHED     …   SYNCING NORMAL 
 DUMP_DG_0009   DUMP_DG_FG1   … CACHED     …   SYNCING NORMAL 
 DUMP_DG_0000   DUMP_DG_FG1   … CACHED     …   SYNCING NORMAL 
 DUMP_DG_0001   DUMP_DG_FG1   … CACHED     …   SYNCING NORMAL 
 DUMP_DG_0008   DUMP_DG_FG1   … CACHED     …   SYNCING NORMAL 
 DUMP_DG_0003   DUMP_DG_FG1   … CACHED     …   SYNCING NORMAL 

Nouvelle discussion : y-a-t-il vraiment des IO ? N'y a-t-il pas simplement un problème d'affichage du résultat de la commande ? Pour les DBA non, pour moi si. Vérifions.

$ asmcmd
ASMCMD> iostat --io -G dump_dg 5
Group_Name  Dsk_Name      Reads  Writes 
DUMP_DG     DUMP_DG_0000  456    67329  
DUMP_DG     DUMP_DG_0001  223    67612  
DUMP_DG     DUMP_DG_0002  223    68516  
DUMP_DG     DUMP_DG_0003  223    66326  
DUMP_DG     DUMP_DG_0004  66887  106845 
DUMP_DG     DUMP_DG_0005  59264  7561   
DUMP_DG     DUMP_DG_0006  58953  6996   
DUMP_DG     DUMP_DG_0007  73000  10251  
DUMP_DG     DUMP_DG_0008  223    66383  
DUMP_DG     DUMP_DG_0009  223    66820
DUMP_DG     DUMP_DG_0010  66421  8176   
DUMP_DG     DUMP_DG_0011  58860  8546   

Group_Name  Dsk_Name      Reads  Writes 
DUMP_DG     DUMP_DG_0000  0.00   32.20  
DUMP_DG     DUMP_DG_0001  0.00   43.80  
DUMP_DG     DUMP_DG_0002  0.00   32.60  
DUMP_DG     DUMP_DG_0003  0.00   35.20  
DUMP_DG     DUMP_DG_0004  35.20  0.20   
DUMP_DG     DUMP_DG_0005  30.40  1.00   
DUMP_DG     DUMP_DG_0006  30.40  0.00   
DUMP_DG     DUMP_DG_0007  28.80  11.40  
DUMP_DG     DUMP_DG_0008  0.00   39.00  
DUMP_DG     DUMP_DG_0009  0.00   27.40  
DUMP_DG     DUMP_DG_0010  27.40  0.20   
DUMP_DG     DUMP_DG_0011  33.60  11.80

Il semble que oui d'un point de vue Oracle. Et d’un point de vue système ?

# iostat –Mxnzt 5
[…]
    0.0  14.4   0.0  14.4  0.0  0.2  0.0  14.5   0   5 c1t5000097408231998d7
    0.0  26.4   0.0  14.5  0.0  0.2  0.0   8.5   0   6 c1t5000097408231998d6
    0.0  14.4   0.0  14.4  0.0  0.2  0.0  15.4   0   6 c1t5000097408231998d5
    0.0  16.2   0.0  16.0  0.0  0.2  0.0  15.0   0   6 c1t5000097408231998d4
   16.0  11.6  16.0   0.0  0.0  0.3  0.0  10.9   0   9 c1t500009740823ED98d7
   16.0   0.0  16.0   0.0  0.0  0.2  0.0  11.4   0   5 c1t500009740823ED98d6
   12.8   0.8  12.8   0.0  0.0  0.2  0.0  17.4   0   7 c1t500009740823ED98d5
   16.0   0.2  16.0   0.0  0.0  0.2  0.0  10.5   0   4 c1t500009740823ED98d4
   16.0  11.6  16.0   0.0  0.0  0.3  0.0  10.7   0   9 c3t500009740823EDA4d7
   16.0   0.0  16.0   0.0  0.0  0.2  0.0  11.4   0   5 c3t500009740823EDA4d6
   12.8   0.2  12.8   0.0  0.0  0.2  0.0  17.5   0   7 c3t500009740823EDA4d5
   16.0   0.2  16.0   0.0  0.0  0.2  0.0  10.4   0   4 c3t500009740823EDA4d4
    0.0  14.4   0.0  14.4  0.0  0.2  0.0  14.5   0   5 c3t50000974082319A4d7
    0.0  26.2   0.0  14.5  0.0  0.2  0.0   8.5   0   6 c3t50000974082319A4d6
    0.0  14.4   0.0  14.4  0.0  0.2  0.0  15.3   0   6 c3t50000974082319A4d5
    0.0  16.2   0.0  16.0  0.0  0.2  0.0  15.0   0   6 c3t50000974082319A4d4
   15.2   0.0  15.2   0.0  0.0  0.3  0.0  18.0   0   8 c1t500009740823ED98d13
   16.0   0.4  16.0   0.0  0.0  0.3  0.0  16.7   0   8 c1t500009740823ED98d12
   15.2   0.0  15.2   0.0  0.0  0.3  0.0  17.6   0   8 c3t500009740823EDA4d13
   16.4   0.0  16.0   0.0  0.0  0.3  0.0  17.9   0   9 c3t500009740823EDA4d12
    0.0  16.2   0.0  16.0  0.0  0.2  0.0  15.0   0   6 c3t50000974082319A4d13
    0.0  17.8   0.0  17.6  0.0  0.3  0.0  14.4   0   7 c3t50000974082319A4d12
    0.0  16.2   0.0  16.0  0.0  0.3  0.0  15.5   0   6 c1t5000097408231998d13
    0.0  17.8   0.0  17.6  0.0  0.3  0.0  14.9   0   7 c1t5000097408231998d12
[…]

Effectivement il y a bien des IO physiques. (pour information, les luns ci-dessus appartiennent bien au diskgroup ASM dump_dg).

Pourquoi la resynchronisation dure aussi longtemps alors que vue d'Oracle, on écrit uniquement les « metadatas » !? C’est vraiment bizarre. Vérifions avec Dtrace ce qui se passe vraiment lors des lectures et écritures.

# dtrace -n 'io:::start { @[pid] = count(); }'
dtrace: description 'io:::start ' matched 6 probes
^C 

    13029                4
    13184                5
    13242                8
     2637               27
    13031              161
    13763             1256

# dtrace -n 'io:::start /pid == 13763/ \
{ @["disk I/O operation", args[1]->dev_pathname, \
args[0]->b_flags & B_READ ? "R" : "W"] \
=  quantize(args[0]->b_lblkno); }’
dtrace: description 'io:::start ' matched 6 probes
^C  

  disk I/O operation           /devices/pseudo/emcp@96:g,blk         R                                                     

           value  ------------- Distribution ------------- count   
        33554432 |                                         0           
        67108864 |@                                        8           
       134217728 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@  288     
       268435456 |                                         0                        

  disk I/O operation           /devices/pseudo/emcp@24:g,blk         W                                                    

           value  ------------- Distribution ------------- count   
        33554432 |                                         0           
        67108864 |@@@@                                     32      
       134217728 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@     280     
       268435456 |                                         0           

  disk I/O operation          /devices/pseudo/emcp@25:g,blk          W                                                    

         value  ------------- Distribution ------------- count   
        33554432 |                                         0           
        67108864 |@@@                                      24          
       134217728 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@    288     
       268435456 |                                         0           

  disk I/O operation         /devices/pseudo/emcp@94:g,blk           R                                                    

           value  ------------- Distribution ------------- count   
        33554432 |                                         0           
        67108864 |@                                        8           
       134217728 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@  296     
       268435456 |                                         0           

  disk I/O operation         /devices/pseudo/emcp@26:g,blk           W                                                    
           value  ------------- Distribution ------------- count   
        33554432 |                                         0           
        67108864 |@@@                                      24          
       134217728 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@    296     
       268435456 |                                         0           

  disk I/O operation         /devices/pseudo/emcp@27:g,blk           W                                                     

           value  ------------- Distribution ------------- count    
        33554432 |                                         0            
        67108864 |@@@                                      24          
       134217728 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@    296     
       268435456 |                                         0        

  disk I/O operation        /devices/pseudo/emcp@613:g,blk           W                                                    

         value  ------------- Distribution ------------- count   
         67108864 |                                         0           
       134217728 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 312     
       268435456 |                                         0           

  disk I/O operation       /devices/pseudo/emcp@913:g,blk            R                                                    

           value  ------------- Distribution ------------- count   
        67108864 |                                         0           
       134217728 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 320     
       268435456 |                                         0           

  disk I/O operation      /devices/pseudo/emcp@97:g,blk              R                                                    

           value  ------------- Distribution ------------- count  
        33554432 |                                         0           
        67108864 |@@                                       16          
       134217728 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@   320     
       268435456 |                                         0           

  disk I/O operation     /devices/pseudo/emcp@95:g,blk               R                                                     

           value  ------------- Distribution ------------- count  
        33554432 |                                         0          
        67108864 |@                                        8           
       134217728 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@  328     
       268435456 |                                         0               

  disk I/O operation     /devices/pseudo/emcp@912:g,blk               R                                                     

           value  ------------- Distribution ------------- count   
        67108864 |                                         0           
       134217728 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 344     
       268435456 |                                         0           

  disk I/O operation     /devices/pseudo/emcp@612:g,blk               W                                                    

           value  ------------- Distribution ------------- count   
        67108864 |                                         0           
       134217728 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 360     
       268435456 |                                         0           

On lit et on écrit toujours sur les mêmes blocks !!

Si nous résumons : l’utilisation du « fastresync » sur un diskgroup ASM ayant un ou des volume(s) ACFS semble poser problème : on synchronise bien les metadatas mais cette opération « boucle » beaucoup trop longtemps (plus d'une heure de resynchronisation alors qu'aucune modification a eu lieu). Un petit call chez Oracle pour avoir quelques précisons…

Je remercie David et Yahya (encore) pour notre analyse commune sur ce problème. “Hardware and software work together” : le slogan suivant s’applique particulièrement bien à ce contexte.