The connect, ping, disconnect trio result in SEGV

[ancientwizard]

I have a PERL script that reproduces this SEGV issue. Strictly speaking the SEGV still happens if the ping step is skipped.

Also I have C program that has roughly the same trio but it does not cause SEGV's. It's not definitive but suggests the issue isn't with the OS or Oracle instant client but with Perl or the DBD:Oracle. This C program has been compiled in the office on RHEL8 & RHEL9. It runs as expected without error as long as you have an Oracle DB and a login

You may need to change the Make file depending how you install oracle instantclient; I used Oracle 23.7.
$ export ORA_SCHEMA=TESTUSER
$ export ORACLE_HOME=/usr/lib/oracle/23/client64
$ export ORA_DBNAME=ORA19C (my server is 19c so it is at work also)
$ export ORACLE_SID=orcl
$ export ORA_PASSWD=BLAHBLAH
$ make
$ ./db-thread -t 5
segv-debug-c.tar.gz
UPDATE: this "C" program has an instability however it's my fault (and CoPilot) I have an updated version that corrects an oversight and I have found it to be rock solid. I have a repo in my space with the fix.

The Perl script is provided below. It has been used with all final versions of Perl from 5.30.3 to 5.40.2; same SEGV results. RHEL8/9. At home I'm using CentOS and Ubuntu.

Confession & Issues

• I don't normally run the tests that are built into this module; not for many years
• My apps test set is HUGE and if something is hinky with any feature it uses; it's not likely to be missed.
• I figured it might be time to rethink this position. I ran into some issues.
• The tests all assume an EMPTY/BLANK password. My DB is 19c and even with the help of AI I was unable to create an account with an empty password.
• While it's possible to create it with a password of " " (single space) login was still impossible.
• I was forced to put a password on the account and update the test scripting to use a hard coded password. It's not like the account is going be used in PRODUCTION or be given any additional GRANTS it won't need. It's just my test rig at home running VM's
• The tests should make noise telling the reader A) what vars to set B) what hard coded password to use. OR the recipe to make Oracle submit for the test user alone. I suspect it written somewhere, I didn't easily find it so I read the code. Even the AI guessed wrong.
• a couple of the built-in test scripts barf out redefined Perl functions; if it's expected its just noise and could hide something important. I see they are in DBI :( otherwise I would have disabled the warning.

This Seems Ugly

t/rt85886.t ............... 1/2 
#   Failed test 'ivok'
#   at t/rt85886.t line 49.
SV = PV(0x5876a916e9c0) at 0x5876a9318ab0
  REFCNT = 1
  FLAGS = (POK,pPOK)
  PV = 0x5876a9330ee0 "1"\0
  CUR = 1
  LEN = 16
# Looks like you failed 1 test of 2.
t/rt85886.t ............... Dubious, test returned 1 (wstat 256, 0x100)

- And if run enough times your own test SEGV's

t/15threads.t           (Wstat: 139 (Signal: SEGV, dumped core) Tests: 0 Failed: 0)
  Non-zero wait status: 139
  Parse errors: No plan found in TAP output

- And this is random

t/16cached.t .............. Can't call method "prepare" on an undefined value at /banktools/trustguardian/git/DBD-Oracle-AW/t/cache2.pl line 21.
t/16cached.t .............. 1/? 
#   Failed test 'clean termination with multiple cached connections'
#   at t/16cached.t line 26.
# Looks like you failed 1 test of 1.
t/16cached.t .............. Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/1 subtests 

[ancientwizard]

The Perl test script. You'll need a schema to connect to (obvious).

$ export ORA_SCHEMA=TESTUSER
$ export ORA_SOURCE=dbi:Oracle:ORA19C
$ export ORA_PASSWD=BLAHBLAH
$ ./t_A_segv.pl

t_A_segv.pl.txt
yes you'll need to rename after download

[twata1]

FYI:
The script appears to me to reproduce the issue on Windows as well.
(For Windows, a message box with the content "Perl interpreter has stopped working" is displayed.)

I used Oracle 18.4.
I tested using Strawberry Perl 5.30.3, 5.40.0.

[twata1]

With regard to Windows, the issue appears to have started with DBD::Oracle 1.90_1.
The issue does not seem to occur in DBD::Oracle 1.83.

[ancientwizard]

I'm no expert with Perl internals NOR OCI for that matter. I do however find this oddly and incorrect, so I believe.

I added code into the two places where the OCI ENV is being initialized. I had expected that would be done once globally and thread make their own handles and such. But per thread is also okay for the OCIEnv*Create.

I found it odd that it gets called twice. First as OCI_DEFAULT; though here I changed it to OCI_THREADED which is what is expected by OCI when it is possible for more than one thread to do DB actions. My "C" test code proves that. Using OCI_DEFAULT completely tosses it cookies. The ..., 1, in the first call is my having used OCI_THREADED, it was a zero. I believe the second call of , 3, is an OR of OCI_THREADED and something else I didn't look up yet. Perhaps this behavior is okay. I suppose I could test that in my "C" code. Okay no harm no foul, I guess. An unnecessary call at best. Called twice in my "C" test and runs without issue.

# CONNECT-ENTER 11
DBD::Oracle::dbd_dr_mng()    ->  OCIEnvNlsCreate(..., 1, ...) called by thread 98217518164592
DBD::Oracle::new_envhp_box() ->  OCIEnvNlsCreate(..., 3, ...) called by thread 98217518164592
# CONNECT-EXIT  11
# CONNECT-ENTER 12
DBD::Oracle::dbd_dr_mng()    ->  OCIEnvNlsCreate(..., 1, ...) called by thread 98217523161600
DBD::Oracle::new_envhp_box() ->  OCIEnvNlsCreate(..., 3, ...) called by thread 98217523161600
# CONNECT-EXIT  12
# CONNECT-ENTER 13
DBD::Oracle::dbd_dr_mng()    ->  OCIEnvNlsCreate(..., 1, ...) called by thread 98217534734960
DBD::Oracle::new_envhp_box() ->  OCIEnvNlsCreate(..., 3, ...) called by thread 98217534734960
# CONNECT-EXIT  13
# CONNECT-ENTER 14
DBD::Oracle::dbd_dr_mng()    ->  OCIEnvNlsCreate(..., 1, ...) called by thread 98217542765280
DBD::Oracle::new_envhp_box() ->  OCIEnvNlsCreate(..., 3, ...) called by thread 98217542765280
# CONNECT-EXIT  14
# CONNECT-ENTER 15
DBD::Oracle::dbd_dr_mng()    ->  OCIEnvNlsCreate(..., 1, ...) called by thread 98217527623536
DBD::Oracle::new_envhp_box() ->  OCIEnvNlsCreate(..., 3, ...) called by thread 98217527623536
# CONNECT-EXIT  15
ok 30 -   ALL->connected
# DISCONNECT-ENTER 11
# DISCONNECT-EXIT  11
# DISCONNECT-ENTER 12
# DISCONNECT-EXIT  12
# DISCONNECT-ENTER 13
# DISCONNECT-EXIT  13
# DISCONNECT-ENTER 14
# DISCONNECT-EXIT  14
# DISCONNECT-ENTER 15
Segmentation fault (core dumped)

When the issue is during the disconnect; very high percentage of the time it's always the last thread doing it's disconnect that barfs.

   // Initialize OCI environment
//  if (OCIEnvCreate(&env, OCI_THREADED, NULL, NULL, NULL, NULL, 0, NULL) != OCI_SUCCESS) {
    // DBD::Oracle uses this function. No harm no foul!
    if (OCIEnvNlsCreate(&env, OCI_DEFAULT, 0, NULL, NULL, NULL, 0, NULL, 0, 0) != OCI_SUCCESS) {
        snprintf(status->error_message, sizeof(status->error_message), "Failed to initialize OCI environment");
        status->connection_status = -1;
        pthread_exit(NULL);
    }

    // DBD::Oracle uses this function. Works here OK
    if (OCIEnvNlsCreate(&env, OCI_THREADED, 0, NULL, NULL, NULL, 0, NULL, 0, 0) != OCI_SUCCESS) {
        snprintf(status->error_message, sizeof(status->error_message), "Failed to initialize OCI environment");
        status->connection_status = -1;
        pthread_exit(NULL);
    }

[twata1]

Maybe the issue started with #150, but I'm not familiar with Perl XS or OCI, so I don't know the cause.

[ancientwizard]

More clues:

• The test perl script only SEGV's when
  -- at least three or more threads connect to the DB at the same time
  -- the last thread SEGV's during the disconnect.
• 1-2 threads can come and go as they please without issue.

In the file dbdcnx.c is found the function cnx_detach(); where you will find these lines.
(I removed a few newlines here, for easier reading I hope)

    // OCISessionEnd SEGV's
    (void)OCISessionEnd(  imp_dbh->svchp, imp_dbh->errhp, imp_dbh->seshp, OCI_DEFAULT );
    (void)OCIServerDetach(imp_dbh->srvhp, imp_dbh->errhp, OCI_DEFAULT);

    // According to AI this is considered the long hand for a single call of OCILogOff
    //   which ... BTW isn't doing a SEGV when I replace the above with ... because the AI was passing
    //   the wrong handle and the status was a -2 == BAD ARG (can you say memory leak :)
    //   Using the correct handle(pointer) once again it's making a SEGV - sigh!
    sword status;
    status = OCILogoff(imp_dbh->svchp, imp_dbh->errhp);
    if(status != OCI_SUCCESS)
        PerlIO_printf(PerlIO_stderr(), "OCILogoff(%p,%p) returned = %u)", imp_dbh->svchp, imp_dbh->errhp, status);

This Implies there is something wrong with the ->svchp handle

[twata1]

DBD-Oracle/dbdcnx.c

Lines 1013 to 1018 in 1cbc9a9

	#endif
	(void)OCISessionEnd(
	imp_dbh->svchp, imp_dbh->errhp, imp_dbh->seshp, OCI_DEFAULT
	);
	(void)OCIServerDetach(imp_dbh->srvhp, imp_dbh->errhp, OCI_DEFAULT);
	#ifdef ORA_OCI_112

According to 3.3.4 About Terminating the Application , I think maybe step 3 and step 4 are needed.

[ancientwizard]

Correct and they are there. See the next function. When reading this code it's important remember this is inside Perl. Closing a session etc are a caller induced behavior while memory cleanup is triggered by Perl DESTROY.

I'm currently looking for all the places this ctx handle gets touched in the head.

[ancientwizard]

The SEGV always takes place on the last thread to connect in a set of threads of three(3) or more. The last one SEGV's on OCISessionEnd(). A good clue but the reason eludes me.

What I've tried to test (confirm its behavior) ; by doing deliberate bad stuff!

• Calling OCIHandleFree( sesson handle ) while active results in a harmless error code -2
  (though Perl being slower, that memory could have been reused and now has new contents foreign to the SessionEnd!!!)
• Calling OCISessionEnd() twice make no error, SUCCESS on both calls!

https://github.com/ancientwizard/Oracle-Linux-SEGV/blob/main/segv-debug-c/db-thread.c

[ancientwizard]

I have also confirmed that DBD::Oracle 1.80 - 1.83 don't SEGV; my test wasn't exhaustive. I'll do that at work tomorrow using the real test; it puts the hammer down, just like THOR!

[ancientwizard]

I have submitted a pull request. Includes "a" bug fix for the SEGv. I also did some housekeeping. Cherry pick what you like. At the very least include the bug fix and the new test.
It would be very keen if this could get approved and published to CPAN soon. I really could use this this fix there. - Thanks

[ancientwizard]

Please note that the pull request is incomplete. It fixes this specific SEGV issue but exposes that there are more bugs in the module. I have a few ideas but until I have a test to reproduce I won't speculate here.

[twata1]

I tested #187 under similar conditions as when I was testing DBD::Oracle 1.90 on Windows.
Here is a summary of the tests:

• The following warning appears during perl Makefile.PL.
  Warning: the following files are missing in your kit:

• The environment variable TNS_ADMIN appears to be required for testing with Oracle.

• There is no which command in Windows (the alternative would be the where command).

• Connection to Oracle failed, causing ORA-28000 (The account is locked; login denied.).

• Eventually t/90-segv.t hung up.

Here you will find a series of logs (passwords etc. are masked).

results-perl-Makefile.PL-on-Strawberry-Perl-5.30.3.log.txt
results-gmake-on-Strawberry-Perl-5.30.3.log.txt
take1-results-gmake-test-on-Strawberry-Perl-5.30.3.log.txt
take2-results-gmake-test-on-Strawberry-Perl-5.30.3.log.txt

Thank you,

[ancientwizard]

I completely agree I made zero effort to make this Windows compatible, and it certainly needs to be adjusted accordingly. I don't readily have Windows for such work. As for passwords etc. I mentioned that any modern oracle version won't allow blank passwords. There may be a way to hammer 🔨 it into submission but I grew tired of trying. I see in the logs no attempt on your part was made to allow most of the DB test to actually run. there is a t/lib/ folder with helpers for making DB connections and skipping tests. As of yet I didn't make use of them until such time a few issues regarding logins can be discussed and a solution agreed upon. Regardless until you allow the tests to connect the testing provides no value. But I suspect you know that.