Friday, January 8, 2021

Firebird Embedded in a sandboxed MacOS App

For those who might not be aware, Firebird on MacOS is now relocatable, in that you don't necessarily have to install it as a Framework, this also means that you can create an embedded version out of the current installer.

A typical structure would look like this

|-- firebird.conf
|-- firebird.msg
|-- intl
| |-- fbintl.conf
| `-- libfbintl.dylib
|-- lib
| |-- libfbclient.dylib
| |-- libib_util.dylib
| |-- libicudata.dylib
| |-- libicui18n.dylib
| |-- libicuuc.dylib
|`-- plugins
    `-- libEngine12.dylib

And the firebird.conf file would then typically be amended for the following

Providers = Engine12
ServerMode = Classic
 
For the last few weeks Alex and I (along with a Firebird user) have been working on getting Firebird embedded to work properly in a sandboxed app that can then be deployed on the App Store...
 
To do this we had to solve issues with temp files, the use of inter process communications by the
Firebird lock manager and the location of the Firebird log file.

Note: adding LSEnvironment to the plist file defining new locations for FIREBIRD_TEMP and FIREBIRD_LOCK does not work.

<key>LSEnvironment</key>
<dict>
    <key>ENV_VAR</key>
    <string>value</string>
</dict>
 
So the first issue to be addressed was the following

macosx    Wed Dec  2 15:33:57 2020
    ConfigStorage: Cannot initialize the shared memory region
    Can not access lock files directory /tmp/firebird.tmp.YDzrhQ

It seems thats sandboxed Apps cannot to write into /tmp at all, they have to use their own /tmp-folder. We can detect whether we are running in a sandboxed environment using the following calls to the Mac security subsystem

task = SecTaskCreateFromSelf(nil),        
value = SecTaskCopyValueForEntitlement(task, "com.apple.security.app-sandbox" as CFString, nil),

Now if we now know if we are sandboxed we can use  NSTemporaryDirectory() for the location of the relevant Firebird temporary files.

Having fixed that issue, it was time to move onto the lock manager.
 
[FireDAC][Phys][FB]lock manager error.)
 
macosx    Thu Dec 17 17:37:00 2020
    event_init()
     operating system directive semctl failed
     Operation not permitted
 
According to the Apple sandbox guide restricting IPC (Inter Process Communication) is also part of MacOS sandbox implementation. After some serious head scratching we tried the following test
 
#include <stdio.h>
#include <pthread.h>
 
#define ER(x) { int tmpState = (x); if (tmpState) { printf("Failed %s error %d\n", #x, tmpState); } }
 
int main() {
 
    pthread_mutex_t mutex;
     pthread_mutexattr_t mutexattr;
 
     ER(pthread_mutexattr_init(&mutexattr));
     ER(pthread_mutexattr_setpshared(&mutexattr, PTHREAD_PROCESS_SHARED));
     ER(pthread_mutex_init(&mutex, &mutexattr));
     ER(pthread_mutex_lock(&mutex));
     ER(pthread_mutex_unlock(&mutex));
     ER(pthread_mutex_destroy(&mutex));
 
c++ test.cpp -pthread
 
In case of failure it will print error messages, on success we should get nothing. We got nothing.

For many years Firebird worked with system V IPC on MacOS which is different to many other *nixes where mutexes and conditional variables in shared memory were used instead. The use of system V IPC was caused by a lack of  for shared mutexes on MacOS when Firebird was originally ported to it. Currently (as was proved by test above) MacOS now supports such mutexes. Based on this it we decided to stop using system V IPC in Firebird and a perform a cleanup of the lock manager code as part of the effort to provide sandbox support. As a side effect Firebird should now run faster using shared mutexes.
 
At the same time as we worked on the lock manager issue we also had an issue with the firebird log file, we were not allowed to write to its normal default location by the sandbox. Because we now know we are running in a sandbox we can relocate the placement of the log file within utils.cpp
 
Supposedly using something like ~/Library/Application Support/Firebird/ should work
 
case Firebird::IConfigManager::DIR_LOG:
             s = "~/Library/Application Support/Firebird/";
             s += name;
             return s;
 
Unfortunately it seems this workaround was removed relatively recently forcing us to resort to putting the log file within the App itself ~/LibraryContainers/Company.App/Data/Library/Application Support/Company/
 
Changing the location of the log file is relatively easy but  although this allowed the App to run and resolved the issue, it was not an ideal solution since the location of the log file is now hard coded within Firebird and is App dependent and this would mean that every time somebody wanted an embedded Firebird for a sandboxed App they would have to change the path to the log file and recompile Firebird.
 
The solution was to do away with the Firebird log file and use the OSLog framework instead, and use this to capture any messages from the database engine, so if you are running embedded Firebird in a sandboxed App you can access the Firebird log messages using the Console or a terminal command like the following
 
log show --predicate 'eventMessage contains "macpro.home"' --start '2021-01-06 14:00:00' --end '2021-01-06 14:30:00' --info
 
where macpro.home is the name of the computer. You don't need the start and end, but it does help reduce the number of messages.
 
There is an added plus from using this. The OSLog framework is also supported on IOS 10+ and now that the App Store will accept dynamic libraries, this means that we should be able to compile an embedded version of Firebird for IOS that can also run sandboxed Apps. If anyone is interested in sponsoring this work please contact me.  

This work was sponsored by kiC Gesellschaft für Softwareentwicklung mbH.

Wednesday, September 11, 2019

Firebird 3 Embedded on MacOSX


I have finally managed to get around to preparing a mechanism for creating an embedded Firebird Framework on MacOSX (many thanks to the customer who sponsored the work). I will be committing the updated makefile embed.darwin shortly to B3_0_Release.
You can use the makefile to create a 32bit or 64 bit framework, depending on the build of Firebird you want.
Once you have a build of Firebird, you need to run make.platform.postfix, and then embed.darwin. The embeded framework is created in gen/Release/frameworks.

If you can't build Firebird from scratch you can download a copy of the embedded framework (currently Firebird 3.0.4) from IBPhoenix. (Approx 15mb)

32bit Embedded Framework
64bit Embedded Framework

Below is the embed.darwin makefile (updated 01-Oct-2019)

# Makefile script to generate an embedded Firebird Framework from a sucessful Firebird build
# To be run from the gen directory of a Firebird Release build
# Your application needs to be placed in the Resources/app directory.

    FBE=../gen/Release/frameworks/FirebirdEmbedded.framework
    BINLOC=../gen/Release/frameworks/FirebirdEmbedded.framework/Versions/A/Resources/bin
    LIBLOC=../gen/Release/frameworks/FirebirdEmbedded.framework/Versions/A
    INTLOC=../gen/Release/frameworks/FirebirdEmbedded.framework/Versions/A/Resources/intl
    PLULOC=../gen/Release/frameworks/FirebirdEmbedded.framework/Versions/A/Resources/plugins
    UTILOC=../gen/Release/frameworks/FirebirdEmbedded.framework/Versions/A/Libraries
    OLDPATH=/Library/Frameworks/Firebird.framework/Versions/A/Libraries

all:

    -$(RM) -rf $(FBE)
    mkdir -p $(FBE)/Versions/A/Resources
    mkdir -p $(FBE)/Versions/A/Resources/intl
    mkdir -p $(FBE)/Versions/A/Resources/plugins
    mkdir -p $(FBE)/Versions/A/Resources/bin
    mkdir -p $(FBE)/Versions/A/Resources/app
    mkdir -p $(FBE)/Versions/A/Headers
    mkdir -p $(FBE)/Versions/A/Libraries
    ln -s Versions/Current/Headers $(FBE)/Headers
    ln -s Versions/Current/Resources $(FBE)/Resources
    ln -s Versions/Current/Libraries $(FBE)/Libraries
    ln -s A $(FBE)/Versions/Current

    cp ../gen/Release/firebird/firebird.conf $(FBE)/Versions/A/Resources/firebird.conf
    cp ../gen/Release/firebird/plugins.conf $(FBE)/Versions/A/Resources/plugins.conf
    cp ../gen/Release/firebird/firebird.msg $(FBE)/Versions/A/Resources/firebird.msg
    cp ../gen/Release/firebird/lib/libfbclient.dylib $(FBE)/Versions/A/libfbclient.dylib
    cp ../gen/Release/firebird/plugins/libEngine12.dylib $(FBE)/Versions/A/Resources/plugins/libEngine12.dylib
    cp ../gen/Release/firebird/lib/libicudata.dylib $(FBE)/Versions/A/libicudata.dylib
    cp ../gen/Release/firebird/lib/libicui18n.dylib $(FBE)/Versions/A/libicui18n.dylib
    cp ../gen/Release/firebird/lib/libicuuc.dylib $(FBE)/Versions/A/libicuuc.dylib
    cp ../gen/Release/firebird/bin/gbak $(FBE)/Versions/A/Resources/bin/gbak
    cp ../gen/Release//firebird/bin/isql $(FBE)/Versions/A/Resources/bin/isql
    cp ../gen/Release/firebird/intl/fbintl.conf $(FBE)/Versions/A/Resources/intl/fbintl.conf
    cp ../gen/Release/firebird/intl/libfbintl.dylib $(FBE)/Versions/A/Resources/intl/libfbintl.dylib
    cp ../gen/Release/firebird/lib/libib_util.dylib $(FBE)/Versions/A/Libraries/libib_util.dylib


    install_name_tool -change /Library/Frameworks/Firebird.framework/Versions/A/Firebird \
     @loader_path/../../libfbclient.dylib $(BINLOC)/isql
    install_name_tool -change /Library/Frameworks/Firebird.framework/Versions/A/Firebird \
     @loader_path/../../libfbclient.dylib $(PLULOC)/libEngine12.dylib
    install_name_tool -change /Library/Frameworks/Firebird.framework/Versions/A/Libraries/libicuuc.dylib \
    @loader_path/../../libicuuc.dylib $(BINLOC)/isql
    install_name_tool -change /Library/Frameworks/Firebird.framework/Versions/A/Libraries/libicudata.dylib \
    @loader_path/../../libicudata.dylib $(BINLOC)/isql
    install_name_tool -change /Library/Frameworks/Firebird.framework/Versions/A/Libraries/libicui18n.dylib \
    @loader_path/../../libicui18n.dylib $(BINLOC)/isql
   
    install_name_tool -change /Library/Frameworks/Firebird.framework/Versions/A/Firebird \
    @loader_path/../../libfbclient.dylib $(BINLOC)/gbak
    install_name_tool -change /Library/Frameworks/Firebird.framework/Versions/A/Libraries/libicuuc.dylib \
    @loader_path/../../libicuuc.dylib $(BINLOC)/gbak
    install_name_tool -change /Library/Frameworks/Firebird.framework/Versions/A/Libraries/libicudata.dylib \
    @loader_path/../../libicudata.dylib $(BINLOC)/gbak
    install_name_tool -change /Library/Frameworks/Firebird.framework/Versions/A/Libraries/libicui18n.dylib \
    @loader_path/../../libicui18n.dylib $(BINLOC)/gbak
   
    install_name_tool -change $(OLDPATH)/libicuuc.dylib @loader_path/libicuuc.dylib \
    $(LIBLOC)/libfbclient.dylib
    install_name_tool -change $(OLDPATH)/libicudata.dylib @loader_path/libicudata.dylib \
    $(LIBLOC)/libfbclient.dylib
    install_name_tool -change $(OLDPATH)/libicui18n.dylib @loader_path/libicui18n.dylib \
    $(LIBLOC)/libfbclient.dylib
    install_name_tool -change $(OLDPATH)/libicudata.dylib @loader_path/libicudata.dylib \
    $(LIBLOC)/libicuuc.dylib
    install_name_tool -change $(OLDPATH)/libicuuc.dylib @loader_path/libicuuc.dylib \
    $(LIBLOC)/libicui18n.dylib
    install_name_tool -change $(OLDPATH)/libicudata.dylib @loader_path/libicudata.dylib \
    $(LIBLOC)/libicui18n.dylib

    install_name_tool -change /Library/Frameworks/Firebird.framework/Versions/A/Firebird \
    @loader_path/../../libfbclient.dylib $(INTLOC)/libfbintl.dylib

    install_name_tool -change /Library/Frameworks/Firebird.framework/Versions/A/Libraries/libicuuc.dylib \
        @loader_path/../libicuuc.dylib $(INTLOC)/libfbintl.dylib
    install_name_tool -change /Library/Frameworks/Firebird.framework/Versions/A/Libraries/libicudata.dylib \
        @loader_path/../libicudata.dylib $(INTLOC)/libfbintl.dylib
    install_name_tool -change /Library/Frameworks/Firebird.framework/Versions/A/Libraries/libicui18n.dylib \
        @loader_path/../libicui18n.dylib $(INTLOC)/libfbintl.dylib

    install_name_tool -id @rpath/libfbclient.dylib $(LIBLOC)/libfbclient.dylib
    install_name_tool -id @rpath/libicudata.dylib $(LIBLOC)/libicudata.dylib
    install_name_tool -id @rpath/libicui18n.dylib $(LIBLOC)/libicui18n.dylib
    install_name_tool -id @rpath/libicudata.dylib $(LIBLOC)/libicudata.dylib
    install_name_tool -id @rpath/libicuuc.dylib $(LIBLOC)/libicuuc.dylib
    install_name_tool -id @rpath/libicudata.dylib $(LIBLOC)/libicudata.dylib
    install_name_tool -id @rpath/libib_util.dylib $(UTILOC)/libib_util.dylib
    install_name_tool -id @rpath/libfbintl.dylib $(INTLOC)/libfbintl.dylib
    install_name_tool -id @rpath/libEngine12.dylib $(PLULOC)/libEngine12.dylib

Wednesday, December 20, 2017

Firebird and Windows 10 Fall Creators Update



Windows 10 is well known for encouraging a love/hate relationship with its users. From the beginning Microsoft implemented a policy of uninstalling users' software without their consent during installation or update of Windows 10. Many articles on the net discuss this, for example however, until now, Firebird seemed to be unaffected.

Not any more. Windows 10 Creators Edition (Build 1709), released in September 2017 seems to have added Firebird to its hit list of dangerous applications. This problem was first reported here.
Because the report mentioned Firebird 1.5 it seemed unfortunate but, hey, Firebird 1.5 has been out of support for eight years. It was first released back in 2003 in the heyday of Windows XP and fourteen years is a long time in the software industry. But not only had Firebird been uninstalled, it was impossible to re-install it. This was likely to affect one of our clients so we spent a long time trying to work around that problem.

Since then we have heard reports from other clients whose customers use Firebird 2.5. After updating Windows 10 to build 1709 Firebird was nowhere to be seen. We have also heard that Windows 10 Enterprise is affected so this is not isolated to the so-called Creators edition. And, at the time of writing we haven't had any feedback about the status of Firebird 3.0.

Luckily, Windows 10 does not block installation of any version of Firebird from V2.0 onwards. But having to re-install is still very annoying, and who knows when Microsoft might decide to change the rules on that?
We spent a long time with our client trying to work out how to re-install Firebird 1.5. Manually setting the configuration using the Properties | Compatibility page and choosing an older version of Windows didn't work. However, running the compatibility troubleshooter did, even though the same setting were being chosen.

Unfortunately, there is so much clicking required that no ordinary user could be relied upon to achieve the desired goal. And trying to talk someone through that over the phone or by email would be extremely frustrating for all involved.

Rebuilding the installer with a newer, Windows 10 aware version of InnoSetup didn't work. But weirdly, a rapidly hacked version using an InterBase template for InstallShield did. Hmmm. And hacking an InnoSetup based installer together with the Firebird 1.5 zip package also worked. So what was going on there?

What criteria does Windows 10 use to decide which programs to ban from installation? There is a known bug in the cpl applet that ships with Firebird 1.5 that can crash Windows Explorer so we tried building an installer without that applet. But no, that wasn't it either.

Finally our client had a brainwave. What happens if Firebird-1.5.6.5026-0-Win32.exe was renamed to, say, setup.exe? Well, what do you know? It works. So simple that we really wanted to kick ourselves for not thinking of it first. And as for the poor Brazilian guy in the forum above, I hope he is sitting down when/if he finds out the solution. It is enough to make anyone want to run screaming into the woods.

Thursday, March 10, 2016

DYLD_LIBRARY_PATH and El Capitan


With the release of Firebird 3 RC2 it is time to concentrate properly on the port of Firebird on MacOSX, Firebird 3 was buildable but we don't have a proper installer, ICU is no longer directly included and there
are a number of other bits and pieces that need to be addressed.

The last time I built Firebird 3 was before I upgraded to El Capitan. I did the upgrade to address some issues with the Firebird installer.

Lets just say when I started ny first build I wasn't expecting too many problems....

When we build Firebird on OSX we arbitrarily point to the ICU location (build_environment/firebird/lib), after copying over the files from the initial ICU build, the Firebird build then finds the ICU libraries dynamically (when needed) because DYLD_LIBRARY_PATH is sey to point to the location via prefix.darwin (make.platform).

While the ICU libraries are being built we set the final actual framework install names and locations of the ICU libraries (using -install_name) and they then placed in the appropriate directory of the framework
when the installer is packaged and built.

Every time I tried to build Firebird it would fall over when trying to build msg.fdb with ISQL.
The error was:
 "Could not find acceptable ICU library"
which seemed very strange as I know it was placed exactly where it should be and where DYLD_LIBRARY_PATH should be picking it up.

Setting DYLD_LIBRARY_PATH in my default terminal worked successfully, checking DYLD_PRINT_ENV seeemed to show the path being set etc., after much head scratching I eventually put the following line of code into unicode_util.cpp at the function:
UnicodeUtil::ConversionICU& UnicodeUtil::getConversionICU()

printf("DYLD_LIBRARY_PATH=%s\n", getenv("DYLD_LIBRARY_PATH"));

When I next ran the build, this is what was returned..

DYLD_LIBRARY_PATH=(null)

Something very strange was definitely going on. Some further research eventually turned up this thread on the
PostgreSQL forums

http://www.postgresql.org/message-id/561E73AB.8060800@gmx.net

"Apparently the behaviour is that DYLD_LIBRARY_PATH is prevented from being inherited into any system binaries. E.g. a shell. But specifying it inside a shell script will allow it to be inherited to children of
that shell, unless the child is a protected process in turn."

My reaction was the same as Tom Lane's but expressed in much stronger terms.

The simplest solution to the problem is to turn off System Integrity Protection

via Recovery Mode (Command-R), and then set csrutil disable in a terminal and reboot.

Note that this is a development only issue and will not affect anybody using the built
binaries when the port is complete.

Wednesday, October 7, 2015

How to install/upgrade Firebird manually on El Capitan


Background: The current Firebird automatic installer on El Capitan is no longer working, its format has been
finally deprecated. The installer is currently undergoing a complete re-write to comply with the Flat Package Format. Until this work is completed, you can manually install Firebird using the following method.

Classic

Download FirebirdCS-2.5.4-26856-x86_64.pkg.zip
Unzip it
cd FirebirdCS-2.5.4-26856-x86_64.pkg
cd Contents
gunzip Archive.pax
pax -r -f Archive.pax (or unpack the file using Finder)
as sudo
cp -r Firebird.framework /Library/Frameworks/Firebird.framework
cd Resources
./postinstall

If you are upgrading from a previous version use:
./preupgrade
and then
./postupgrade

SuperServer

Download FirebirdSS-2.5.4-26856-x86_64.pkg.zip
Then follow the same steps as above, and ignore any errror messages that reference
a, /Resources/doc/doc: No such file or directory
b, chmod /Library/StartupItems/Firebird/Firebird: No such file or directory
c, /Library/LaunchDemons/org.firebird.gds.plist: service is already loaded.

a ps -eaf should show both fbserver and fbguard running.

Monday, July 27, 2015

gbak -stat

From the Bug Tracker:

Vlad Khorsun commented on CORE-1999:
------------------------------------

gbak now has a new command-line switch

    -ST(ATISTICS) TDRW    show statistics:
        T                  time from start
        D                  delta time
        R                  page reads
        W                  page writes

Sample output:

firebird>gbak -v -stat tdrw -r o a.fbk a.fdb
gbak:opened file a.fbk
gbak: time     delta  reads  writes
gbak:    0.173  0.173      0      0 transportable backup -- data in XDR format
gbak:    0.175  0.002      0      0             backup file is compressed
gbak:    0.177  0.001      0      0 backup version is 10
gbak:    0.270  0.092      0    650 created database s:\Temp\A+.FDB, page_size 8192 bytes
gbak:    0.273  0.002      0      2 started transaction
gbak:    0.274  0.001      0      0 restoring domain RDB$29
gbak:    0.275  0.001      0      0 restoring domain RDB$12
...
gbak:   18.661  0.002      0      0 restoring data for table TEST1
gbak:   18.698  0.036      0      0    10000 records restored
gbak:   18.735  0.036      0      0    20000 records restored
...
gbak:   25.177  0.036      0      0    1770000 records restored
gbak:   25.220  0.042      0   1633    1780000 records restored
gbak:   25.256  0.036      0      0    1790000 records restored
...
gbak:   38.702  0.002      0      0     restoring privilege for user SYSDBA
gbak:   38.707  0.004     22      0 creating indexes
gbak:   45.015  6.308     82  38394     activating and creating deferred index T2_VAL
gbak:   45.132  0.116      3      9     activating and creating deferred index TEST_S_UNQ
gbak:   46.486  1.354      0  10775     activating and creating deferred index RDB$PRIMARY1
gbak:   46.566  0.079      0      9     activating and creating deferred index T1_IDX
gbak:   46.661  0.095      5     15 committing metadata
gbak:   46.665  0.003      8     10 fixing views dbkey length
gbak:   46.666  0.001      1     18 updating ownership of packages, procedures and tables
gbak:   46.671  0.005      0      0 adding missing privileges
gbak:   46.673  0.001      0      0 fixing system generators
gbak:   46.682  0.008      4     13 finishing, closing, and going home
gbak:   46.684  0.002    171  82442 total statistics
gbak:adjusting the ONLINE and FORCED WRITES flags

Monday, June 1, 2015

The semantics of isc_tpb_autocommit

From Vlad Horsun:

A simplified overview of the autocommit code:

When a transaction, marked as TRA_autocommit performs any of following actions, it is marked also as TRA_perform_autocommit

  • insert
  • update
  • delete
  • select with lock
  • post event
The TRA_perform_autocommit flag is checked when
  • the engine receives input message
  • the engine sends an output message
  • the engine starts to execute a request
  • the engine finishes executing a DDL request
When the TRA_perform_autocommit flag is detected, the engine runs on-commit triggers (not for DDL, that looks like a bug) and performs commit retaining. A new transaction will have TRA_autocommit flag set and TRA_perform_autocommit not set.